Infrared and visible image sequence fusion method based on joint low-rank sparse decomposition
By combining low-rank sparse decomposition and targeted fusion rules, infrared and visible image sequences are fused, which solves the problem of insufficient background information of the fusion image, and achieves a fusion effect with clear texture and high contrast, which is suitable for video fusion applications.
Patent Information
- Application Number
- CN202210739930.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-28
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-28
AI Technical Summary
The existing infrared and visible image fusion technology has the problem of insufficient fused image background information, especially when infrared and visible image sequences are fused, it is difficult to effectively retain texture information and saliency information.
The combined low-rank sparse decomposition technology is used to decompose infrared and visible image sequences to obtain common low-rank components, unique low-rank components and sparse components, and fuse them through weighted average and fusion rules based on guiding filtering and phase consistency. The design is aimed at to improve the quality of fusion.
The texture clarity of the fusion image is improved, the infrared radiation information is sufficient, and the contrast is high, which can better retain the texture information and significance information of the source image sequence, and is suitable for video fusion applications.
Smart Images

Figure CN115082369B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of digital image processing, and in particular relates to a method for fusing infrared and visible light image sequences based on joint low-rank sparse decomposition. Background Art
[0002] Infrared images and visible light images are widely used in scientific research, engineering, military and daily life, such as face recognition and access control. The characteristics of the image formed by the infrared sensor are strong penetration ability, less affected by light, strong thermal radiation performance, and can also solve the problem of object occlusion, but the infrared sensor imaging also has the disadvantage of insufficient texture detail information. The characteristics of the image formed by the visible light sensor are high resolution, sufficient texture detail information, and the image looks relatively clear subjectively, but because its imaging method is based on light reflection, it is easily affected by lighting conditions. The infrared and visible light image fusion technology can complement the respective advantages of the images obtained by the two types of sensors, ensuring that the final fused image is full of thermal radiation information, contrast information and detail information, so as to better understand the image target information and finally realize the system's all-weather operation. In recent years, important progress has been made in multi-scale image fusion technology. Generally, the infrared and visible light image fusion scheme based on multi-scale transformation includes three steps. First, each source image is decomposed into a series of multi-scale representations, and then the multi-scale representations of the source image are fused according to the given fusion rules, and finally the fused image is subjected to the corresponding multi-scale inverse transformation. For the fusion of infrared and visible light image sequences, the multi-scale decomposition method is currently used for fusion, but there is a disadvantage that the background information of the fused image is not sufficient. Summary of the invention
[0003] The purpose of the present invention is to provide a method for fusion of infrared and visible light image sequences based on joint low-rank sparse decomposition. Taking into account that the characteristics of infrared images and visible light sequence images determine that they have the same background, and their source images themselves have the characteristics of local correlation and target significance, a joint low-rank sparse decomposition technology is introduced, and then appropriate fusion rules are designed for each component obtained by the decomposition according to their respective characteristics, so as to achieve the purpose of improving the fusion quality.
[0004] The technical solution adopted by the present invention is a method for fusion of infrared and visible light image sequences based on joint low-rank sparse decomposition, which is specifically implemented according to the following steps:
[0005] Step 1: Joint low-rank sparse decomposition: Select image pairs including continuous frames of infrared and visible light images as the data set. First, select M pairs of infrared and visible light image sequences in the data set, and mark them as X. k and Y k, where k represents the kth image sequence, and then the image sequence is subjected to joint low-rank sparse decomposition, and the common low-rank components of M pairs of images, as well as their respective unique low-rank components and sparse components are obtained through the joint low-rank sparse decomposition;
[0006] Step 2: Use the weighted average fusion rule method to fuse the sparse components and obtain the fused image.
[0007] Step 3: Use the fusion rule method based on guided filtering and phase consistency to fuse the unique low-rank components, so that the optimal fusion weights can be designed for the most important part of the joint low-rank sparse decomposition - the unique low-rank components, and then fuse them to obtain the fused image.
[0008] Step 4: The fused image obtained in step 2 and step 3 and The common low-rank component L obtained by the same decomposition C The fusion image F of the kth pair of infrared and visible light images can be obtained by adding them together. k .
[0009] The present invention is also characterized in that:
[0010] Step 1 performs a joint low-rank sparse decomposition of the image sequence as follows:
[0011]
[0012]
[0013] Where X k is the kth infrared sequence image, Y k is the kth visible light sequence image, L c is the common low-rank component, and is the unique low-rank component of the kth infrared and visible light sequence image; and is the sparse component of the kth infrared and visible light sequence image; is the unique low-rank component of the kth infrared sequence image, is the unique low-rank component of the kth visible light sequence image; is the sparse component of the kth infrared sequence image, is the sparse component of the kth visible light sequence image, k is the kth image pair, and K is the total number of K image pairs;
[0014] The specific limiting conditions for decomposition are as follows:
[0015]
[0016]
[0017] Where || || * is the nuclear norm, || ||1 is the 1-norm, and λ1 and λ2 are regularization parameters used to balance the contribution of these two items.
[0018] Step 2 is as follows:
[0019]
[0020] In the formula is the fusion image of the kth pair of sparse components of infrared and visible light sequence images, where the values of s1 and s2 are 1 / 2.
[0021] Step 3 is as follows:
[0022] Step 3.1: The unique low-rank components represent the differences between different frames of the image sequence. First, phase consistency is used to extract the salient features of the two unique low-rank components of the infrared and visible light sequence images, which are expressed as and Then use the maximum rule with selected consistency check to obtain the weight map by comparing the significant features as follows:
[0023]
[0024]
[0025] In the formula is the weight map of the low-rank components unique to infrared sequence images, is the weight map of the low-rank components unique to visible light sequence images, is the salient feature of the low-rank component unique to infrared sequence images, It is the salient feature of the low-rank component unique to the visible light sequence image;
[0026] Step 3.2: Use a spatial consistency solution based on guided filters to solve this problem. Perform a guided filter on each weight map obtained by phase consistency to obtain a refined weight map:
[0027]
[0028]
[0029] in and represents the refinement weight map of infrared and visible light image sequences, where represents the refinement weight map of the infrared image sequence, Represents the refinement weight map of the visible light image sequence, G r,ε represents guided filtering, where r and ε represent two hyperparameters of guided filtering;
[0030] Step 3.3: The refined weight map obtained in step 3.2 is used to fuse the unique low-rank components of the infrared and visible light image sequences. The fusion rules are as follows:
[0031]
[0032] In the formula It is the fusion image of the unique low-rank components of the kth pair of infrared and visible light sequence images.
[0033] Step 4 is as follows:
[0034] The fused image obtained in step 2 And the fused image obtained in step 3 And the common low-rank component L obtained in step 1 C Overlay to get the fused image:
[0035]
[0036] The final F k It is the fused image obtained from the kth image sequence.
[0037] The beneficial effect of the present invention is a method for fusing infrared and visible light image sequences based on joint low-rank sparse decomposition. (1) Taking into full consideration the characteristics of the infrared image and visible light image sequences having the same background and the local correlation and target saliency between the two, the joint low-rank sparse decomposition technology is introduced to decompose the infrared image and visible light image sequences into common low-rank components, unique low-rank components and sparse components, so as to more fully extract the texture information and saliency information of the source image sequence. (2) The method of the present invention takes into full consideration the characteristics of the joint low-rank sparse decomposition image, and proposes a suitable fusion strategy for the decomposed components, thereby retaining more texture information and saliency information of the source image sequence during the fusion process. (3) The method of the present invention fuses the source image sequence, and the obtained fused image has clear texture, sufficient infrared radiation information and high contrast, so it can be applied to video fusion, and therefore has good practical engineering prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 It is the overall flow chart of the method of the present invention;
[0039] Figure 2a It is a partial sequence component map obtained by the joint low-rank sparse decomposition of the infrared image of the Nato_camp_sequence scene by the method of the present invention;
[0040] Figure 2b is a partial sequence component graph obtained by the joint low-rank sparse decomposition of the visible light image of the Nato_camp_sequence scene by the method of the present invention;
[0041] Figure 3a It is a partial sequence component image obtained by the joint low-rank sparse decomposition of the infrared image of the OSU Color and Thermal Database image sequence scene by the method of the present invention;
[0042] Figure 3b It is a partial sequence component image obtained by the joint low-rank sparse decomposition of the visible light image of the OSU Color and Thermal Database image sequence scene by the method of the present invention;
[0043] Figure 4a It is a partial sequence component map obtained by the joint low-rank sparse decomposition of the infrared image of the Bristol Eden Project Multi-Sensor Dataset image sequence scene by the method of the present invention;
[0044] Figure 4b It is a partial sequence component image obtained by the joint low-rank sparse decomposition of the visible light image of the Bristol Eden Project Multi-Sensor Dataset image sequence scene by the method of the present invention;
[0045] Figure 5 It is the fusion result of the Nato_camp_sequence image sequence pair;
[0046] Figure 6 It is the fusion result of OSU Color and Thermal Database image sequence;
[0047] Figure 7 It is the fusion result of the Bristol Eden Project Multi-Sensor Dataset image sequence pair. DETAILED DESCRIPTION
[0048] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0049] Considering the actual engineering application of infrared and visible light image sequence fusion, a method of infrared and visible light image sequence fusion based on joint low-rank sparse decomposition is proposed: first, the source image sequence is decomposed by joint low-rank sparse decomposition to obtain the common low-rank components, unique low-rank components and sparse components of the source image sequence, and then the sparse components are fused by weighted average fusion rules, and the unique low-rank components are fused by appropriate fusion rules designed based on guided filtering and phase consistency. Finally, the final fusion result is obtained by superimposing the fusion results of the common low-rank components, sparse components and unique low-rank components. The overall flow chart is as follows Figure 1 As shown in FIG, the infrared and visible light image sequence fusion method based on joint low-rank sparse decomposition mainly includes the following steps:
[0050] The present invention provides a method for fusion of infrared and visible light image sequences based on joint low-rank sparse decomposition, which is characterized by being implemented in the following steps:
[0051] Step 1: Joint low-rank sparse decomposition: The problem solved by this method is the fusion of infrared and visible light image sequences, so we select image pairs including continuous frames of infrared and visible light images as the data set, such as Nato campsequence. First, we select M pairs of infrared and visible light image sequences in the data set, correspondingly marked as X. k and Y k , where k represents the kth image sequence, and then the image sequence is subjected to joint low-rank sparse decomposition, and the common low-rank components of M pairs of images, as well as their respective unique low-rank components and sparse components are obtained through the joint low-rank sparse decomposition;
[0052] Step 1 performs a joint low-rank sparse decomposition of the image sequence as follows:
[0053]
[0054]
[0055] Where X k is the kth infrared sequence image, Y k is the kth visible light sequence image, L c is the common low-rank component, and is the unique low-rank component of the kth infrared and visible light sequence image; and is the sparse component of the kth infrared and visible light sequence image; is the unique low-rank component of the kth infrared sequence image, is the unique low-rank component of the kth visible light sequence image; is the sparse component of the kth infrared sequence image, is the sparse component of the kth visible light sequence image, k is the kth image pair, and K is the total number of K image pairs;
[0056] The specific limiting conditions for decomposition are as follows:
[0057]
[0058]
[0059] Where || || * is the nuclear norm, || ||1 is the 1-norm, and λ1 and λ2 are regularization parameters used to balance the contribution of these two items.
[0060] Step 2: Use the weighted average fusion rule method to fuse the sparse components and obtain the fused image.
[0061] Step 2 is as follows:
[0062]
[0063] In the formula is the fusion image of the kth pair of sparse components of infrared and visible light sequence images, where the values of s1 and s2 are 1 / 2.
[0064] Step 3: Use the fusion rule method based on guided filtering and phase consistency to fuse the unique low-rank components, so that the optimal fusion weights can be designed for the most important part of the joint low-rank sparse decomposition - the unique low-rank components, and then fuse them to obtain the fused image.
[0065] Step 3 is as follows:
[0066] Step 3.1: The unique low-rank components represent the differences between different frames of the image sequence. First, phase consistency is used to extract the salient features of the two unique low-rank components of the infrared and visible light sequence images, which are expressed as and Then use the maximum rule with selected consistency check to obtain the weight map by comparing the significant features as follows:
[0067]
[0068]
[0069] In the formula is the weight map of the low-rank components unique to infrared sequence images, is the weight map of the low-rank components unique to visible light sequence images, is the salient feature of the low-rank component unique to infrared sequence images, It is the salient feature of the low-rank component unique to visible light sequence images.
[0070] Step 3.2: Since the weight map is noisy and not aligned with the object boundary, which may produce artifacts in the fused image, a spatial consistency scheme based on guided filters is used to solve this problem. A guided filter is applied to each weight map obtained by phase consistency to obtain a refined weight map:
[0071]
[0072]
[0073] in and represents the refinement weight map of infrared and visible light image sequences, where represents the refinement weight map of the infrared image sequence, Represents the refinement weight map of the visible light image sequence, G r,ε represents guided filtering, where r and ε represent two hyperparameters of guided filtering. Because the unique low-rank component represents the detail differences between different frames, the filter parameters here are selected to be small.
[0074] Step 3.3: The refined weight map obtained in step 3.2 is used to fuse the unique low-rank components of the infrared and visible light image sequences. The fusion rules are as follows:
[0075]
[0076] In the formula It is the fusion image of the unique low-rank components of the kth pair of infrared and visible light sequence images.
[0077] Step 4: The fused image obtained in step 2 and step 3 and The common low-rank component L obtained by the same decomposition C The fusion image F of the kth pair of infrared and visible light images can be obtained by adding them together. k .
[0078] The step 4 is specifically as follows:
[0079] The fused image obtained in step 2 And the fused image obtained in step 3 And the common low-rank component L obtained in step 1 C Overlay to get the fused image:
[0080]
[0081] The final F kIt is the fused image obtained from the kth image sequence.
[0082] In order to verify the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.
[0083] The present invention simulates the whole process in the MATLAB2019b environment, and conducts three groups of experiments using the Nato_camp_sequence, OSU Color and Thermal Database image sequence, and Bristol Eden ProjectMulti-Sensor Dataset image sequence data sets.
[0084] First, the source image is subjected to joint low-rank sparse decomposition to obtain its component images. Figure 2a The figure shows a partial sequence component graph obtained by the joint low-rank sparse decomposition of the infrared image of the Nato_camp_sequence scene by the method of the present invention, where X k represents the kth infrared image, L c Represents the common low-rank components of the entire infrared image sequence, represents the unique low-rank component of the kth infrared image, Represents the sparse component of the kth infrared image; Figure 2b The figure shows a partial sequence component graph obtained by the joint low-rank sparse decomposition of the visible light image of the Nato_camp_sequence scene using the method of the present invention, where Y k represents the kth visible light image, L c Represents the common low-rank component of the entire visible light image sequence, represents the unique low-rank component of the kth visible light image, Represents the sparse component of the kth visible light image. From Figure 2, we can see that different components contain different information of the source image. The common low-rank component contains the background information of the entire sequence, the unique low-rank component presents better low-rank information, and the sparse component is a good supplement to the rest of the information, such as the saliency information and texture information of the image sequence; Figure 3a and Figure 3b It is a partial sequence component map obtained by the joint low-rank sparse decomposition of the infrared and visible light images of the OSU Color and Thermal Database image sequence scene by the method of the present invention; Figure 4a and 4bIt is a partial sequence component map obtained by the joint low-rank sparse decomposition of the infrared and visible light images of the Bristol Eden Project Multi-Sensor Dataset image sequence scene using the method of the present invention.
[0085] Figure 5 It is the Nato_camp_sequence image sequence pair. Figure 2a and 2b The final partial fusion result map, Figure 6 is the OSU Color and Thermal Database image sequence. Figure 3a and 3b The final partial fusion result map; Figure 7 It is the Bristol Eden Project Multi-Sensor Dataset image sequence. Figure 4a and 4b The final partial fusion result diagram. Figure 5 It can be seen that for the fusion of infrared and visible light image sequences, the fused image obtained after adopting the guided filtering and weighted averaging fusion rules based on the phase consistency principle has the characteristics of high contrast, prominent target information and complete preservation of thermal radiation information, which is conducive to later applications in other directions, such as behavior recognition and target monitoring.
Claims
1. A method for fusion of infrared and visible light image sequences based on joint low-rank sparse decomposition, characterized in that: Follow the steps below to implement it: Step 1: Joint low-rank sparse decomposition: Select image pairs including continuous frames of infrared and visible light images as the data set. First, select M pairs of infrared and visible light image sequences in the data set, and mark them as X. k and Y k , where k represents the kth image sequence, and then the image sequence is subjected to joint low-rank sparse decomposition, and the common low-rank components of M pairs of images, as well as their respective unique low-rank components and sparse components are obtained through the joint low-rank sparse decomposition; The step 1 performs a joint low-rank sparse decomposition of the image sequence as follows: Where X k is the kth infrared sequence image, Y k is the kth visible light sequence image, L c is the common low-rank component, and is the unique low-rank component of the kth infrared and visible light sequence image; and is the sparse component of the kth infrared and visible light sequence image; is the unique low-rank component of the kth infrared sequence image, is the unique low-rank component of the kth visible light sequence image; is the sparse component of the kth infrared sequence image, is the sparse component of the kth visible light sequence image, k is the kth image pair, and K is the total number of K image pairs; The specific limiting conditions for decomposition are as follows: In the formula * is the nuclear norm, 1 is the 1-norm, λ1 and λ2 are regularization parameters used to balance the contribution of these two terms; Step 2: Use the weighted average fusion rule method to fuse the sparse components and obtain the fused image. Step 3: Use the fusion rule method based on guided filtering and phase consistency to fuse the unique low-rank components, so that the optimal fusion weights can be designed for the most important part of the joint low-rank sparse decomposition - the unique low-rank components, and then fuse them to obtain the fused image. The step 3 is as follows: Step 3.1: The unique low-rank components represent the differences between different frames of the image sequence. First, phase consistency is used to extract the salient features of the two unique low-rank components of the infrared and visible light sequence images, which are expressed as and Then use the maximum rule with selected consistency check to obtain the weight map by comparing the significant features as follows: In the formula is the weight map of the low-rank components unique to infrared sequence images, is the weight map of the low-rank components unique to visible light sequence images, is the salient feature of the low-rank component unique to infrared sequence images, It is the salient feature of the low-rank component unique to the visible light sequence image; Step 3.2: Use a guided filter-based spatial consistency solution to solve this problem. Perform a guided filter on each weight map obtained by phase consistency to obtain a refined weight map: in and represents the refinement weight map of infrared and visible light image sequences, where represents the refinement weight map of the infrared image sequence, Represents the refinement weight map of the visible light image sequence, G r,ε represents guided filtering, where r and ε represent two hyperparameters of guided filtering; Step 3.3: The refined weight map obtained in step 3.2 is used to fuse the unique low-rank components of the infrared and visible light image sequences. The fusion rules are as follows: In the formula is the fusion image of the unique low-rank components of the kth pair of infrared and visible light sequence images; Step 4: The fused image obtained in step 2 and step 3 and The common low-rank component L obtained by the same decomposition C The fusion image F of the kth pair of infrared and visible light images can be obtained by adding them together. k .
2. The infrared and visible light image sequence fusion method based on joint low-rank sparse decomposition according to claim 1, characterized in that: The step 2 is specifically as follows: In the formula is the fusion image of the kth pair of sparse components of infrared and visible light sequence images, where the values of s1 and s2 are 1 / 2.
3. The method for fusion of infrared and visible light image sequences based on joint low-rank sparse decomposition according to claim 2, characterized in that: The step 4 is specifically as follows: The fused image obtained in step 2 And the fused image obtained in step 3 And the common low-rank component L obtained in step 1 C Overlay to get the fused image: The final F k It is the fused image obtained from the kth image sequence.
Citation Information
Patent Citations
An infrared image and visible light image fusion method
CN109886908A
Multi-focus image fusion method based on superpixel clustering and joint low-rank representation
CN110097530A
Infrared and visible light image fusion method based on multi-scale low-rank matrix decomposition
CN111462025A
Image fusion method based on potential low-rank representation nested rolling guide image filtering
CN112950518A
Light-weight pose estimation network with multi-scale heatmap fusion
WO2021163103A1
