Infrared and visible light image fusion method based on RPCANet

Through the low-rank sparse decomposition method based on RPCANet, the problem of real-time and accuracy requirements of infrared and visible image fusion technology in different application scenarios is solved, and a more efficient and accurate image fusion effect is achieved.

CN118887508AInactive Publication Date: 2024-11-01NANJING UNIV OF POSTS & TELECOMM
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411366559.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-29
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing infrared and visible image fusion technology is difficult to meet the real-time and accuracy requirements in different application scenarios, and traditional methods fail to effectively consider the intrinsic differences in images, resulting in poor fusion effect.

Method used

Using the RPCANet-based method, the data set containing infrared images and visible light images were constructed, and the image registration pre-processing was performed, and the low-rank sparse decomposition was used to perform low-rank sparse decomposition. After decomposition, weighted fusion and VGG-19 network fusion were performed respectively, and finally the Final fused image.

Benefits of technology

It improves the efficiency and accuracy of image fusion, makes up for the problems of insufficient infrared targets and blurred edges in traditional methods, and improves the objective evaluation index of the image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887508A_ABST
    Figure CN118887508A_ABST
Patent Text Reader

Abstract

The invention discloses an infrared and visible light image fusion method based on RPCANet, and the method comprises the following steps: constructing a data set containing an infrared image and a visible light image, and carrying out the image registration preprocessing; an RPCANet model is constructed to decompose the infrared image and the visible light image, the RPCANet model comprises a background estimation module, a target extraction module and an image reconstruction module, and low-rank parts and sparse parts of the infrared image and the visible light image are obtained after decomposition; and respectively fusing the low-rank part and the sparse part, and reconstructing the image to obtain a final fused image. According to the method, the problems that in a traditional infrared and visible light image fusion method, an infrared target is not prominent enough, and the edge is fuzzy are solved, and the evaluation index is improved compared with the traditional fusion method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image information processing, and in particular to an infrared and visible light image fusion method based on RPCANet. Background Art

[0002] Image fusion is the most challenging and long-standing task in the field of image processing. Due to the limitations of imaging technology or sensors, it is difficult for an image acquired by a single sensor to represent a comprehensive or detailed scene. For example, infrared images can detect targets under various lighting conditions. However, it usually provides limited scene details at a low spatial resolution. In contrast, visible light images are generated by capturing visible light, which reveals rich texture details and is consistent with the human visual system. However, visible light images are greatly affected by weather. Therefore, infrared and visible light image fusion can overcome the limitations of hardware devices and has been widely used in remote sensing, target detection, military reconnaissance and other fields. Infrared and visible light image fusion methods can be roughly divided into two categories: traditional methods and deep learning methods. Traditional methods deform the input image into a specific transform domain through certain mathematical transformations, and then manually set fusion rules, such as multi-scale transformation, sparse representation, saliency detection, etc. However, traditional methods do not consider the intrinsic differences of images, and use a unified mathematical model to extract image features indiscriminately, resulting in poor fusion effects in some cases. In recent years, with the rapid development of deep learning in computer vision tasks, a large number of deep learning based image fusion methods have been proposed, such as fusion methods based on convolutional neural networks and fusion methods based on generative adversarial networks.

[0003] In recent years, deep learning registration methods have shown better results and faster speeds than traditional methods in the registration of medical images and remote sensing images. Some researchers have tried to apply deep learning networks such as convolutional neural networks and generative adversarial networks to infrared and visible light image registration and have made some progress, but there is still a certain gap in registration accuracy compared with traditional methods. Deep learning methods face many challenges in achieving excellent infrared and visible image registration.

[0004] The infrared and visible light registration technology has been developed to cover the fields of power, military, assisted driving and face recognition. Although there are many infrared and visible light image registration algorithms, different application scenarios have different application requirements. For example, the field of online monitoring of power systems has high requirements for real-time image performance, while the medical field has high requirements for image registration accuracy. Therefore, there is currently no comprehensive image registration algorithm that can cover all application scenarios. Summary of the invention

[0005] The purpose of the present invention is to address the shortcomings of existing infrared and visible light image fusion technologies and propose an infrared and visible light image fusion method based on RPCANet, comprising the following steps: Construct a data set containing infrared images and visible light images, and perform image registration preprocessing; The RPCANet model is constructed to decompose the infrared image and the visible light image. The RPCANet model includes a background estimation module, a target extraction module and an image reconstruction module. After decomposition, the low-rank part and the sparse part of the infrared image and the visible light image are obtained. The low-rank part and the sparse part are fused separately, and the image is reconstructed to obtain the final fused image.

[0006] Furthermore, in the RPCANet model, the model input includes infrared images and visible light images , and Represent the height and width of the image respectively, and the update parameters are initialized as , ; Pass the parameters through decomposition stages, each of which corresponds to an iterative matrix low-rank sparse decomposition process, simulating the multiple iterative update operations in the model-driven method; the updated parameters and Send to decomposition stages, including , respectively calculate the background of the current stage through the background estimation module, target extraction module and image reconstruction module ,Target And the reconstruction results .

[0007] Furthermore, the background estimation module uses a residual structure for background estimation, including an intermediate layer and two convolutional layers; the target extraction module includes a convolutional layer and a ReLU activation function; and the image reconstruction module adopts a CNN architecture.

[0008] Furthermore, the low-rank part and the sparse part are fused separately, including: the low-rank part adopts weighted fusion, and the sparse part adopts VGG-19 network fusion.

[0009] Furthermore, the low-rank part adopts weighted fusion, and the specific formula is: (6) in, and They represent the low-rank parts of the infrared image and the visible light image after being decomposed by the RPCANet network. and Respectively represent the corresponding weight values, represents the low-rank partially fused image.

[0010] Furthermore, the sparse part is fused with the VGG-19 network, which includes: Through upsampling and weight adjustment, the details and sparse information of the image are restored and preserved: (7) in, Indicated by The first layer of extraction The features of the sparse part, ; Indicates The number of channels of the layer, , ; Represents the first layer, ; , Represents sparse partial images of infrared and visible light images; Get deep features Afterwards, through The norm and block-based average operators compute activity levels: (8) in, Respectively represent layer1, layer2, layer3, layer4, represents the average operator window size, and Respectively represent relative to the current position The offset of ; After obtaining the activity level, the initial weight is calculated through the soft-max operator: (9) in, A quantity representing the final activity level, express The value of the initial weight in the range; The final weight is obtained by calculating the upsampling operator : (10) in, , , ; Calculate all weights obtained , get the initial fusion graph of the sparse part: (11) in, Represents the sparse content of an image; Get the final fused image of the sparse part: (12) in, Represents the final fused image of the sparse part.

[0011] Furthermore, the image is reconstructed to obtain the final fused image. The specific formula is: (13) in, Represents the final fused image.

[0012] According to one aspect of the present invention, a storage medium is provided, wherein instructions are stored in the storage medium. When a computer reads the instructions, the computer executes any one of the above-mentioned infrared and visible light image fusion methods based on RPCANet.

[0013] According to another aspect of the present invention, there is provided an electronic device, comprising a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention uses the pre-trained deep neural network VGG19 as the feature extractor for image fusion. Because no specific data is required for training, the fusion efficiency is greatly improved.

[0015] 2. The present invention adopts the principle of low-rank sparse decomposition to construct a neural network model, which realizes the decomposition and processing of the background low-rank area and sparse target area of ​​the image, and makes up for the problems of insufficient prominence of infrared targets and blurred edges in traditional infrared and visible light image fusion methods.

[0016] 3. The objective image evaluation index of the infrared and visible light fusion method based on RPCANet proposed in this invention is improved compared with the traditional fusion method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention; Figure 2 is a system framework diagram of an embodiment of the present invention; Figure 3 This is a structural diagram of the RPCANet model of an embodiment of the present invention; Figure 4 This is a diagram of the VGG19 network structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] like Figure 1 As shown in FIG. 1 , the infrared and visible light image fusion method based on RPCANet includes: Step 1: Dataset preprocessing: Construct a dataset containing infrared and visible light images for neural network training and optimization, and perform preprocessing. Step 2, RPCANet model construction: A low-rank sparse decomposition method is used to construct a neural network model, which includes a background estimation module, a target extraction module, and an image reconstruction module, which is used to realize the decomposition and reconstruction fusion of infrared and visible light images; Step 3: Image fusion: The low-rank parts of the infrared and visible light images decomposed by RPCANet are fused by weighted average, and the sparse parts are fused by VGG-19 network. Finally, the image is reconstructed to obtain the fused image.

[0020] This paper designs a network architecture RPCANet based on low-rank sparse decomposition, which uses neural layers to approximate the sparsity constraint function in the target extraction module (TEM). In addition, a background extraction module (BEM) is designed to simulate the proximal function of the convolutional layer. An image reconstruction module (IRM) is introduced to achieve iterative combination output of the target and background.

[0021] Model input includes infrared images and visible light images , and Represent the height and width of the image respectively, and the update parameters are initialized as , ; Pass the parameters through decomposition stages, each of which corresponds to an iterative matrix low-rank sparse decomposition process, simulating the multiple iterative update operations in the model-driven method; the updated parameters and Send to decomposition stages, including , respectively calculate the background of the current stage through the background estimation module, target extraction module and image reconstruction module ,Target And the reconstruction results .generally, Represents the latent variable of the current decomposition stage and does not participate in parameter transfer between stages.

[0022] 2.1 Background Estimation Module (BEM) The residual proxNet(·) is used to simulate this operator: (1) in, Indicates the structure The convolution group of Depend on The middle layer and two convolutional layers.

[0023] 2.2 Target Extraction Module (TEM) This module has an updated background ,Target , reconstruction results In addition, we set γ = 0.5 to equalize the sizes of the three inputs and rewrite the target matrix as: (2) Assign the parameter learning task to for , which is a learnable and unshared parameter that is independent of each reconstruction stage. Continuous gradient function , is a single-layer scalar CNN consisting of a convolutional layer and a ReLU activation function. Therefore, in order to avoid complex network design, a simple convolutional layer and ReLU activation function are used to simulate . Introduced the last With the updated To enhance the continuous target features, The update function is: (3) in, By the initial convolution layer, the middle layer and reconstruction layers.

[0024] 2.3 Image Reconstruction Module (IRM) To cooperate with the RPCA process of mapping the target and background into the restored image, an IRM is designed, which uses a neural network Convert the decomposition task into an image reconstruction task: (4) The present invention does not use residual blocks or other complex networks in the reconstruction module, but adopts a simple CNN architecture to learn image features and effectively map the decomposed background and target. resemblance, There are three types of convolutional layers, the middle layer is ,in .

[0025] In summary, this paper proposes an end-to-end training framework for low-rank sparse decomposition tasks called RPCANet. Its trainable parameter set Contains the convolutional network parameters in BEM, TEM, and IRM at each decomposition stage, as well as independent trainable scalars in TEM , whose set is , where is the number of overall decomposition stages.

[0026] 2.4 RPCA network loss function The infrared and visible light image fusion task is divided into image decomposition and reconstruction, and the loss function is and The former uses SoftIoU to measure the performance of decomposing sparse targets, and the latter uses reconstructed images. With the original image The least square error between is used to measure the performance of infrared and visible light image reconstruction. The loss function is defined as follows: (5) Among them, TP is the number of samples that are actually positive examples and are correctly identified as positive examples, FP is the number of samples that are actually negative examples but are mistakenly identified as positive examples, TN is the number of samples that are actually negative examples and are correctly identified as negative examples, and FN is the number of samples that are actually positive examples but are mistakenly identified as negative examples. and are the total number of training times and the total pixels of each image, respectively. is the regularization parameter, which is set to 0.01 in the experiment.

[0027] 3.1 Low-rank partial fusion After the infrared and visible light images are decomposed by RPCANet low-rank sparse decomposition, the low-rank part fusion formula is: (6) in , They respectively represent the low-rank parts of the infrared and visible light images after decomposition through the RPCANet network.

[0028] 3.2 Sparse Part Fusion VGG (visual geometry group) is a classic deep convolutional neural network model. Its core idea is to build a deep network by stacking multiple smaller convolutional layers and pooling layers. For a given receptive field, it is better to use a small convolution kernel than a large convolution kernel. The VGG19 model has 19 hidden layers, including 16 convolutional layers (including 5 maximum pooling layers) and 3 fully connected layers. Multiple nonlinear layers increase the depth of the network, enabling VGG19 to learn more complex feature expression capabilities. Because the deep characteristics of VGG19 have a strong advantage in feature extraction, the model is fine-tuned to extract the deep features of the image and retain more detailed information of the source image. The present invention uses a pre-trained deep neural network VGG19 as a feature extractor for image fusion. Because no specific data is required for training, the fusion efficiency is greatly improved.

[0029] For the sparse part, the VGG19 model is used to extract deep features, and the fusion weights are obtained through a multi-layer fusion strategy. The fused sparse information is reconstructed through the weights and the details in the sparse part. Through upsampling technology and weight adjustment, it can be ensured that the details and sparse information of the image are restored and preserved when the image size is enlarged: (7) in, Indicated by The first layer of extraction The features of the sparse part, ; Indicates The number of channels of the layer, , ; Represents the first layer, ; , Represents sparse partial images of infrared and visible light images; Get deep features Afterwards, through norm and block-based average operator to calculate its activity level : (8) in, Respectively represent layer1, layer2, layer3, layer4, represents the average operator window size, if If it is too large, it will lead to the loss of detailed information, so . and Respectively represent relative to the current position The offset of . Get activity level After that, the initial weight is calculated by the soft-max operator : (9) in, A quantity representing the final activity level, express The value of the initial weight in the range. Because the pooling operation in the VGG network is a subsampling method that can reduce the size of the feature map, each time the pooling operation is performed, the feature map will be reduced to the original , is the step size of the pooling operation, . So in different layers, the size of the feature map is , .

[0030] In order to make the output size consistent with the input detail content size, the upsampling operator calculates the final weights : (10) in, , , .

[0031] Calculate all weights obtained , and obtain the initial fusion graph of the sparse part.

[0032] (11) in, Represents the sparse content of an image; Get the final fused image of the sparse part: (12) in, Represents the final fused image of the sparse part.

[0033] 3.3 Output fused image After obtaining the low-rank partial fusion image Fusion of images with sparse parts After that, the fused image can be obtained by the following formula : (13) in, Represents the final fused image.

[0034] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.

Claims

1. A method for fusion of infrared and visible light images based on RPCANet, characterized in that: The following steps are involved: Construct a data set containing infrared images and visible light images, and perform image registration preprocessing; The RPCANet model is constructed to decompose the infrared image and the visible light image. The RPCANet model includes a background estimation module, a target extraction module and an image reconstruction module. After decomposition, the low-rank part and the sparse part of the infrared image and the visible light image are obtained. The low-rank part and the sparse part are fused separately, and the image is reconstructed to obtain the final fused image.

2. The method according to claim 1, characterized in that In the RPCANet model, the model input includes infrared images and visible light images , and Represent the height and width of the image respectively, and the update parameters are initialized as , ; Pass the parameters through decomposition stages, each of which corresponds to an iterative matrix low-rank sparse decomposition process, simulating the multiple iterative update operations in the model-driven method; the updated parameters and Send to decomposition stages, including , respectively calculate the background of the current stage through the background estimation module, target extraction module and image reconstruction module ,Target And the reconstruction results .

3. The method according to claim 2, characterized in that The background estimation module uses a residual structure for background estimation, including an intermediate layer and two convolutional layers; the target extraction module includes a convolutional layer and a ReLU activation function; and the image reconstruction module adopts a CNN architecture.

4. The method according to claim 3, characterized in that The low-rank part and the sparse part are fused separately, including: the low-rank part uses weighted fusion, and the sparse part uses VGG-19 network fusion.

5. The method according to claim 4, characterized in that The low-rank part adopts weighted fusion, and the specific formula is: (6) in, and They represent the low-rank parts of the infrared image and the visible light image after being decomposed by the RPCANet network. and Respectively represent the corresponding weight values, represents the low-rank partially fused image.

6. The method according to claim 4, characterized in that The sparse part uses VGG-19 network fusion, which includes: Through upsampling and weight adjustment, the details and sparse information of the image are restored and preserved: (7) in, Indicated by The first layer of extraction The features of the sparse part, ; Indicates The number of channels of the layer, , ; Represents the first layer, ; , Represents sparse partial images of infrared and visible light images; Get deep features Afterwards, through The norm and block-based average operators compute activity levels: (8) in, Respectively represent layer1, layer2, layer3, layer4, represents the average operator window size, and Respectively represent relative to the current position The offset of ; After obtaining the activity level, the initial weight is calculated through the soft-max operator: (9) in, A quantity representing the final activity level, express The value of the initial weight in the range; The final weight is obtained by calculating the upsampling operator : (10) in, , , ; Calculate all weights obtained , get the initial fusion graph of the sparse part: (11) in, Represents the sparse content of an image; Get the final fused image of the sparse part: (12) in, Represents the final fused image of the sparse part.

7. The method according to claim 1, characterized in that The image is reconstructed to obtain the final fused image. The specific formula is: (13) in, Represents the final fused image.

8. A storage medium, characterized in that: The storage medium stores instructions, and when a computer reads the instructions, the computer executes the infrared and visible light image fusion method based on RPCANet as described in any one of claims 1-7.

9. An electronic device, characterized in that: The invention comprises a processor and the storage medium as claimed in claim 8, wherein the processor executes instructions in the storage medium.

Citation Information

Patent Citations

  • An infrared image and visible light image fusion method

    CN109886908A

  • Infrared and visible light image fusion method and system based on low-rank sparse representation

    CN114648475A