An infrared image and visible light image fusion method and system based on RWKV feature fusion

Abstract

The application provides an infrared image and visible light image fusion method based on RWKV feature fusion, and relates to the technical field of image processing. First, an infrared image and a visible light image are acquired, and a data set is formed. The data set is augmented and expanded, and the augmented and expanded data set is divided into a training set and a test set. An image fusion model is constructed. The image fusion model is trained using the training set to obtain a trained image fusion model. The test set is input into the trained image fusion model to obtain an infrared image and visible light image fusion result. The application realizes high-quality fusion of infrared images and visible light images through the trained image fusion model.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing, and more specifically relates to a method and system for fusing infrared and visible light images based on RWKV feature fusion. Background Technology

[0002] In today's digital age, the development of computer vision technology has brought profound changes to many fields. Multimodal image fusion technology, as a key branch of image processing, has gradually become a focus of research and application. The core of multimodal image fusion lies in integrating information from different sensors or imaging modalities to generate more comprehensive and accurate representations of scenes and objects. Multimodal image fusion can not only overcome the limitations of single-modal images in information representation but also provide a richer data foundation for subsequent image analysis, target recognition, and decision support.

[0003] Infrared and visible light image fusion effectively combines thermal radiation information from an input infrared image with complex texture details from a visible light image to generate a fused image. This fused image significantly mitigates the limitations of visible light images caused by variations in illumination, preserving rich textures, while simultaneously utilizing infrared images to capture temperature distribution information of objects. By merging information from both modalities, it enhances information about the same scene and improves the image's detail representation.

[0004] Fusion of infrared and visible light image information yields higher-quality input, making it more suitable for various computer vision tasks. For example, in the military field, fused images can improve the accuracy of target detection at night, while in autonomous driving, fused images can enhance vehicle perception in complex environments to ensure driving safety. Selecting appropriate image fusion strategies to improve fusion results based on different image features and modal differences is a research challenge and hot topic in infrared and visible light image fusion. Traditional methods have limitations in fusion performance because they cannot fully consider the complex relationships between different image modalities. Furthermore, existing research in image fusion mainly focuses on the architectural design of fusion models, neglecting key issues inherent in the training data itself. This makes it difficult for existing fusion methods to learn universally applicable multimodal information fusion rules. When faced with unseen complex scenes or new modal differences, their generalization ability is significantly insufficient, resulting in limitations such as loss of image detail and weak scene adaptability in the final output image fusion results, making it difficult to meet the demands of complex real-world applications for fused images. Summary of the Invention

[0005] To address the issues of weak scene adaptability and poor fusion effect in existing image fusion methods, this invention proposes a method and system for fusing infrared and visible light images based on RWKV feature fusion, which meets the requirements of complex practical application scenarios for fused images and achieves high-quality fusion of infrared and visible light images.

[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: S1: Acquire infrared and visible light images and construct a dataset. Enhance and expand the dataset, and divide the enhanced and expanded dataset into a training set and a test set. S2: Construct an image fusion model; S3: Train the image fusion model using the training set to obtain the trained image fusion model; S4: Input the test set into the trained image fusion model to obtain the fusion result of infrared and visible light images.

[0007] Furthermore, the process of enhancing and expanding the dataset is as follows: the infrared image is diffused using the visible light image as a reference frame, and T interpolated images are output through T-step linear interpolation; the first image is selected from the T interpolated images to expand the dataset; The operational expression for the T-step linear interpolation is:

[0008] In the formula, Represents infrared image data, Represents visible light image data, Indicates the number of interpolation steps. SD This represents the linear interpolation operation in the Difusion model. This represents the output interpolated image.

[0009] Furthermore, the image fusion model includes: a first encoder module for extracting features from infrared and visible light images, a first RWKV feature fusion module for image feature fusion, and a first decoding module for decoding the fused features; The first encoder module includes: a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer connected in sequence for extracting infrared image features; and a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, and an eighth feature extraction layer connected in sequence for extracting visible light image features. The first RWKV feature fusion module includes: a first RWKV feature fusion submodule, a second RWKV feature fusion submodule, a third RWKV feature fusion submodule, and a fourth RWKV feature fusion submodule; the outputs of the first feature extraction layer and the fifth feature extraction layer are concatenated and connected to the input of the first RWKV feature fusion submodule; the outputs of the second feature extraction layer and the sixth feature extraction layer are concatenated and connected to the input of the second RWKV feature fusion submodule; the outputs of the third feature extraction layer and the seventh feature extraction layer are concatenated and connected to the input of the third RWKV feature fusion submodule; and the outputs of the fourth feature extraction layer and the eighth feature extraction layer are concatenated and connected to the input of the fourth RWKV feature fusion submodule. The first decoding module includes: a first feature decoding layer, a second feature decoding layer, a third feature decoding layer, and a fourth feature decoding layer connected in sequence; the output of the fourth RWKV feature fusion submodule is connected to the input of the fourth feature decoding layer; the output of the third RWKV feature fusion submodule is connected to the input of the third feature decoding layer; the output of the second RWKV feature fusion submodule is connected to the input of the second feature decoding layer; and the output of the first RWKV feature fusion submodule is connected to the input of the first feature decoding layer.

[0010] Furthermore, any feature extraction layer in the first encoder module and any feature decoding layer in the first decoder module include: a first feature mapping operation, a first Vision Transformer operation, and a first CNN operation; a first channel concatenation operation, a first feature mapping operation, and a second CNN operation connected in sequence; the output of the first feature mapping operation is connected to the input of the first Vision Transformer operation and the first CNN operation respectively; the outputs of the first Vision Transformer operation and the first CNN operation are both connected to the input of the first channel concatenation operation; and the input of the first encoder module is connected to the input of the second CNN operation.

[0011] Furthermore, any RWKV feature fusion submodule in the first RWKV feature fusion module includes: a first spatial mixing module and a first channel mixing module; The first spatial mixing module includes: a first normalization layer, a first scale mixing convolutional layer, and a first linear projection layer connected in sequence; it also includes: a first recurrent WKV attention layer and a first activation function layer; the output of the first linear projection layer is connected to the input of the first recurrent WKV attention layer and the first activation function layer respectively; the outputs of the first recurrent WKV attention layer and the first activation function layer are multiplied together and then added to the input of the first spatial mixing module.

[0012] Furthermore, the first channel mixing module includes: a second normalization layer and a second scale mixing convolutional layer connected in sequence; a second linear projection layer and a second activation function layer connected in sequence; a third linear projection layer, a third activation function layer and a fourth linear projection layer connected in sequence; the output of the second scale mixing convolutional layer is connected to the input of the second linear projection layer and the third linear projection layer respectively; the outputs of the second activation function layer and the fourth linear projection layer are multiplied together and then added to the input of the first channel mixing module.

[0013] Furthermore, in the first spatial mixing module: First, after the infrared and visible light images are extracted by the encoder module, the feature map X is stitched together in the channel dimension and then layer normalized to unify the feature distribution. Next, scale-enhanced convolution is performed to extract the basic features of the feature map X spatial dimension, resulting in a multi-scale feature map. The expression is:

[0014]

[0015] In the formula, This indicates scale-enhanced convolution. This represents depthwise separable convolution. , Represents standard convolution. 、 、 、 Indicates learnable weights, Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is:

[0016] In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; 、 Represents a learnable linear projection matrix; The global attention result is calculated using the recurrent WKV attention mechanism, expressed as:

[0017] In the formula, This represents the result of global attention. This represents the recurrent WKV attention mechanism; Using the receiver matrix The sigmoid gate function modulates the attention result and concatenates it with the residual of the input feature map X to obtain the output of the spatial mixing module, expressed as:

[0018] In the formula, This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

[0019] Furthermore, in the first channel mixing module: First, the output of the spatial mixing module After layer normalization and multi-scale hybrid convolution, multi-scale feature maps are obtained. The expression is:

[0020] In the formula, This indicates scale-enhanced convolution. Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is:

[0021] In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; , This represents a learnable linear projection matrix. Indicates the activation function; Using the receiver matrix The gated function modulation value matrix and the output of the spatial mixing module. The residual connection yields the output of the channel mixing module, expressed as:

[0022] In the formula, This indicates the output of the channel mixing module. This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

[0023] Furthermore, during the training of the image fusion model using the training set, a loss function that combines image pixel loss and gradient loss is designed. When the loss function converges, the trained image fusion model is obtained. The expression for image pixel loss is:

[0024] In the formula, Indicates image pixel loss, Represents a visible light image. Represents an infrared image. Indicates a fused image. This indicates the operation of retrieving the maximum value; The expression for image gradient loss is:

[0025] In the formula, Represents image gradient loss, Represents the gradient operator; The expression for the loss function is:

[0026] In the formula, This represents the loss function.

[0027] The present invention also provides an infrared image and visible light image fusion system based on RWKV feature fusion, comprising: The data augmentation module is used to acquire infrared and visible light images, form a dataset, augment and expand the dataset, and divide the augmented dataset into training and testing sets. The model building module is used to build image fusion models; The training module is used to train the image fusion model using the training set to obtain the trained image fusion model. The testing module is used to input the test set into the trained image fusion model to obtain the fusion result of infrared and visible light images.

[0028] Compared with existing technologies, the beneficial effects of this method are: This invention provides a method and system for fusing infrared and visible light images based on RWKV feature fusion. First, infrared and visible light images are acquired and combined into a dataset. This dataset is then augmented and divided into training and testing sets. An image fusion model is constructed. The training set is used to train the image fusion model, resulting in a well-trained model. The testing set is then input into the trained model to obtain the fused infrared and visible light images. This invention achieves high-quality fusion of infrared and visible light images by constructing and training a well-trained image fusion model. By training the constructed image fusion model through data augmentation, this invention effectively combines the thermal radiation information of infrared images with the complex texture details of visible light images. This results in image fusion that clearly preserves scene details while highlighting key target information, achieving high-quality fusion of infrared and visible light images and meeting the needs of complex practical application scenarios for fused images. Attached Figure Description

[0029] Figure 1 A flowchart illustrating the infrared and visible light image fusion method based on RWKV feature fusion proposed in this embodiment of the invention; Figure 2 This diagram illustrates the enhancement and expansion of the dataset proposed in this embodiment of the invention. Figure 3 This diagram illustrates the structure of the image fusion model proposed in this embodiment of the invention. Figure 4 This diagram illustrates the structure of the feature extraction layer and feature decoding layer proposed in this embodiment of the invention. Figure 5 This diagram illustrates the structure of the RWKV feature fusion module proposed in this embodiment of the invention. Figure 6 This image shows the result of fusing infrared and visible light images based on RWKV feature fusion as proposed in this embodiment of the invention. Figure 7 This diagram illustrates the structure of the infrared and visible light image fusion system based on RWKV feature fusion proposed in this embodiment of the invention. Detailed Implementation

[0030] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a method for fusing infrared and visible light images based on RWKV feature fusion, such as... Figure 1 The flowchart shown illustrates the method, and the method proposed in this embodiment generally includes the following steps: S1: Acquire infrared and visible light images and construct a dataset. Enhance and expand the dataset, and divide the enhanced and expanded dataset into a training set and a test set. S2: Construct an image fusion model; S3: Train the image fusion model using the training set to obtain the trained image fusion model; S4: Input the test set into the trained image fusion model to obtain the fusion result of infrared and visible light images.

[0033] In this embodiment, the MSRS dataset was selected for both training and testing. The MSRS dataset is a standard dataset widely used in the field of infrared and visible light image fusion. The images in this dataset have a resolution of 640×480 pixels, covering a wide variety of scenes and fully reflecting the differences and connections between infrared and visible light images at the feature and semantic levels. The dataset was divided into a training set and a test set. The training set contains 1083 image pairs, providing sufficient and diverse samples for training the constructed image fusion model. The test set contains 361 image pairs, used to objectively evaluate the fusion effect and generalization ability of the trained image fusion model. These image pairs were rigorously selected to ensure data quality.

[0034] By inputting the training image pairs from the MSRS dataset into the Difusion model, additional data was effectively generated to expand the existing training set. This new data was then integrated into the original training set, resulting in a larger and more diverse training dataset, thus enriching the number of training samples.

[0035] Images in the training set are cropped into 128×128 pixel image blocks. To ensure the quality of the training samples, contrast filtering is performed to discard samples with low contrast, preventing samples with insufficient visual information from adversely affecting the training of the image fusion model.

[0036] In this embodiment, as Figure 2 The diagram shown illustrates the augmentation and expansion of the dataset. The process of augmenting and expanding the dataset is as follows: the infrared image is diffused using the visible light image as a reference frame, and T interpolated images are output through T-step linear interpolation; the first image is selected from the T interpolated images to expand the dataset. The operational expression for the T-step linear interpolation is:

[0037] In the formula, Represents infrared image data, Represents visible light image data, Indicates the number of interpolation steps. SD This represents the linear interpolation operation in the Difusion model. This represents the output interpolated image.

[0038] In this embodiment, a data augmentation method is proposed to address the problem of multimodal image datasets. This method uses a difusion model to generate images to expand the dataset, thereby enriching the image data and making the image fusion model more robust. To avoid deviating too much from the original data domain, only the first image from each set of T output images is selected to expand the dataset. This channel generation method effectively broadens the dataset size and enriches the image diversity, improving the robustness of the image fusion model.

[0039] The Difusion model transforms the global smoothness objective into a stepwise, achievable objective in each training step by introducing a progressive regularization technique. By constraining the Jacobian matrix of the diffusion system, it ensures that changes in input noise are mapped to changes in output at a fixed ratio, achieving smoothness throughout the diffusion process. The Difusion model is chosen for linear interpolation to generate intermediate frames; its smooth generation allows for the creation of images without information loss while enhancing their richness.

[0040] Existing data augmentation methods mainly rely on simple geometric transformations and color adjustments. While these methods can expand the dataset to some extent, the generated images often lack diversity and are difficult to effectively improve the model's generalization ability. The data augmentation method proposed in this invention introduces a generative model to smoothly interpolate the data to generate images. This eliminates the need for complex domain transfer training processes, generates new samples with styles similar to the target dataset, better matches the distribution of the target dataset, effectively enriches the dataset's diversity, and significantly improves the robustness of the trained image fusion model.

[0041] In this embodiment, as Figure 3 The image fusion model structure diagram shown includes: a first encoder module for extracting features from infrared and visible light images, a first RWKV feature fusion module for image feature fusion, and a first decoding module for decoding the fused features. The first encoder module includes: a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer connected in sequence for extracting infrared image features; and a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, and an eighth feature extraction layer connected in sequence for extracting visible light image features. The first RWKV feature fusion module includes: a first RWKV feature fusion submodule, a second RWKV feature fusion submodule, a third RWKV feature fusion submodule, and a fourth RWKV feature fusion submodule; the outputs of the first feature extraction layer and the fifth feature extraction layer are concatenated and connected to the input of the first RWKV feature fusion submodule; the outputs of the second feature extraction layer and the sixth feature extraction layer are concatenated and connected to the input of the second RWKV feature fusion submodule; the outputs of the third feature extraction layer and the seventh feature extraction layer are concatenated and connected to the input of the third RWKV feature fusion submodule; and the outputs of the fourth feature extraction layer and the eighth feature extraction layer are concatenated and connected to the input of the fourth RWKV feature fusion submodule. The first decoding module includes: a first feature decoding layer, a second feature decoding layer, a third feature decoding layer, and a fourth feature decoding layer connected in sequence; the output of the fourth RWKV feature fusion submodule is connected to the input of the fourth feature decoding layer; the output of the third RWKV feature fusion submodule is connected to the input of the third feature decoding layer; the output of the second RWKV feature fusion submodule is connected to the input of the second feature decoding layer; and the output of the first RWKV feature fusion submodule is connected to the input of the first feature decoding layer.

[0042] In the image fusion model, the input infrared image and visible light image are processed by an encoder to extract features and then concatenated along the channel dimension to obtain the input X. Next, the input is processed by the RWKV feature fusion module to perform feature fusion and then decoded by a decoder to finally obtain the fused image.

[0043] In this embodiment, the RWKV model is applied to the image fusion model, which not only efficiently captures long-distance dependencies and global semantic information of images, compensating for the shortcomings of CNNs, but also reduces computational complexity and memory consumption when processing long sequence data, thus solving the efficiency bottleneck of Transformer. Furthermore, it better balances local details and global information, further improving the fusion effect.

[0044] While existing image fusion models can extract and fuse image features to a certain extent, their effectiveness is often limited when processing complex multimodal data due to difficulties in capturing long-distance dependencies, global semantic information, and performance constraints. The image fusion model provided in this embodiment, with its unique gating mechanism and perceptual capabilities, can efficiently process long sequence data, accurately capture complex semantic dependencies, and also performs excellently in capturing local details. It solves the problems of insufficient semantic consistency, loss of detail, and poor training results in existing image fusion techniques, achieving high-quality multimodal image fusion results and significantly improving the application performance of image processing in multimodal scenarios.

[0045] like Figure 4 The structural diagram shown includes the following: any feature extraction layer in the first encoder module and any feature decoding layer in the first decoder module include: a first feature mapping operation, a first Vision Transformer operation, and a first CNN operation; a first channel concatenation operation, a first feature mapping operation, and a second CNN operation connected in sequence; the output of the first feature mapping operation is connected to the input of the first Vision Transformer operation and the first CNN operation, respectively; the outputs of the first Vision Transformer operation and the first CNN operation are both connected to the input of the first channel concatenation operation; and the input of the first encoder module is connected to the input of the second CNN operation.

[0046] like Figure 5 The structural diagram shown illustrates that any RWKV feature fusion submodule within the first RWKV feature fusion module includes a first spatial mixing module and a first channel mixing module. The first spatial mixing module includes a first normalization layer, a first scale-mixed convolutional layer, and a first linear projection layer connected in sequence; it also includes a first recurrent WKV attention layer and a first activation function layer; the output of the first linear projection layer is connected to the inputs of the first recurrent WKV attention layer and the first activation function layer, respectively; the outputs of the first recurrent WKV attention layer and the first activation function layer are multiplied together and then added to the input of the first spatial mixing module.

[0047] The first channel mixing module includes: a second normalization layer and a second scale mixing convolutional layer connected in sequence; a second linear projection layer and a second activation function layer connected in sequence; a third linear projection layer, a third activation function layer and a fourth linear projection layer connected in sequence; the output of the second scale mixing convolutional layer is connected to the input of the second linear projection layer and the third linear projection layer respectively; the outputs of the second activation function layer and the fourth linear projection layer are multiplied together and then added to the input of the first channel mixing module.

[0048] In this embodiment, feature extraction is achieved through a first spatial mixing module and a first channel mixing module, used for long-range dependency capture in the spatial dimension and feature fusion in the channel dimension. This provides richer feature representations for the subsequent decoder. It not only preserves the key information of the original input image but also improves the performance of the image fusion model through multi-scale and multi-level feature fusion. Finally, after decoding by the decoder, a high-quality image fusion result is generated.

[0049] The image fusion model involved in this invention comprises a U-Net structure consisting of multiple encoders and decoders, with the network including three downsampling and upsampling operations. In the network, the input infrared and visible light images are respectively processed by their encoders for feature extraction. The extracted features are then input together into an RWKV feature fusion module, where the fused features are output to the decoder. The decoder performs layer-by-layer decoding to finally obtain the fused image.

[0050] In the first spatial mixing module: First, after the infrared image and the visible light image are extracted by the encoder module, the feature map X is stitched together in the channel dimension and then layer normalized to unify the feature distribution. Next, scale-enhanced convolution is performed to extract the basic features of the feature map X spatial dimension, resulting in a multi-scale feature map. The expression is:

[0051]

[0052] In the formula, This indicates scale-enhanced convolution. This represents depthwise separable convolution. , Represents standard convolution. 、 、 、 Indicates learnable weights, Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is:

[0053] In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; 、 Represents a learnable linear projection matrix; The global attention result is calculated using the recurrent WKV attention mechanism, expressed as:

[0054] In the formula, This represents the result of global attention. This represents the recurrent WKV attention mechanism; Using the receiver matrix The sigmoid gate function modulates the attention result and concatenates it with the residual of the input feature map X to obtain the output of the spatial mixing module, expressed as:

[0055] In the formula, This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

[0056] In the first channel mixing module: First, the output of the spatial mixing module... After layer normalization and multi-scale hybrid convolution, multi-scale feature maps are obtained. The expression is:

[0057] In the formula, This indicates scale-enhanced convolution. Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is:

[0058] In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; , This represents a learnable linear projection matrix. Indicates the activation function; Using the receiver matrix The gated function modulation value matrix and the output of the spatial mixing module. The residual connection yields the output of the channel mixing module, expressed as:

[0059] In the formula, This indicates the output of the channel mixing module. This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

[0060] Example 2 In this embodiment, the process of training an image fusion model using a training set to obtain a trained image fusion model is described in detail. During the training process, a loss function combining image pixel loss and gradient loss is designed. When the loss function converges, the trained image fusion model is obtained. Image pixel loss measures the difference between the fused image and the target image at the pixel level. It is usually calculated using L1 loss, and the expression for image pixel loss is:

[0061] In the formula, Indicates image pixel loss, Represents a visible light image. Represents an infrared image. Indicates a fused image. This indicates the operation of retrieving the maximum value; Besides pixel loss, gradient loss measures the difference in edge and structural information between the fused image and the target image. Gradient loss helps image fusion models better preserve image details and edge information, thereby improving the visual quality of the fused image. The expression for image gradient loss is:

[0062] In the formula, Represents image gradient loss, This represents the gradient operator.

[0063] The expression for the image gradient operator is:

[0064] In the formula, Representing an image I gradient operator, Representing an image I exist x Partial derivatives in direction, Representing an image I exist y Partial derivatives in direction.

[0065] The expression for the loss function is:

[0066] In the formula, This represents the loss function.

[0067] For example, during training, the image fusion model is trained using the Adam optimizer and a cosine annealing strategy, with the batch size and training epochs set to 4 and 120, respectively.

[0068] To evaluate the performance of this invention, a test set was input into a trained image fusion model to obtain the fusion results of infrared and visible light images. Information entropy (EN), standard deviation (SD), spectral fidelity (SF), absolute gradient (AG), spatial consistency measure (SCD), and fidelity (VIF) were used as evaluation metrics.

[0069] like Figure 6 The image fusion results shown are based on a trained image fusion model, tested on the MSRS test set, and compared with other fusion methods. During testing, the test set was input into the trained image fusion model, which processed the data to generate high-quality fused images. These fused images show improvements in texture detail and structural contrast, demonstrating the image fusion model's ability to capture and integrate multimodal image information.

[0070] Example 3 This embodiment proposes an infrared and visible light image fusion system based on RWKV feature fusion, such as... Figure 7 The structural diagram shown includes: The data augmentation module is used to acquire infrared and visible light images, form a dataset, augment and expand the dataset, and divide the augmented dataset into training and testing sets. The model building module is used to build image fusion models; The training module is used to train the image fusion model using the training set to obtain the trained image fusion model. The testing module is used to input the test set into the trained image fusion model to obtain the fusion result of infrared and visible light images.

[0071] The system proposed in this embodiment improves the performance of image feature fusion and solves the problem of poor results caused by loss of details in existing image fusion technologies.

[0072] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for fusing infrared and visible light images based on RWKV feature fusion, characterized in that, Includes the following steps: Infrared and visible light images are acquired and used to form a dataset. The dataset is augmented and expanded, and then divided into a training set and a test set. Construct an image fusion model; The image fusion model includes: a first encoder module for extracting features from infrared and visible light images, a first RWKV feature fusion module for image feature fusion, and a first decoding module for decoding the fused features; The first encoder module includes: a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, and a fourth feature extraction layer connected in sequence for extracting infrared image features; and a fifth feature extraction layer, a sixth feature extraction layer, a seventh feature extraction layer, and an eighth feature extraction layer connected in sequence for extracting visible light image features. The first RWKV feature fusion module includes: a first RWKV feature fusion submodule, a second RWKV feature fusion submodule, a third RWKV feature fusion submodule, and a fourth RWKV feature fusion submodule; the outputs of the first feature extraction layer and the fifth feature extraction layer are concatenated and connected to the input of the first RWKV feature fusion submodule; the outputs of the second feature extraction layer and the sixth feature extraction layer are concatenated and connected to the input of the second RWKV feature fusion submodule; the outputs of the third feature extraction layer and the seventh feature extraction layer are concatenated and connected to the input of the third RWKV feature fusion submodule; and the outputs of the fourth feature extraction layer and the eighth feature extraction layer are concatenated and connected to the input of the fourth RWKV feature fusion submodule. The first decoding module includes: a first feature decoding layer, a second feature decoding layer, a third feature decoding layer, and a fourth feature decoding layer connected in sequence; the output of the fourth RWKV feature fusion submodule is connected to the input of the fourth feature decoding layer; the output of the third RWKV feature fusion submodule is connected to the input of the third feature decoding layer; the output of the second RWKV feature fusion submodule is connected to the input of the second feature decoding layer; and the output of the first RWKV feature fusion submodule is connected to the input of the first feature decoding layer. The image fusion model is trained using the training set to obtain the trained image fusion model; The test set is input into the trained image fusion model to obtain the fusion result of infrared and visible light images.

2. The infrared and visible light image fusion method based on RWKV feature fusion according to claim 1, characterized in that, The process of enhancing and expanding the dataset is as follows: the infrared image is diffused using the visible light image as a reference frame, and T interpolated images are output through T-step linear interpolation; the first image is selected from the T interpolated images to expand the dataset; The operational expression for the T-step linear interpolation is: In the formula, Represents infrared image data, Represents visible light image data, Indicates the number of interpolation steps. SD This represents the linear interpolation operation in the Difusion model. This represents the output interpolated image.

3. The infrared and visible light image fusion method based on RWKV feature fusion according to claim 1, characterized in that, Each feature extraction layer in the first encoder module and each feature decoding layer in the first decoding module includes: a first feature mapping operation, a first Vision Transformer operation, and a first CNN operation; a first channel concatenation operation, a first feature mapping operation, and a second CNN operation connected in sequence; the output of the first feature mapping operation is connected to the input of the first Vision Transformer operation and the first CNN operation, respectively; the outputs of the first Vision Transformer operation and the first CNN operation are both connected to the input of the first channel concatenation operation; and the input of the first encoder module is connected to the input of the second CNN operation.

4. The infrared and visible light image fusion method based on RWKV feature fusion according to claim 1, characterized in that, Each RWKV feature fusion submodule in the first RWKV feature fusion module includes: a first spatial mixing module and a first channel mixing module; The first spatial mixing module includes: a first normalization layer, a first scale mixing convolutional layer, and a first linear projection layer connected in sequence; it also includes: a first recurrent WKV attention layer and a first activation function layer; the output of the first linear projection layer is connected to the input of the first recurrent WKV attention layer and the first activation function layer respectively; the outputs of the first recurrent WKV attention layer and the first activation function layer are multiplied together and then added to the input of the first spatial mixing module.

5. The infrared image and visible light image fusion method based on RWKV feature fusion according to claim 4, characterized in that, The first channel mixing module includes: a second normalization layer and a second scale mixing convolutional layer connected in sequence; a second linear projection layer and a second activation function layer connected in sequence; a third linear projection layer, a third activation function layer and a fourth linear projection layer connected in sequence; the output of the second scale mixing convolutional layer is connected to the input of the second linear projection layer and the third linear projection layer respectively; the outputs of the second activation function layer and the fourth linear projection layer are multiplied together and then added to the input of the first channel mixing module.

6. The infrared and visible light image fusion method based on RWKV feature fusion according to claim 4, characterized in that, In the first spatial mixing module: First, after the infrared and visible light images are extracted by the encoder module, the feature map X is stitched together in the channel dimension and then layer normalized to unify the feature distribution. Next, scale-enhanced convolution is performed to extract the basic features of the feature map X spatial dimension, resulting in a multi-scale feature map. The expression is: In the formula, This indicates scale-enhanced convolution. This represents depthwise separable convolution. , Represents standard convolution. 、 、 、 Indicates learnable weights, Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is: In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; 、 Represents a learnable linear projection matrix; The global attention result is calculated using the recurrent WKV attention mechanism, expressed as: In the formula, This represents the result of global attention. This represents the recurrent WKV attention mechanism; Using the receiver matrix The sigmoid gate function modulates the attention result and concatenates it with the residual of the input feature map X to obtain the output of the spatial mixing module, expressed as: In the formula, This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

7. The infrared image and visible light image fusion method based on RWKV feature fusion according to claim 5, characterized in that, In the first channel mixing module: First, the output of the spatial mixing module After layer normalization and multi-scale hybrid convolution, multi-scale feature maps are obtained. The expression is: In the formula, This indicates scale-enhanced convolution. Represents multi-scale feature maps, Representation layer normalization; Multi-scale feature maps Projection mapping to the receiver matrix Key matrix Value matrix The expression is: In the formula, Represents the receiver matrix. Represents the key matrix. Represents a value matrix; , This represents a learnable linear projection matrix. Indicates the activation function; Using the receiver matrix The gated function modulation value matrix and the output of the spatial mixing module. The residual connection yields the output of the channel mixing module, expressed as: In the formula, This indicates the output of the channel mixing module. This indicates the output of the spatial mixing module. This represents the activation function. This indicates element-wise multiplication.

8. The infrared image and visible light image fusion method based on RWKV feature fusion according to claim 1, characterized in that, In the process of training the image fusion model using the training set, a loss function that combines image pixel loss and gradient loss is designed. When the loss function converges, the trained image fusion model is obtained. The expression for image pixel loss is: In the formula, Indicates image pixel loss, Represents a visible light image. Represents an infrared image. Indicates a fused image. This indicates the operation of retrieving the maximum value; The expression for image gradient loss is: In the formula, Represents image gradient loss, Represents the gradient operator; The expression for the loss function is: In the formula, This represents the loss function.

9. A system for fusing infrared and visible light images based on RWKV feature fusion, used to implement the method for fusing infrared and visible light images based on RWKV feature fusion as described in any one of claims 1-8, characterized in that, include: The data augmentation module is used to acquire infrared and visible light images, form a dataset, augment and expand the dataset, and divide the augmented dataset into training and testing sets. The model building module is used to build image fusion models; The training module is used to train the image fusion model using the training set to obtain the trained image fusion model. The testing module is used to input the test set into the trained image fusion model to obtain the fusion result of infrared and visible light images.

Images