Image fusion method and system based on visual state space model, computer equipment and storage medium

Abstract

The invention provides an image fusion method and system based on a visual state space model, computer equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: respectively carrying out image partitioning on an infrared image and a visible light image to obtain a plurality of two-dimensional image blocks; converting the plurality of two-dimensional image blocks into a one-dimensional sequence; performing feature extraction on the one-dimensional sequence to obtain features of the infrared light image and the visible light image before fusion; inputting the features before fusion into a plurality of 3 * 3 convolution kernels to obtain a first fusion image; and inputting the first fused image into a decoder comprising a plurality of stacked visual state space modules, and outputting to obtain a second fused image as a final fused image. The thermal information of the infrared image and the detail information of the visible light image can be reserved, and the quality of the fused image is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image fusion method and system based on a bidirectional state space model. Background Art

[0002] The fusion technology of infrared images and visible light images is one of the important research directions in the field of computer vision. With the advancement of sensor technology and image processing algorithms, the fusion of infrared images and visible light images has shown great potential in applications such as night vision monitoring, medical imaging, and military reconnaissance. Infrared images can provide clear thermal information under low illumination or bad weather conditions, while visible light images can provide rich details and color information. By effectively fusing the two images, more comprehensive and accurate environmental information can be obtained, improving the performance of target recognition and detection.

[0003] At present, the fusion of infrared images and visible light images is mainly based on pixel-level fusion, that is, the pixel-level fusion method directly processes the pixels of infrared images and visible light images. A fused image can be obtained by weighted averaging the infrared image and the visible light image, or by extracting the main components of the infrared image and the visible light image through principal component analysis and fusion of these components, or by using wavelet transform to perform multi-scale decomposition of the infrared image and the visible light image, and then fuse the images at different scales. However, the pixel-level fusion method usually finds it difficult to simultaneously retain the thermal information of the infrared image and the detail information of the visible light image, resulting in low quality of the fused image. Summary of the invention

[0004] The present invention provides an image fusion method, system, computer device and storage medium based on a visual state space model to solve the problem in the related art that it is difficult to simultaneously retain the thermal information of infrared images and the detail information of visible light images, resulting in low quality of fused images.

[0005] In a first aspect, the present invention provides an image fusion method based on a visual state space model, comprising:

[0006] The infrared image and the visible light image are divided into blocks respectively to obtain a plurality of two-dimensional image blocks;

[0007] Convert multiple two-dimensional image blocks into a one-dimensional sequence;

[0008] Perform feature extraction on the one-dimensional sequence to obtain the features before the fusion of the infrared image and the visible light image;

[0009] Input the features before fusion into multiple 3x3 convolution kernels to obtain the first fused image;

[0010] The first fused image is input to a decoder including a plurality of visual state space modules stacked together, and a second fused image is output as a final fused image.

[0011] In one embodiment of the present invention, the image segmentation of the infrared image and the visible light image to obtain a plurality of two-dimensional image blocks includes:

[0012] The infrared image and the visible light image are uniformly cropped to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

[0013] In one embodiment of the present invention, converting the multiple two-dimensional image blocks into a one-dimensional sequence includes:

[0014] Construct a one-dimensional sequence T0 expression:

[0015]

[0016] Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

[0017] In one embodiment of the present invention, extracting features from the one-dimensional sequence to obtain features before the infrared image and the visible light image are fused includes:

[0018] Constructing basic features of infrared and visible light images expression:

[0019]

[0020] Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation;

[0021] Constructing high-level features of infrared and visible light images expression:

[0022]

[0023] Construct the feature F before fusion of infrared image and visible light image out expression:

[0024]

[0025] Where ⊙ represents point-by-point multiplication.

[0026] In a second aspect, the present invention provides an image fusion system based on a visual state space model, comprising:

[0027] An image segmentation module is used to segment the infrared image and the visible light image into blocks to obtain a plurality of two-dimensional image blocks;

[0028] An image conversion module, used for converting a plurality of two-dimensional image blocks into a one-dimensional sequence;

[0029] A feature extraction module is used to extract features from the one-dimensional sequence to obtain features before the infrared image and the visible light image are fused;

[0030] A first fused image generation module, used for inputting the features before fusion into a plurality of 3x3 convolution kernels to obtain a first fused image;

[0031] The second fused image generating module is used to input the first fused image into a decoder including a plurality of visual state space modules stacked together, and output a second fused image as a final fused image.

[0032] In one embodiment of the present invention, the image conversion module includes:

[0033] The image cropping unit is used to perform uniform cropping operations on the infrared image and the visible light image respectively to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

[0034] In one embodiment of the present invention, the image conversion module includes:

[0035] The first building block is used to construct a one-dimensional sequence T0 expression:

[0036]

[0037] Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

[0038] In one embodiment of the present invention, the feature extraction module includes:

[0039] The second construction unit is used to construct the basic features of infrared light images and visible light images expression:

[0040]

[0041] Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation;

[0042] The third construction unit is used to construct high-order features of infrared light images and visible light images expression:

[0043]

[0044] The fourth construction unit is used to construct the feature F before the fusion of the infrared light image and the visible light image. out expression:

[0045]

[0046] Where ⊙ represents point-by-point multiplication.

[0047] In a third aspect, the present invention provides a computer device comprising a processor and a memory; wherein, when the processor executes a computer program stored in the memory, the steps of the image fusion method based on the visual state space model described in the first aspect are implemented.

[0048] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the image fusion method based on the visual state space model described in the first aspect are implemented.

[0049] The present invention provides an image fusion method, system, computer device and storage medium based on a visual state space model, wherein the method comprises performing image segmentation on an infrared image and a visible light image respectively to obtain a plurality of two-dimensional image blocks; converting the plurality of two-dimensional image blocks into a one-dimensional sequence; extracting features from the one-dimensional sequence to obtain features before fusion of the infrared light image and the visible light image; inputting the features before fusion into a plurality of 3x3 convolution kernels to obtain a first fused image; inputting the first fused image into a decoder including a plurality of visual state space modules stacked, and outputting a second fused image as a final fused image. The present invention can retain the thermal information of the infrared image and the detail information of the visible light image, and improve the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of a flow chart of an image fusion method based on a visual state space model provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of the structure of an image fusion system based on a visual state space model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment provides an image fusion method based on a visual state space model, including:

[0056] Step 101 , dividing the infrared image and the visible light image into blocks respectively to obtain a plurality of two-dimensional image blocks.

[0057] In this step, the infrared image and the visible light image are uniformly cropped to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

[0058] Step 102: convert multiple two-dimensional image blocks into a one-dimensional sequence.

[0059] For example, construct a one-dimensional sequence T0 expression:

[0060]

[0061] Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

[0062] Step 103, extracting features from the one-dimensional sequence to obtain features before the fusion of the infrared light image and the visible light image.

[0063] In the task of infrared and visible light fusion, the goal of feature extraction is to fuse the spectral information of infrared and visible light to improve the overall performance of the visual model. Infrared light can capture thermal information that visible light cannot obtain, while visible light provides rich color and texture details. By fusing these two types of information, more expressive features can be generated, resulting in better performance in complex visual tasks.

[0064] The visual state space module (VSS) is specifically implemented as follows: a 2D state space model is introduced on the underlying visual image super-resolution model, and a two-dimensional selective scanning model SS2D is used to implement feature modeling in four directions (which can generate richer and more diverse feature representations. These multi-directional features contain more contextual information and can better describe the global structure and local details of the image).

[0065] Exemplarily, construct basic features of infrared light images and visible light images expression:

[0066]

[0067] Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation.

[0068] Depthwise separable convolution (DWConv) further enhances the fused features locally, retains important information, and reduces computational complexity. LayerNorm and nonlinear activation are normalized and activated after each step, respectively, to ensure the stability of feature distribution and nonlinear expression ability, to extract the main patterns and information in the input features, and to capture the basic characteristics of infrared and visible light.

[0069] Constructing high-level features of infrared and visible light images expression:

[0070]

[0071] Construct the feature F before fusion of infrared image and visible light image out expression:

[0072]

[0073] Where ⊙ represents point-by-point multiplication.

[0074] Step 104, input the features before fusion into multiple 3x3 convolution kernels to obtain a first fused image.

[0075] Step 105 , input the first fused image to a decoder including a plurality of visual state space modules stacked together, and output a second fused image as a final fused image.

[0076] In steps 104-105, feature fusion uses VC-block composed of multiple 3x3 convolution kernels to further process and fuse the input features (features before fusion). Specifically, through the convolution operation of the 3x3 convolution kernel, VC-block can capture the local spatial relationship and pattern between features, so as to better extract the detailed information in the features. In addition, feature fusion also includes combining and interacting features of different channels or levels to enhance their complementarity, so that the final output features can more comprehensively express the important information in the original input.

[0077] The fused features are further input into the decoder stacked by VSS blocks to gradually restore image details and quality and perform image reconstruction. The decoder outputs the fused image.

[0078] In summary, this embodiment provides an image fusion method based on a visual state space model, including image segmentation of infrared images and visible light images respectively to obtain multiple two-dimensional image blocks; converting multiple two-dimensional image blocks into one-dimensional sequences; extracting features from the one-dimensional sequences to obtain features before fusion of infrared light images and visible light images; inputting the features before fusion into multiple 3x3 convolution kernels to obtain a first fused image; inputting the first fused image into a decoder including multiple visual state space modules stacked, and outputting a second fused image as the final fused image. The image fusion method provided in this embodiment can retain the thermal information of infrared images and the detail information of visible light images, thereby improving the quality of fused images.

[0079] Example 2

[0080] Based on the same inventive concept as Example 1, this embodiment provides an image fusion system based on a visual state space model. Since the principle of solving the problem by this system is similar to the aforementioned image fusion method based on a visual state space model, the implementation of this system can refer to the implementation of the image fusion method based on a visual state space model.

[0081] like Figure 2 As shown, the image fusion system based on the visual state space model includes:

[0082] The image segmentation module 10 is used to segment the infrared image and the visible light image into blocks to obtain a plurality of two-dimensional image blocks.

[0083] The image conversion module 20 is used to convert a plurality of two-dimensional image blocks into a one-dimensional sequence.

[0084] The feature extraction module 30 is used to extract features from the one-dimensional sequence to obtain features before the infrared image and the visible light image are fused.

[0085] The first fused image generating module 40 is used to input the features before fusion into a plurality of 3x3 convolution kernels to obtain a first fused image.

[0086] The second fused image generating module 50 is used to input the first fused image into a decoder including a plurality of visual state space modules stacked together, and output a second fused image as a final fused image.

[0087] Exemplarily, the image conversion module includes:

[0088] The image cropping unit is used to perform uniform cropping operations on the infrared image and the visible light image respectively to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

[0089] Exemplarily, the image conversion module includes:

[0090] The first building block is used to construct a one-dimensional sequence T0 expression:

[0091]

[0092] Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

[0093] Exemplarily, the feature extraction module includes:

[0094] The second construction unit is used to construct the basic features of infrared light images and visible light images expression:

[0095]

[0096] Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation;

[0097] The third construction unit is used to construct high-order features of infrared light images and visible light images expression:

[0098]

[0099] The fourth construction unit is used to construct the feature F before the infrared image and the visible light image are fused out expression:

[0100]

[0101] Where ⊙ represents point-by-point multiplication.

[0102] For more specific working processes of the above modules, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0103] Example 3

[0104] This embodiment provides a computer device, including a processor and a memory; wherein the processor implements the steps of the image fusion method based on the visual state space model described in Embodiment 1 when executing the computer program stored in the memory.

[0105] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0106] Example 4

[0107] This embodiment provides a computer-readable storage medium for storing a computer program; when the computer program is executed by a processor, the steps of the image fusion method based on the visual state space model described in Embodiment 1 are implemented.

[0108] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0109] Example 5

[0110] This embodiment provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the steps of the image fusion method based on the visual state space model described in Embodiment 1 are implemented.

[0111] For more specific details of the above method, please refer to the corresponding contents disclosed in Example 1, which will not be repeated here.

[0112] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the systems, devices, storage media, and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0113] Those skilled in the art can clearly understand that the technology in the embodiments of the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention or some parts of the embodiments.

[0114] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0115] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0116] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0117] The present invention has been described in detail above in conjunction with specific implementations and exemplary examples, but these descriptions cannot be understood as limiting the present invention. Those skilled in the art understand that, without departing from the spirit and scope of the present invention, a variety of equivalent substitutions, modifications or improvements may be made to the technical solution of the present invention and its implementation methods, all of which fall within the scope of the present invention. The scope of protection of the present invention shall be subject to the attached claims.

Claims

1. An image fusion method based on a visual state space model, characterized in that: include: The infrared image and the visible light image are divided into blocks respectively to obtain a plurality of two-dimensional image blocks; Convert multiple two-dimensional image blocks into a one-dimensional sequence; Perform feature extraction on the one-dimensional sequence to obtain the features before the fusion of the infrared image and the visible light image; Input the features before fusion into multiple 3x3 convolution kernels to obtain the first fused image; The first fused image is input to a decoder including a plurality of visual state space modules stacked together, and a second fused image is output as a final fused image.

2. The image fusion method based on the visual state space model according to claim 1, characterized in that: The infrared image and the visible light image are divided into blocks respectively to obtain a plurality of two-dimensional image blocks, including: The infrared image and the visible light image are uniformly cropped to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

3. The image fusion method based on the visual state space model according to claim 1, characterized in that: The step of converting the plurality of two-dimensional image blocks into a one-dimensional sequence comprises: Construct a one-dimensional sequence T0 expression: Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

4. The image fusion method based on the visual state space model according to claim 3, characterized in that: The feature extraction of the one-dimensional sequence to obtain the features before the fusion of the infrared light image and the visible light image includes: Constructing basic features of infrared and visible light images expression: Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation; Constructing high-level features of infrared and visible light images expression: Construct the feature F before fusion of infrared image and visible light image out expression: Where ⊙ represents point-by-point multiplication.

5. An image fusion system based on a visual state space model, characterized in that: include: An image segmentation module is used to segment the infrared image and the visible light image into blocks to obtain a plurality of two-dimensional image blocks; An image conversion module, used for converting a plurality of two-dimensional image blocks into a one-dimensional sequence; A feature extraction module is used to extract features from the one-dimensional sequence to obtain features before the infrared image and the visible light image are fused; A first fused image generation module, used for inputting the features before fusion into a plurality of 3x3 convolution kernels to obtain a first fused image; The second fused image generating module is used to input the first fused image into a decoder including a plurality of visual state space modules stacked together, and output a second fused image as a final fused image.

6. The image fusion system based on the visual state space model according to claim 5, characterized in that: The image conversion module comprises: The image cropping unit is used to perform uniform cropping operations on the infrared image and the visible light image respectively to obtain a plurality of infrared image blocks and a plurality of visible light image blocks; wherein the plurality of infrared image blocks and the plurality of visible light image blocks are all square image blocks; the total number of infrared image blocks or visible light image blocks is H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image; and P is the side length of the infrared image block or the visible light image block.

7. The image fusion system based on the visual state space model according to claim 5, characterized in that: The image conversion module comprises: The first building block is used to construct a one-dimensional sequence T0 expression: Among them, x cls It is a category token, which is added to the front of the image block sequence during input and outputs the category prediction in the classification task; is the Nth image block in the infrared image; is the Nth image block in the visible light image; N is the total number of infrared image blocks or visible light image blocks; W IR is the linear projection parameter matrix of the infrared image block; W VIS is the linear projection parameter matrix of the visible light image block; E pos is the position embedding, which is used to encode the position information for each image block.

8. The image fusion system based on the visual state space model according to claim 7, characterized in that: The feature extraction module comprises: The second construction unit is used to construct the basic features of infrared light images and visible light images expression: Among them, LN(·) represents LayerNorm operation; SS2D(·) represents multi-directional selective scanning operation; SiLU(·) represents nonlinear activation function; DWConv(·) represents depthwise separable convolution operation; Linear(·) represents linear layer operation; The third construction unit is used to construct high-order features of infrared light images and visible light images expression: The fourth construction unit is used to construct the feature F before the infrared image and the visible light image are fused out expression: Where ⊙ represents point-by-point multiplication.

9. A computer device, characterized in that: The method comprises a processor and a memory; wherein, when the processor executes the computer program stored in the memory, the steps of the image fusion method based on the visual state space model described in any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium, characterized in that: Used to store computer programs; when the computer programs are executed by the processor, the steps of the image fusion method based on the visual state space model described in any one of claims 1 to 4 are implemented.

Images