Multi-modal image fusion method and model based on global and local text perception

Abstract

The invention belongs to the technical field of image processing, and discloses a multi-modal image fusion method and model based on global and local text perception, which are used for respectively processing global and local information of images by combining two visual language models, namely CLIP and BLIP, and realizing effective fusion of the images under complex scenes and severe weather conditions. The global text sensing module utilizes CLIP features to enhance the understanding of the model for the whole scene, and the local text sensing module utilizes BLIP features to improve the processing capability for local details. According to the double text perception mechanism, the model can more comprehensively utilize the advantages of the visual language model, and the problem that only simple text prompt is relied on or local details are excessively emphasized is avoided. The advantages of a visual language model can be fully utilized, global and local information processing is considered, and the adaptability and generalization performance of the model under complex scenes and severe weather conditions are improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal image fusion method and model based on global and local text perception. Background Art

[0002] The purpose of infrared image fusion with visible light image is to utilize information from different sensors or data sources to generate more expressive, accurate and robust image representation. This technology is widely used in tasks such as object detection, semantic segmentation and depth estimation.

[0003] In recent years, with the development of large language models, the guidance of text semantic information has been able to effectively promote the establishment of visual models. Text usually has a clear semantic structure and can reveal important information in the image with clear descriptions. Introducing text descriptions in the establishment of visual models is conducive to helping the model associate with the information and status of objects in the scene, and improving the model's ability to capture information in image processing.

[0004] At present, some studies have explored the introduction of visual language models into the field of image fusion to improve its performance in practical applications, but there are still some unresolved issues. In the prior art, some methods only use simple text as degradation cues for embedding, which fails to give full play to the advantages of visual language models. In complex scenes, the degradation not only involves global information, but also includes many local details, such as color changes, motion states, and blur levels of objects. These details are usually not fully reflected in simplified text descriptions (such as subtitles). Therefore, only considering the visual language model as a control condition for adjusting the weights of image features may lead to a waste of computing resources.

[0005] Other existing techniques use multiple task labels in the source image to generate detailed scene description text embeddings. This approach may cause the model to overemphasize intricate local details, thereby reducing the model's ability to adapt to global patterns and affecting generalization performance. During training, embedding rich text information can greatly enhance the impact of text on feature extraction. However, due to the complexity of natural images, detailed descriptions of individual images often do not generalize well to other images.

[0006] In addition, current multimodal image fusion models based on visual language models have difficulty adapting to severe weather conditions. Severe weather can bring about multiple attenuations, affecting visible light images and causing a significant decrease in contrast in infrared images. Existing multimodal image fusion models based on visual language models lack the ability to handle multiple extreme degradation conditions simultaneously. Achieving effective processing of uniform weights under various severe weather conditions remains a challenge, especially in practical applications.

[0007] In view of the above problems, the existing technology needs to be improved urgently. Summary of the invention

[0008] The purpose of this application is to provide a multimodal image fusion method and model based on global and local text perception, which can fully utilize the advantages of visual language models, take into account global and local information processing, and improve the adaptability and generalization performance of the model in complex scenes and severe weather conditions.

[0009] In a first aspect, the present application provides a multimodal image fusion model based on global and local text perception, comprising: Input layer, global text perception module, CLIP image encoder, CLIP text encoder, BLIP text encoder, feature extraction network, decoder and output layer; The input layer is used to input a source image and input the source image into the global text perception module, the CLIP image encoder, the CLIP text encoder and the BLIP text encoder respectively; the source image includes an infrared source image and a visible light source image that are registered with each other; The CLIP image encoder is used to generate CLIP image features of the source image and input them into the global text perception module and the feature extraction network respectively; the CLIP text encoder is used to generate CLIP text features of the source image and input them into the global text perception module; the BLIP text encoder is used to generate BLIP text features of the source image and input them into the feature extraction network; The global text perception module is used to fuse the source images and integrate the CLIP image features into the fusion results according to the CLIP text features to obtain a preliminary fusion result to enrich the global information of the preliminary fusion result, and input the preliminary fusion result into the feature extraction network; The feature extraction network is embedded with at least one local text perception module, and the feature extraction network is used to extract features from the preliminary fusion result, and in the feature extraction process, the CLIP image features are integrated into the feature extraction result according to the BLIP text features to enrich the local information of the feature extraction result, and the feature extraction result is input into the decoder; The decoder is used to decode the feature extraction result to generate a final fused image, and output the final fused image through the output layer.

[0010] The model combines the CLIP and BLIP visual language models to process the global and local information of the image respectively, thus achieving effective fusion of images in complex scenes and severe weather conditions. The global text perception module uses CLIP features to enhance the model's understanding of the overall scene, while the local text perception module uses BLIP features to improve the ability to process local details. This dual text perception mechanism enables the model to more fully utilize the advantages of the visual language model, avoiding the problem of relying solely on simple text cues or over-emphasizing local details. It can fully utilize the advantages of the visual language model while taking into account global and local information processing, improving the model's adaptability and generalization performance in complex scenes and severe weather conditions.

[0011] Preferably, the global text perception module includes a first convolutional layer, a first separation layer, a first maximum pooling layer, a first residual module, a first cross attention module, a second convolutional layer, a second separation layer, a second maximum pooling layer, a second residual module and a second cross attention module; The first convolution layer is used to input the visible light source image and output the visible light feature map to the first separation layer, and the first separation layer is used to divide the visible light feature map into two parts; the second convolution layer is used to input the infrared source image and output the infrared feature map to the second separation layer, and the second separation layer is used to divide the infrared feature map into two parts; the visible light feature map and the corresponding part of the infrared feature map are respectively subjected to the maximum pooling processing of the first maximum pooling layer and the second maximum pooling layer, and then channel-connected and input into the first residual module; the visible light feature map and the corresponding other part of the infrared feature map are channel-connected and input into the second residual module; The output feature of the first residual module is added to the CLIP image feature and then input into the first cross attention module, and the output feature of the second residual module is added to the CLIP image feature and then input into the second cross attention module; The first cross-attention module uses the CLIP text feature as a query vector and uses the image feature input to the first cross-attention module as a key vector and a value vector to perform a cross-attention calculation to obtain a first fused feature; the second cross-attention module uses the CLIP text feature as a query vector and uses the image feature input to the second cross-attention module as a key vector and a value vector to perform a cross-attention calculation to obtain a second fused feature; the first fused feature and the second fused feature are added to obtain the preliminary fusion result.

[0012] This global text perception module achieves effective fusion of source images through the collaboration of multiple sub-modules, and integrates CLIP text features and CLIP image features into the fusion results. This design makes full use of text information to guide the fusion of image features through multi-way parallel processing and text-guided attention mechanism, improving the quality and semantic relevance of the fusion results.

[0013] Preferably, the first residual module and the second residual module each include a plurality of first extraction layers and a second extraction layer, the plurality of first extraction layers are serially connected in sequence, the input end of the first first extraction layer is connected to the input end of the corresponding residual module, the output feature of the last first extraction layer is added to the input of the corresponding residual module and then input to the second extraction layer, and the output end of the second extraction layer is connected to the output end of the corresponding residual module; The first extraction layer includes a third convolution layer and a first ReLU activation function layer, and the second extraction layer includes a fourth convolution layer and a second ReLU activation function layer.

[0014] This design can effectively improve the feature extraction capability while maintaining the trainability of the network. The series connection of multiple first extraction layers increases the network depth, while the residual connection facilitates the effective transmission of information. The second extraction layer, as the last layer, can make the final adjustment to the features. This structural design improves the expressiveness and performance of the model while maintaining computational efficiency.

[0015] Preferably, the feature extraction network includes multiple layers of residual space state modules, wherein a local text perception module is embedded after every N layers of residual space state modules, and N is a positive integer; the CLIP image encoder inputs the CLIP image features into each of the local text perception modules; and the BLIP text encoder inputs the BLIP text features into each of the local text perception modules.

[0016] Preferably, the residual space state module comprises a first regularization layer, a visual state space module, a first scaling layer, a second regularization layer, a seventh convolutional layer, a channel attention module and a second scaling layer; The first regularization layer and the visual state space module are connected in sequence, and the second regularization layer, the seventh convolutional layer and the channel attention module are connected in sequence; the input end of the first regularization layer and the input end of the first scaling layer are both connected to the input end of the residual space state module, and the output of the visual state space module and the output of the first scaling layer are added and input into the second regularization layer and the second scaling layer; the output of the channel attention module and the output of the second scaling layer are added and output from the output end of the residual space state module.

[0017] Preferably, the visual state space module comprises a first linear layer, a depth-separable convolutional layer, a first Swish activation function layer, a 2D selective scanning module, a layer normalization module, a second linear layer, a second Swish activation function layer and a third linear layer; The first linear layer, the depthwise separable convolutional layer, the first Swish activation function layer, the 2D selective scanning module and the layer normalization module are connected in sequence, and the second linear layer and the second Swish activation function layer are connected in sequence; the input end of the first linear layer and the input end of the second linear layer are both connected to the input end of the visual state space module, the output of the layer normalization module and the output of the second Swish activation function layer are multiplied and input into the third linear layer, and the output end of the third linear layer is connected to the output end of the visual state space module.

[0018] Preferably, the local text perception module includes several layers of squeeze excitation modules, an embedding module, a third cross attention module, several hole convolution layers and a fifth convolution layer; Several layers of the squeeze excitation module, the embedding module and the third cross attention module are connected in sequence, and the input end of the squeeze excitation module of the first layer is connected to the input end of the local text perception module; The squeeze excitation module is used to enhance important features and suppress redundant information of image features input to the squeeze excitation module through an adaptive channel attention mechanism; The embedding module is used to map the CLIP image features to the semantic space of the image features input to the embedding module through a multi-layer perceptron and then embed them into the image features input to the embedding module; The third cross attention module uses the BLIP text feature as a query vector and the image feature input to the third cross attention module as a key vector and a value vector to perform cross attention calculation, and inputs the calculation results into each of the hole convolution layers respectively; The expansion rates of the dilated convolutional layers are different, and the outputs of all the dilated convolutional layers are channel-connected and then input into the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the output end of the local text perception module.

[0019] In a second aspect, the present application provides a multimodal image fusion method based on global and local text perception, comprising the steps of: A1. constructing an initial multimodal image fusion model based on global and local text perception; the initial multimodal image fusion model based on global and local text perception is the multimodal image fusion model based on global and local text perception described above; A2. training the initial multimodal image fusion model based on global and local text perception to obtain a trained multimodal image fusion model based on global and local text perception; A3. Input the source image to be fused into the trained multimodal image fusion model based on global and local text perception to obtain a fused image output by the trained multimodal image fusion model based on global and local text perception.

[0020] Preferably, step A2 comprises: A201. Acquire multiple training samples; the training samples include sample source images and corresponding clean source images; the sample source images include sample infrared source images and sample visible light source images, the clean source images include clean infrared source images and clean visible light source images, the sample source images include degradation information, the clean source images do not include degradation information, and the degradation information includes weather interference information; A202. Select the training sample, input the corresponding sample source image into the initial multimodal image fusion model based on global and local text perception, and obtain the fused image output by the initial multimodal image fusion model based on global and local text perception, which is recorded as the first fused image; A203. Input the first fused image into the CLIP image encoder to obtain a CLIP image feature corresponding to the first fused image, recorded as a first CLIP image feature; A204. Input the clean source image of the selected training sample into the CLIP text encoder to obtain a CLIP text feature corresponding to the clean source image, recorded as a first CLIP text feature; A205. Align the first CLIP text feature with the first CLIP image feature in the same feature space; A206. Calculate a loss function according to the first fused image, the aligned first CLIP text features and the clean source image of the selected training sample; A207. Optimizing the model parameters of the initial multimodal image fusion model based on global and local text perception according to the loss function; A208. Repeat steps A202 to A207 until the loss function is less than a preset loss function threshold or the number of iterations reaches a preset number threshold.

[0021] Preferably, the loss function is: ; ; ; in, is the loss function value, is the pixel-level loss, For drive loss, is the first fused image, is the first CLIP text feature after the alignment, represents the norm, is the clean infrared source image of the selected training sample, is the clean visible light source image of the selected training sample, Represents the 1-norm.

[0022] Beneficial effects: The multimodal image fusion method and model based on global and local text perception provided by the present application, by combining the two visual language models of CLIP and BLIP, respectively processes the global and local information of the image, thereby realizing the effective fusion of images in complex scenes and severe weather conditions. The global text perception module uses CLIP features to enhance the model's understanding of the overall scene, while the local text perception module uses BLIP features to improve the ability to process local details. This dual text perception mechanism enables the model to more fully utilize the advantages of the visual language model, avoiding the problem of relying solely on simple text prompts or over-emphasizing local details. It can fully utilize the advantages of the visual language model, while taking into account global and local information processing, and improve the adaptability and generalization performance of the model in complex scenes and severe weather conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of the structure of a multimodal image fusion model based on global and local text perception provided in an embodiment of the present application.

[0024] Figure 2 Schematic diagram of the structure of the global text perception module.

[0025] Figure 3 Schematic diagram of the structure of the first residual module and the second residual module.

[0026] Figure 4 Schematic diagram of the feature extraction network.

[0027] Figure 5 Schematic diagram of the structure of the residual space state module.

[0028] Figure 6 Schematic diagram of the structure of the visual state space module.

[0029] Figure 7 Schematic diagram of the structure of the local text perception module.

[0030] Figure 8 A schematic diagram of the decoder structure.

[0031] Fig. 9 A flowchart of a multimodal image fusion method based on global and local text perception provided in an embodiment of the present application.

[0032] Fig.10 This is a comparison chart of the first set of experimental results.

[0033] Fig.11 This is a comparison chart of the second set of experimental results.

[0034] Fig.12 This is a comparison chart of the third set of experimental results.

[0035] Fig.13 This is a comparison chart of the fourth set of experimental results.

[0036] Fig.14 This is a comparison chart of the fifth set of experimental results.

[0037] Fig.15 This is a comparison chart of the sixth set of experimental results.

[0038] Description of the reference numerals: 1. Input layer; 2. Global text perception module; 201. First convolutional layer; 202. First separation layer; 203. First maximum pooling layer; 204. First residual module; 205. First cross attention module; 206. Second convolutional layer; 207. Second separation layer; 208. Second maximum pooling layer; 209. Second residual module; 210. Second cross attention module; 3. CLIP image encoder; 4. CLIP text encoder; 5. BLIP text encoder; 6. Feature extraction network; 7. Decoder; 701. Decoding module; 702. Wavelet convolution layer; 703. Third ReLU activation function layer; 704. Sixth convolution layer; 8. Output layer; 9. Local text perception module; 901. Squeeze excitation module; 902. Embedding module; 903. Third cross attention module; 904. Hollow convolution layer; 905. Fifth convolution layer; 906. SE attention module; 10. First extraction layer; 1001. Third convolution layer; 1002. First ReLU activation function layer; 11. Second extraction layer; 1101. Fourth convolution layer; 1102. Second ReLU activation function layer; 12. Residual space state module; 1201. First regularization layer; 1202. Visual state space module; 1203. First scaling layer; 1204. Second regularization layer; 1205. Seventh convolutional layer; 1206. Channel attention module; 1207. Second scaling layer; 13. First linear layer; 14. Depth-wise separable convolutional layer; 15. First Swish activation function layer; 16. 2D selective scanning module; 17. Layer normalization module; 18. Second linear layer; 19. Second Swish activation function layer; 20. Third linear layer. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0040] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0041] In the field of multimodal image fusion, especially in the fusion of infrared and visible light images, using visual language models to improve the fusion effect has become an important research direction. However, existing multimodal image fusion technologies based on visual language models still face significant challenges in processing global and local information and adapting to severe weather conditions. Specifically, these technologies are often oversimplified or overly complicated when using text information, resulting in the inability to fully utilize the advantages of visual language models. Among them, simple text embedding methods are difficult to capture detailed information in complex scenes, while overly detailed scene descriptions may cause the model to focus too much on local features, affecting its generalization ability. In addition, under severe weather conditions, existing models find it difficult to handle multiple extreme degradation situations at the same time, which severely limits their performance in practical applications.

[0042] To do this, please refer to Figure 1-Figure 8 , the present application provides a multimodal image fusion model based on global and local text perception, including: Input layer 1, global text perception module 2, CLIP image encoder 3 (CLIP stands for Contrastive Language-Image Pre-Training, language image pre-training based on contrastive learning), CLIP text encoder 4, BLIP text encoder 5 (BLIP stands for Bootstrapping Language-Image Pre-training, guided language image pre-training), feature extraction network 6, decoder 7 and output layer 8; The input layer 1 is used to input the source image and input the source image into the global text perception module 2, the CLIP image encoder 3, the CLIP text encoder 4 and the BLIP text encoder 5 respectively; the source image includes an infrared source image and a visible light source image that are registered with each other; The CLIP image encoder 3 is used to generate the CLIP image features of the source image (O1 in the figure represents the CLIP image features) and input them into the global text perception module 2 and the feature extraction network 6 respectively; the CLIP text encoder 4 is used to generate the CLIP text features of the source image (O2 in the figure represents the CLIP text features) and input them into the global text perception module 2; the BLIP text encoder 5 is used to generate the BLIP text features of the source image (O3 in the figure represents the BLIP text features) and input them into the feature extraction network 6; The global text perception module 2 is used to fuse the source images and integrate the CLIP image features (i.e., O1) into the fusion results according to the CLIP text features (i.e., O2), obtain a preliminary fusion result, enrich the global information of the preliminary fusion result, and input the preliminary fusion result into the feature extraction network 6; The feature extraction network 6 is embedded with at least one local text perception module 9, and the feature extraction network 6 is used to extract features from the preliminary fusion result, and in the feature extraction process, the CLIP image feature (i.e., O1) is integrated into the feature extraction result according to the BLIP text feature (i.e., O3) to enrich the local information of the feature extraction result, and the feature extraction result is input into the decoder 7; The decoder 7 is used to decode the feature extraction result to generate a final fused image, and output the final fused image through the output layer 8.

[0043] The model combines the two visual language models, CLIP and BLIP, to process the global and local information of the image respectively, thus achieving effective fusion of images in complex scenes and severe weather conditions. The global text perception module 2 uses CLIP features to enhance the model's understanding of the overall scene, while the local text perception module 9 uses BLIP features to improve the ability to process local details. This dual text perception mechanism enables the model to more fully utilize the advantages of the visual language model, avoiding the problem of relying solely on simple text prompts or over-emphasizing local details. It can fully utilize the advantages of the visual language model while taking into account global and local information processing, improving the adaptability and generalization performance of the model in complex scenes and severe weather conditions.

[0044] Therefore, the input source image may be a source image containing degradation information, wherein the degradation information includes weather interference information, such as at least one of rain, snow, fog, and the like.

[0045] Among them, the CLIP image encoder 3, the CLIP text encoder 4 and the BLIP text encoder 5 can all adopt existing technologies, which will not be described in detail here.

[0046] Specifically, see Figure 2 , the global text perception module 2 includes a first convolution layer 201, a first separation layer 202, a first maximum pooling layer 203, a first residual module 204, a first cross attention module 205, a second convolution layer 206, a second separation layer 207, a second maximum pooling layer 208, a second residual module 209 and a second cross attention module 210; The first convolutional layer 201 is used to input a visible light source image (i.e., Iv in the figure) and output a visible light feature map to the first separation layer 202, and the first separation layer 202 is used to divide the visible light feature map into two parts (such as the upper half and the lower half, or the left half and the right half); the second convolutional layer 206 is used to input an infrared source image (i.e., Ir in the figure) and output an infrared feature map to the second separation layer 207, and the second separation layer 207 is used to divide the infrared feature map into two parts (correspondingly, the upper half and the lower half, or the left half and the right half). The corresponding parts of the visible light feature map and the infrared feature map are respectively subjected to the maximum pooling processing of the first maximum pooling layer 203 and the second maximum pooling layer 208 (for example, the left half of the visible light feature map is subjected to the maximum pooling processing of the first maximum pooling layer 203, and the left half of the infrared feature map is subjected to the maximum pooling processing of the second maximum pooling layer 208), and then channel-connected and input into the first residual module 204; the other corresponding parts of the visible light feature map and the infrared feature map (for example, the right half) are channel-connected and input into the second residual module 209; The output feature of the first residual module 204 is added to the CLIP image feature (i.e., O1) and then input into the first cross attention module 205, and the output feature of the second residual module 209 is added to the CLIP image feature (i.e., O1) and then input into the second cross attention module 210; The first cross-attention module 205 uses the CLIP text feature (i.e., O2) as a query vector and uses the image feature input to the first cross-attention module 205 as a key vector and a value vector to perform a cross-attention calculation to obtain a first fused feature; the second cross-attention module 210 uses the CLIP text feature (i.e., O2) as a query vector and uses the image feature input to the second cross-attention module 210 as a key vector and a value vector to perform a cross-attention calculation to obtain a second fused feature; the first fused feature and the second fused feature are added to obtain a preliminary fusion result.

[0047] This global text perception module 2 achieves effective fusion of source images through the collaboration of multiple sub-modules, and integrates CLIP text features and CLIP image features into the fusion results. This design makes full use of text information to guide the fusion of image features through multi-channel parallel processing and text-guided attention mechanism, improving the quality and semantic relevance of the fusion results.

[0048] The first convolution layer 201 and the second convolution layer 206 process the visible light source image and the infrared source image respectively to extract preliminary features. The two convolution layers can use 1×1 convolution kernels to expand the feature channels of the source images.

[0049] The first separation layer 202 and the second separation layer 207 divide their respective feature maps into two parts to prepare for subsequent parallel processing. The separation operation can be achieved through channel segmentation or spatial segmentation. Channel segmentation can divide the feature map into two parts according to the number of channels, while spatial segmentation can be divided according to the spatial position of the feature map, such as up and down or left and right segmentation.

[0050] The first maximum pooling layer 203 and the second maximum pooling layer 208 reduce the dimension of a part of the feature map to reduce the amount of calculation. The pooling operation can not only reduce the spatial dimension of the feature map, but also improve the robustness of the model to small image transformations.

[0051] The channel connection operation fuses the visible light and infrared features. This step can be achieved by simple channel splicing or by weighted summation, where weights are assigned according to the importance of different modal features.

[0052] The first residual module 204 processes the pooled features, and the second residual module 20 processes the unpooled features, which helps to retain information of different scales, enhance the representativeness of the features and minimize feature loss. The design of the residual module can include multiple convolutional layers and activation functions, and alleviate the gradient vanishing problem through short-circuit connections, improve the effect of feature extraction, and improve the robustness of features.

[0053] For example, the first residual module 204 and the second residual module 209 may adopt Figure 3 The residual module shown includes a plurality of first extraction layers 10 and a second extraction layer 11, wherein the plurality of first extraction layers 10 are connected in series, the input end of the first first extraction layer 10 is connected to the input end of the corresponding residual module (the first residual module 204 or the second residual module 209), the output feature of the last first extraction layer 10 is added to the input of the corresponding residual module and then input to the second extraction layer 11, and the output end of the second extraction layer 11 is connected to the output end of the corresponding residual module; The first extraction layer 10 includes a third convolution layer 1001 and a first ReLU activation function layer 1002 , and the second extraction layer 11 includes a fourth convolution layer 1101 and a second ReLU activation function layer 1102 .

[0054] This design can effectively improve the feature extraction capability while maintaining the trainability of the network. The series connection of multiple first extraction layers 10 increases the network depth, while the residual connection facilitates the effective transmission of information. As the last layer, the second extraction layer 11 can make final adjustments to the features. This structural design improves the expressiveness and performance of the model while maintaining computational efficiency. Among them, the number of first extraction layers 10 and the convolution kernel sizes of the third convolution layer 1001 and the fourth convolution layer 1101 can be set according to actual needs. For example, the number of first extraction layers 10 is 2, the convolution kernel size of the third convolution layer 1001 is 3×3, and the convolution kernel size of the fourth convolution layer 1101 is 1×1, but is not limited to this.

[0055] Among them, the CLIP image features are added to the output of the residual module to ensure the consistency of multimodal information.

[0056] The first cross-attention module 205 and the second cross-attention module 210 use CLIP text features as query vectors to integrate semantic information, achieve text-guided feature fusion, and improve model understanding. The cross-attention calculation can use a standard scaled dot product attention mechanism or introduce a multi-head attention mechanism to capture attention information in different subspaces.

[0057] The cross-attention calculation performed by the first cross-attention module 205 can be expressed as: ; in, is the cross attention function, To input the image features of the first cross-attention module 205, For the corresponding The key vector of To correspond to The value vector of for The dimension of is the softmax function, is the query vector corresponding to O2.

[0058] Similarly, the cross-attention calculation performed by the second cross-attention module 210 can be expressed as: ; in, To input the image features of the second cross-attention module 210, To correspond to The key vector of For the corresponding The value vector of for Dimension.

[0059] Through this module design and feature fusion process, the global text perception module 2 can effectively solve the technical problem of how to integrate CLIP text features and CLIP image features into the fusion results. This method not only considers the visual features of the image, but also introduces text information as global semantic guidance, thus achieving more intelligent and semantically aware image fusion.

[0060] Further, see Figure 4 The feature extraction network 6 includes multiple layers of residual space state modules 12, wherein a local text perception module 9 is embedded after every N layers of residual space state modules 12, where N is a positive integer; the CLIP image encoder 3 inputs the CLIP image feature (i.e., O1) into each local text perception module 9; and the BLIP text encoder 5 inputs the BLIP text feature (i.e., O3) into each local text perception module 9.

[0061] The feature extraction network 6 adopts a structure of multi-layer residual space state modules 12, which has good feature extraction capabilities. By embedding local text perception modules 9 in this structure, the present application realizes multi-scale and multi-level text guidance of image features. Specifically, a local text perception module 9 is embedded in every N layers of residual space state modules 12, where N is a positive integer. This layout method is flexible, and the value of N can be adjusted according to actual needs to balance the efficiency of feature extraction and the frequency of text perception. For example, when N=2, a local text perception module 9 will be embedded after every two layers of residual space state modules 12. This configuration can frequently introduce text information while maintaining a high feature extraction efficiency. When N=3 or greater, the distribution of local text perception modules 9 will be more sparse, which may be suitable for scenarios with limited computing resources or high real-time requirements.

[0062] The CLIP image encoder 3 and the BLIP text encoder 5 provide different types of features to the local text perception module 9 respectively. CLIP image features usually contain higher-level semantic information, while BLIP text features may contain more fine-grained text descriptions. By simultaneously inputting these two features into each local text perception module 9, the present application achieves a deep fusion of image and text information. This design enables the feature extraction network 6 to capture the semantic association between image and text at different levels of abstraction. Through this multi-level text perception mechanism, the feature extraction network 6 of the present application can better handle image fusion tasks in complex scenes and severe weather conditions.

[0063] In some embodiments, see Figure 5 , the residual space state module 12 includes a first regularization layer 1201, a visual state space module 1202, a first scaling layer 1203 (scaling layer is Scale layer), a second regularization layer 1204, a seventh convolutional layer 1205, a channel attention module 1206 and a second scaling layer 1207; The first regularization layer 1201 and the visual state space module 1202 are connected in sequence, and the second regularization layer 1204, the seventh convolutional layer 1205 and the channel attention module 1206 are connected in sequence; the input end of the first regularization layer 1201 and the input end of the first scaling layer 1203 are both connected to the input end of the residual space state module 12, and the output of the visual state space module 1202 and the output of the first scaling layer 1203 are added and input into the second regularization layer 1204 and the second scaling layer 1207; the output of the channel attention module 1206 and the output of the second scaling layer 1207 are added and output from the output end of the residual space state module 12.

[0064] The residual space state module 12 is designed to cleverly combine residual connection, regularization, attention mechanism and multi-scale feature fusion, effectively solving the problem of feature extraction and fusion. Residual connection helps to alleviate the gradient vanishing problem, allowing the network to learn deep features more effectively. Multi-scale feature fusion can capture image information of different scales and improve the model's ability to understand complex scenes. The introduction of the channel attention mechanism further enhances the model's attention to important features and improves the efficiency and accuracy of feature extraction.

[0065] This design can not only effectively extract and fuse image features, but also maintain good performance when processing complex scenes, especially when processing images containing multiple weather conditions. In this way, the module provides powerful feature extraction and fusion capabilities for the entire multimodal image fusion model, laying a solid foundation for subsequent image fusion tasks.

[0066] Among them, the first regularization layer 1201 and the second regularization layer 1204 help stabilize the network training process and improve the convergence speed and generalization ability of the model by standardizing the input data.

[0067] Among them, the visual state space module 1202 is responsible for extracting and processing complex visual features. This module can adopt a variety of advanced visual feature extraction technologies, such as self-attention mechanism, convolutional neural network or transformer structure.

[0068] For example, in some embodiments, see Figure 6 , the visual state space module 1202 includes a first linear layer 13, a depth-separable convolutional layer 14, a first Swish activation function layer 15 (Swish activation function layer is SiLU layer), a 2D selective scanning module 16 (i.e. 2D-SSM module), a layer normalization module 17, a second linear layer 18, a second Swish activation function layer 19 and a third linear layer 20; The first linear layer 13, the depthwise separable convolutional layer 14, the first Swish activation function layer 15, the 2D selective scanning module 16 and the layer normalization module 17 are connected in sequence, and the second linear layer 18 and the second Swish activation function layer 19 are connected in sequence; the input end of the first linear layer 13 and the input end of the second linear layer 18 are both connected to the input end of the visual state space module 1202, the output of the layer normalization module 17 and the output of the second Swish activation function layer 19 are multiplied and input into the third linear layer 20, and the output end of the third linear layer 20 is connected to the output end of the visual state space module 1202.

[0069] The visual state space module 1202 can effectively extract and integrate local and global features of the image through multi-level feature extraction and nonlinear transformation. The use of deep separable convolution and 2D selective scanning improves the model's perception of spatial information, while the multiple activation functions and normalization operations enhance the model's nonlinear expression ability and training stability. This structural design helps to improve the quality of feature extraction and the overall performance of the model while maintaining computational efficiency.

[0070] The first scaling layer 1203 and the second scaling layer 1207 adjust the scale of features to achieve effective fusion of features at different levels. These scaling operations can adaptively adjust the importance of features through learnable parameters.

[0071] Among them, the seventh convolution layer 1205 further extracts and transforms features to enhance the expression ability of the model. This layer can use a 3×3 convolution kernel.

[0072] The channel attention module 1206 highlights important features and suppresses irrelevant information by assigning different weights to different channels. This can be achieved through technologies such as Squeeze-and-Excitation (SE) blocks or ECA (Efficient Channel Attention).

[0073] Further, see Figure 7 , the local text perception module 9 includes several layers of squeeze excitation modules 901, an embedding module 902, a third cross attention module 903, several hole convolution layers 904 and a fifth convolution layer 905; Several layers of squeeze excitation modules 901, embedding modules 902 and third cross attention modules 903 are connected in sequence, and the input end of the first layer squeeze excitation module 901 is connected to the input end of the local text perception module 9; The squeeze excitation module 901 is used to enhance important features and suppress redundant information of image features input to the squeeze excitation module 901 through an adaptive channel attention mechanism; The embedding module 902 is used to map the CLIP image feature (i.e., O1) to the semantic space of the image feature of the input embedding module 902 through a multi-layer perceptron and then embed it into the image feature of the input embedding module 902; The third cross attention module 903 uses the BLIP text feature (i.e., O3) as a query vector and uses the image feature input to the third cross attention module 903 as a key vector and a value vector to perform cross attention calculation, and inputs the calculation results into each hole convolution layer 904 respectively; The expansion rates of the various atrous convolutional layers 904 are different, and the outputs of all atrous convolutional layers 904 are channel-connected and then input into the fifth convolutional layer 905; the output end of the fifth convolutional layer 905 is connected to the output end of the local text perception module 9.

[0074] As the number of network layers in the feature extraction network 6 increases, the complexity and dimension of the feature map will continue to increase, which makes it increasingly important to effectively focus on the feature responses of different channels. For this reason, a squeeze excitation module 901 is introduced in the local text perception module 9 to process the fused features. By explicitly modeling the dependencies between channels, the squeeze excitation module 901 can strengthen important features and suppress redundant information through an adaptive channel attention mechanism, thereby ensuring that the input features have good expressiveness at the local level.

[0075] As a preferred embodiment, see Figure 7 The squeeze excitation module 901 includes an SE attention module 906, the input end of the SE attention module 906 is connected to the input end of the squeeze excitation module 901, and the input of the squeeze excitation module 901 is multiplied by the output of the SE attention module 906 and then output from the output end of the squeeze excitation module 901. This design can further enhance the module's ability to identify important features and improve the efficiency of feature extraction.

[0076] The number of layers of the extrusion excitation module 901 can be set according to actual needs, for example, two layers are set, but it is not limited thereto.

[0077] Among them, the embedding module 902 uses a multi-layer perceptron to map the CLIP image features to the semantic space of the input image features. This step realizes the semantic alignment and fusion of features, so that the global image information can be effectively embedded in the fused features. This design helps to enhance the semantic expression ability of features and provide a richer information basis for subsequent processing.

[0078] Specifically, the embedding module 902 performs a mapping operation on the CLIP image features through a multi-layer perceptron to obtain scale adjustment parameters and deviation control parameters to perform the following operations: ; in, is the output feature of the embedding module 902, is the scale adjustment parameter, is the deviation control parameter, is the image feature input to the embedding module 902, Represents the element-by-element multiplication operation. This design allows the module to dynamically adjust the fusion process according to the characteristics of the input features, improving the flexibility and adaptability of feature fusion.

[0079] Among them, the third cross-attention module 903 uses the BLIP text feature as the query vector and the input image feature as the key vector and value vector to perform cross-attention calculation. This step realizes text-guided feature enhancement and further integrates multimodal information from vision and text. In this way, the semantic richness of the fused feature is improved, including more comprehensive scene and detail information.

[0080] The cross-attention calculation performed by the third cross-attention module 903 can be expressed as: ; in, To input the image features of the third cross-attention module 903, For the corresponding The key vector of For the corresponding The value vector of for The dimension of is the query vector corresponding to O3.

[0081] The calculation results of the cross-attention calculation are input into several dilated convolution layers 904. These dilated convolution layers 904 have different expansion rates and can capture multi-scale contextual information. By using dilated convolutions with different expansion rates, the module can sparsely sample image information at a specific step size, effectively removing redundant information while retaining key details. This design significantly enhances the expressiveness of features, allowing the model to better understand the local structure and global semantics of the image.

[0082] The number of atrous convolution layers 904 and the expansion rate of each atrous convolution layer 904 can be set according to actual needs. For example, there are three atrous convolution layers 904, and the expansion rates of the three atrous convolution layers 904 are 1, 2, and 3 respectively; but the present invention is not limited thereto.

[0083] Among them, the outputs of all the hole convolution layers 904 are connected by channels and then input into the fifth convolution layer 905. The fifth convolution layer 905 finally integrates the features extracted by the previous components to generate the output features of the local text perception module 9. This step ensures that the fused features have stronger recognition ability and robustness. The convolution kernel size of the fifth convolution layer 905 can be set to 1×1.

[0084] Through this design, the local text perception module 9 can achieve text-guided feature enhancement and fusion in local areas. The module makes full use of text information to enhance image features, while improving the expressiveness and robustness of features through multi-scale feature extraction and adaptive attention mechanism. This method can not only capture the local detail information of the image, but also effectively integrate text semantic information into the feature representation, thereby providing richer and more accurate feature input for subsequent image fusion tasks.

[0085] Specifically, see Figure 8 The decoder 7 includes a plurality of decoding modules 701 connected in series, and the decoding module 701 includes a wavelet convolution layer 702, a third ReLU activation function layer 703 and a sixth convolution layer 704 connected in series; the input end of the first decoding module 701 is connected to the input end of the decoder 7, and the output end of the last decoding module 701 is connected to the output end of the decoder 7.

[0086] The wavelet convolution layer 702 can effectively extract multi-scale features, the third ReLU activation function layer 703 introduces nonlinearity, and the sixth convolution layer 704 performs feature fusion. This combination can gradually restore image details while maintaining the grasp of global information. Among them, the number of decoding modules 701 can be set according to actual needs, for example, 3, but not limited to this. Among them, the sixth convolution layer 704 can use a 3×3 convolution kernel.

[0087] refer to Fig. 9 , the present application provides a multimodal image fusion method based on global and local text perception, comprising the steps of: A1. Construct an initial multimodal image fusion model based on global and local text perception; the initial multimodal image fusion model based on global and local text perception is the multimodal image fusion model based on global and local text perception mentioned above; A2. training the initial multimodal image fusion model based on global and local text perception to obtain a trained multimodal image fusion model based on global and local text perception; A3. Input the source images to be fused (including the mutually registered infrared source images to be fused and the light source images to be fused) into the trained multimodal image fusion model based on global and local text perception, and obtain the fused image output by the trained multimodal image fusion model based on global and local text perception.

[0088] This method can more comprehensively utilize text information to guide the image fusion process by introducing global and local text perception mechanisms. Global text perception helps capture the overall semantic information of the image, while local text perception can focus on the detailed features of the image. This dual perception mechanism enables the fusion process to better balance global and local information, thereby generating higher quality fused images.

[0089] Preferably, step A2 comprises: A201. Acquire multiple training samples; the training samples include sample source images and corresponding clean source images; the sample source images include sample infrared source images and sample visible light source images, the clean source images include clean infrared source images and clean visible light source images, the sample source images include degradation information, the clean source images do not include degradation information, and the degradation information includes weather interference information (such as at least one of rain, snow, fog, etc.); A202. Select training samples, input the corresponding sample source images into the initial multimodal image fusion model based on global and local text perception, and obtain a fused image output by the initial multimodal image fusion model based on global and local text perception, which is recorded as the first fused image; A203. Input the first fused image into the CLIP image encoder 3 to obtain a CLIP image feature corresponding to the first fused image, recorded as the first CLIP image feature; A204. Input the clean source image of the selected training sample into the CLIP text encoder 4 to obtain the CLIP text feature corresponding to the clean source image, which is recorded as the first CLIP text feature; A205. Align the first CLIP text feature with the first CLIP image feature in the same feature space; A206. Calculate the loss function according to the first fused image, the aligned first CLIP text features and the clean source image of the selected training sample; A207. Optimizing model parameters of an initial multimodal image fusion model based on global and local text perception according to a loss function; A208. Repeat steps A202 to A207 until the loss function is less than a preset loss function threshold or the number of iterations reaches a preset number threshold.

[0090] It should be understood that the two images in the sample source image are registered with each other, and the clean source image is an ideal image obtained by removing degradation information from the sample source image.

[0091] In step A202, the training samples may be selected sequentially or randomly.

[0092] In step A205, the first CLIP text feature and the first CLIP image feature are aligned in the same feature space. This step ensures the comparability of the text feature and the image feature, laying a foundation for the subsequent loss calculation.

[0093] Preferably, the loss function is: ; ; ; in, is the loss function value, is the pixel-level loss, For drive loss, is the first fused image, is the first CLIP text feature after the alignment, represents the norm, is the clean infrared source image of the selected training sample, is the clean visible light source image of the selected training sample, Represents the 1-norm.

[0094] This loss function includes pixel-level loss and driving loss, which comprehensively considers the image reconstruction quality and the consistency of the feature space. It can ensure the enhancement of the semantic features of the fused image and improve the model's ability to express natural scenes.

[0095] Among them, in step A207, the model parameters of the initial multimodal image fusion model based on global and local text perception can be optimized using the Adam optimizer according to the loss function, the initial learning efficiency can be set to 0.0001, the number of iterations threshold can be set to 350, and the loss function threshold can be set to 0.01, but are not limited to this.

[0096] Figure 10-15 The fusion results of the source images containing different degradation information are shown in the present invention. Figure 10-15 In the figure, a represents the infrared source image, b represents the visible light source image, and c represents the fused image; Fig.10 and Fig.11 The degradation information contained in the source image is fog. Fig.12 and Fig.13 The degradation information contained in the source image is rain, Fig.14 and Fig.15The degradation information contained in the source image is snow. As can be seen from the figure, in various degradation scenes, the infrared source image generally has the problem of low contrast, which shows that the present invention not only needs to effectively deal with degradation when processing degraded images, but also needs to enhance the image contrast to highlight the details of significant objects. At the same time, it can be observed that degradation factors such as fog, rain, and snow pollute the visible light image to varying degrees and block most of the useful information. In contrast, the fusion results of the present invention show excellent performance: on the basis of effectively removing degraded pixels, it successfully realizes the deep interaction of multimodal features, fully captures the complementary information from different modalities, and significantly improves the contrast and clarity of the scene.

[0097] In summary, this application has at least the following advantages: (1) This paper proposes a novel multimodal image fusion method for severe weather conditions. This method utilizes global and local context perception within a unified architecture and can effectively handle multiple degradations simultaneously. (2) This paper proposes a novel global and local text perception module to guide feature extraction from both macro and micro perspectives, enhancing versatility while maintaining high-fidelity fusion; (3) This paper introduces a novel visual language model driven loss function, which uses the feature space of CLIP to guide the generation of fused images based on text information; this loss function comprehensively evaluates the image-text feature alignment through semantic features; (4) The present invention has been extensively tested under various severe weather conditions, demonstrating its excellent performance in coping with extreme environments and can be applied to many real-world application scenarios.

[0098] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0099] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0101] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0102] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A multimodal image fusion model based on global and local text perception, characterized in that: include: Input layer (1), global text perception module (2), CLIP image encoder (3), CLIP text encoder (4), BLIP text encoder (5), feature extraction network (6), decoder (7) and output layer (8); The input layer (1) is used to input a source image and input the source image into the global text perception module (2), the CLIP image encoder (3), the CLIP text encoder (4) and the BLIP text encoder (5) respectively; the source image includes an infrared source image and a visible light source image that are registered with each other; The CLIP image encoder (3) is used to generate CLIP image features of the source image and input them into the global text perception module (2) and the feature extraction network (6) respectively; the CLIP text encoder (4) is used to generate CLIP text features of the source image and input them into the global text perception module (2); the BLIP text encoder (5) is used to generate BLIP text features of the source image and input them into the feature extraction network (6); The global text perception module (2) is used to fuse the source images and integrate the CLIP image features into the fusion results according to the CLIP text features to obtain a preliminary fusion result, so as to enrich the global information of the preliminary fusion result, and input the preliminary fusion result into the feature extraction network (6); The feature extraction network (6) is embedded with at least one local text perception module (9), and the feature extraction network (6) is used to extract features from the preliminary fusion result, and in the feature extraction process, integrate the CLIP image features into the feature extraction result according to the BLIP text features to enrich the local information of the feature extraction result, and input the feature extraction result into the decoder (7); The decoder (7) is used to decode the feature extraction result to generate a final fused image, and output the final fused image through the output layer (8).

2. The multimodal image fusion model based on global and local text perception according to claim 1, characterized in that: The global text perception module (2) comprises a first convolution layer (201), a first separation layer (202), a first maximum pooling layer (203), a first residual module (204), a first cross attention module (205), a second convolution layer (206), a second separation layer (207), a second maximum pooling layer (208), a second residual module (209) and a second cross attention module (210); The first convolution layer (201) is used to input the visible light source image and output a visible light feature map to the first separation layer (202), and the first separation layer (202) is used to divide the visible light feature map into two parts; the second convolution layer (206) is used to input the infrared source image and output an infrared feature map to the second separation layer (207), and the second separation layer (207) is used to divide the infrared feature map into two parts; the visible light feature map and the corresponding part of the infrared feature map are respectively subjected to maximum pooling processing by the first maximum pooling layer (203) and the second maximum pooling layer (208), and then channel-connected and input into the first residual module (204); the visible light feature map and the corresponding other part of the infrared feature map are channel-connected and input into the second residual module (209); The output feature of the first residual module (204) is added to the CLIP image feature and then input into the first cross-attention module (205); the output feature of the second residual module (209) is added to the CLIP image feature and then input into the second cross-attention module (210); The first cross-attention module (205) uses the CLIP text feature as a query vector and uses the image feature input to the first cross-attention module (205) as a key vector and a value vector to perform a cross-attention calculation to obtain a first fused feature; the second cross-attention module (210) uses the CLIP text feature as a query vector and uses the image feature input to the second cross-attention module (210) as a key vector and a value vector to perform a cross-attention calculation to obtain a second fused feature; the first fused feature and the second fused feature are added to obtain the preliminary fusion result.

3. The multimodal image fusion model based on global and local text perception according to claim 2, characterized in that: The first residual module (204) and the second residual module (209) each comprise a plurality of first extraction layers (10) and a second extraction layer (11), the plurality of first extraction layers (10) being connected in series in sequence, the input end of the first first extraction layer (10) being connected to the input end of the corresponding residual module, the output feature of the last first extraction layer (10) being added to the input of the corresponding residual module and then input into the second extraction layer (11), and the output end of the second extraction layer (11) being connected to the output end of the corresponding residual module; The first extraction layer (10) includes a third convolution layer (1001) and a first ReLU activation function layer (1002), and the second extraction layer (11) includes a fourth convolution layer (1101) and a second ReLU activation function layer (1102).

4. The multimodal image fusion model based on global and local text perception according to claim 1, characterized in that: The feature extraction network (6) includes multiple layers of residual space state modules (12), wherein a local text perception module (9) is embedded after every N layers of the residual space state modules (12), where N is a positive integer; the CLIP image encoder (3) inputs the CLIP image features into each of the local text perception modules (9); and the BLIP text encoder (5) inputs the BLIP text features into each of the local text perception modules (9).

5. The multimodal image fusion model based on global and local text perception according to claim 4, characterized in that: The residual space state module (12) comprises a first regularization layer (1201), a visual state space module (1202), a first scaling layer (1203), a second regularization layer (1204), a seventh convolutional layer (1205), a channel attention module (1206) and a second scaling layer (1207); The first regularization layer (1201) and the visual state space module (1202) are connected in sequence, and the second regularization layer (1204), the seventh convolutional layer (1205) and the channel attention module (1206) are connected in sequence; the input end of the first regularization layer (1201) and the input end of the first scaling layer (1203) are both connected to the input end of the residual space state module (12); the output of the visual state space module (1202) and the output of the first scaling layer (1203) are added and input into the second regularization layer (1204) and the second scaling layer (1207); the output of the channel attention module (1206) and the output of the second scaling layer (1207) are added and output from the output end of the residual space state module (12).

6. The multimodal image fusion model based on global and local text perception according to claim 5, characterized in that: The visual state space module (1202) includes a first linear layer (13), a depth-separable convolutional layer (14), a first Swish activation function layer (15), a 2D selective scanning module (16), a layer normalization module (17), a second linear layer (18), a second Swish activation function layer (19) and a third linear layer (20); The first linear layer (13), the depthwise separable convolutional layer (14), the first Swish activation function layer (15), the 2D selective scanning module (16) and the layer normalization module (17) are connected in sequence, and the second linear layer (18) and the second Swish activation function layer (19) are connected in sequence; the input end of the first linear layer (13) and the input end of the second linear layer (18) are both connected to the input end of the visual state space module (1202), the output of the layer normalization module (17) and the output of the second Swish activation function layer (19) are multiplied and input into the third linear layer (20), and the output end of the third linear layer (20) is connected to the output end of the visual state space module (1202).

7. The multimodal image fusion model based on global and local text perception according to claim 1, characterized in that: The local text perception module (9) comprises a plurality of layers of squeeze excitation modules (901), an embedding module (902), a third cross attention module (903), a plurality of hole convolution layers (904) and a fifth convolution layer (905); Several layers of the squeeze excitation module (901), the embedding module (902) and the third cross attention module (903) are connected in sequence, and the input end of the squeeze excitation module (901) of the first layer is connected to the input end of the local text perception module (9); The squeeze excitation module (901) is used to enhance important features and suppress redundant information of image features input into the squeeze excitation module (901) through an adaptive channel attention mechanism; The embedding module (902) is used to map the CLIP image features to the semantic space of the image features input to the embedding module (902) through a multi-layer perceptron and then embed the features into the image features input to the embedding module (902); The third cross attention module (903) uses the BLIP text feature as a query vector and the image feature input to the third cross attention module (903) as a key vector and a value vector to perform cross attention calculation, and inputs the calculation results into each of the hole convolution layers (904); The expansion rates of the various atrous convolutional layers (904) are different, and the outputs of all the atrous convolutional layers (904) are channel-connected and then input into the fifth convolutional layer (905); the output end of the fifth convolutional layer (905) is connected to the output end of the local text perception module (9).

8. A multimodal image fusion method based on global and local text perception, characterized in that: Includes steps: A1. constructing an initial multimodal image fusion model based on global and local text perception; the initial multimodal image fusion model based on global and local text perception is the multimodal image fusion model based on global and local text perception according to any one of claims 1-7; A2. training the initial multimodal image fusion model based on global and local text perception to obtain a trained multimodal image fusion model based on global and local text perception; A3. Input the source image to be fused into the trained multimodal image fusion model based on global and local text perception to obtain a fused image output by the trained multimodal image fusion model based on global and local text perception.

9. The multimodal image fusion method based on global and local text perception according to claim 8, characterized in that: Step A2 includes: A201. Acquire multiple training samples; the training samples include sample source images and corresponding clean source images; the sample source images include sample infrared source images and sample visible light source images, the clean source images include clean infrared source images and clean visible light source images, the sample source images include degradation information, the clean source images do not include degradation information, and the degradation information includes weather interference information; A202. Select the training sample, input the corresponding sample source image into the initial multimodal image fusion model based on global and local text perception, and obtain the fused image output by the initial multimodal image fusion model based on global and local text perception, which is recorded as the first fused image; A203. Input the first fused image into the CLIP image encoder (3) to obtain a CLIP image feature corresponding to the first fused image, recorded as a first CLIP image feature; A204. Input the selected clean source image of the training sample into the CLIP text encoder (4) to obtain a CLIP text feature corresponding to the clean source image, recorded as a first CLIP text feature; A205. Align the first CLIP text feature with the first CLIP image feature in the same feature space; A206. Calculate a loss function according to the first fused image, the aligned first CLIP text features and the clean source image of the selected training sample; A207. Optimizing the model parameters of the initial multimodal image fusion model based on global and local text perception according to the loss function; A208. Repeat steps A202 to A207 until the loss function is less than a preset loss function threshold or the number of iterations reaches a preset number threshold.

10. The multimodal image fusion method based on global and local text perception according to claim 9, characterized in that: The loss function is: ； ； ； in, is the loss function value, is the pixel-level loss, For drive loss, is the first fused image, is the first CLIP text feature after the alignment, represents the norm, is the clean infrared source image of the selected training sample, is the clean visible light source image of the selected training sample, Represents the 1-norm.

Images