Generative multi-modal image fusion detection method based on state space model

Abstract

The invention discloses a generative multi-modal image fusion detection method based on a state space model, and belongs to the technical field of multi-modal image processing and target detection.The method comprises the steps that a network model comprising a generator and a discriminator is adopted, the generator is composed of a feature flow, a fusion flow and a reconstruction flow, extracting low-level features of the source image by using a convolution module, a Mangbar module and a guide type Mangbar module in the feature flow; performing shallow layer and deep layer fusion by using a cross-modal interaction fusion module based on Mangban in the fusion stream, and guiding deep layer feature extraction by using a shallow layer fusion result; and generating a fusion image through up-sampling in the reconstruction stream, and integrating a target detection module to carry out end-to-end detection. Wherein the Mangbar module utilizes the linear complexity characteristic of a state space model to realize global perception modeling of image features. According to the method, the quality and information richness of multi-modal image fusion are effectively improved, and the accuracy and robustness of target detection in a complex scene are enhanced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of multi-modal image processing and target detection, and particularly relates to a generative multi-modal image fusion detection method based on a state space model. BACKGROUND

[0002] Multi-modal fusion detection technology refers to the comprehensive processing of image data from different sensors or different modalities (such as infrared, visible light, radar, etc.) to improve the accuracy and robustness of target detection, recognition and classification. This technology has been widely used in security monitoring, intelligent driving and other fields. For example, in the security monitoring scene, by fusing visible light images and infrared images, visible light images can provide rich texture and color information, while infrared images can present the thermal radiation characteristics of objects, and the fusion of the two can more comprehensively capture target information, thereby effectively improving the detection performance under complex lighting conditions.

[0003] However, the existing multi-modal image fusion detection method still faces severe efficiency bottleneck and cost constraints in actual hardware deployment and application. First, in order to capture the global context information of the image, the existing high-precision fusion method often uses a feature extraction model based on a Transformer architecture or a deep convolutional neural network (CNN). For the Transformer architecture, the core self-attention mechanism has a quadratic growth of computational complexity and memory consumption with the image resolution. When processing high-resolution security video streams, this will cause an explosive increase in video memory usage, greatly limiting the concurrent processing capacity of edge devices, and also requiring expensive large video memory GPUs to run, significantly increasing hardware costs. For deep CNN, in order to obtain a large enough receptive field, the network depth must be continuously increased, which directly leads to an increase in floating-point operation and an increase in inference delay, making it difficult to meet the real-time detection system's requirement for high frame rate. Second, the existing multi-modal processing scheme usually processes "image fusion" and "target detection" as two independent and serial stages. That is, a complex fusion network is used to generate a fused image, which is written to memory, and then the detection network reads the image for inference. This step-by-step processing architecture causes frequent off-chip memory access, causing huge data transmission bandwidth pressure. In addition, at the feature fusion level, existing technologies often use simple channel splicing or element-by-element addition followed by convolution. In order to make up for the information loss caused by this simple interaction, existing methods have to design extremely wide network channels. This makes the model face the contradiction between high parallelism and low data reuse rate when executed on hardware, resulting in low utilization of hardware computing units and inability to achieve the best detection performance under limited power budget. Therefore, how to break through the computational complexity bottleneck of traditional architecture in long sequence processing under the premise of ensuring multi-modal feature depth interaction and detection accuracy, reduce redundant feature calculation and memory access overhead, and build an end-to-end detection system with low cost and high efficiency, is a technical problem to be solved in the field. SUMMARY

[0004] To solve the above technical problems, the present application provides a generative multi-modal image fusion detection method based on a state space model to solve the problems existing in the prior art.

[0005] To achieve the above purpose, the present application provides a generative multi-modal image fusion detection method based on a state space model, comprising: obtaining source images of different modalities in an intelligent traffic road environment perception scene, the source images including visible light images and infrared images; inputting the source images into a pre-trained network model, the network model including a generator and a discriminator; wherein the generator includes a feature flow, a fusion flow and a reconstruction flow; In the feature flow, the source image is subjected to feature extraction using a convolution module and a Mamba module to obtain low-level features, and then a guided Mamba module is used to extract deeper features from the extracted low-level features, wherein the Mamba module utilizes the linear complexity property of the state space model to realize global perception modeling of image features. In the fusion flow, a Mamba-based cross-modal interaction fusion module is used to perform shallow fusion and deep fusion on the low-level features, wherein the result of shallow fusion is used to guide the extraction of deep features, and the Mamba-based cross-modal interaction fusion module utilizes the linear complexity property of the state space model to realize global perception modeling of image features. In the reconstruction flow, an up-sampling module is used to up-sample the fused features to generate a fused image, and a target detection module is integrated into the reconstruction flow to perform target detection on the fused image.

[0006] Preferably, in the feature flow, after the low-level features are extracted, the obtained low-level features and the shallowly fused features are spliced and then subjected to down-sampling operation by a down-sampling module.

[0007] Preferably, the guided Mamba module includes two state space model modules and adopts a symmetric structure design to perform asymmetric interaction processing on the input infrared feature map and visible light feature map, wherein in the infrared feature map branch, the infrared feature plays a leading role and the visible light feature plays an auxiliary role, and in the visible light feature map branch, the visible light feature plays a leading role and the infrared feature plays an auxiliary role.

[0008] Preferably, in the shallow fusion stage of the fusion flow, a Mamba-based cross-modal interaction fusion module is used to fuse the low-level features to obtain preliminary fusion features, wherein the Mamba module utilizes the linear complexity property of the state space model to realize global perception modeling of image features.

[0009] Preferably, in the deep fusion stage of the fusion flow, a Mamba-based cross-modal interaction fusion module is used, and before the cross-modal interaction fusion module is input, the result of shallow fusion is spliced with the result of deep feature extraction to form two fusion paths, which are respectively interacted with two feature flows and then input into the cross-modal interaction fusion module.

[0010] Preferably, two state space model modules are included in the manba-based cross-modal interaction fusion module, a symmetrical structure design is adopted, and asymmetric interaction processing is performed on the input infrared feature map and visible light feature map, wherein the infrared feature plays a leading role and the visible light feature plays an auxiliary role in the infrared feature map branch, and the visible light feature plays a leading role and the infrared feature plays an auxiliary role in the visible light feature map branch, and finally the features output by the two branches are spliced to obtain the final output of the manba-based cross-modal interaction fusion module.

[0011] Preferably, in the reconstruction flow, the upsampling module gradually expands the size of the fused feature map by combining convolution operation and manba module, wherein the manba module utilizes the linear complexity of the state space model to realize global perception modeling of image features.

[0012] Preferably, the target detection module is a YOLOv5 detection module.

[0013] Preferably, the method further comprises designing a loss function, and the loss function comprises a bounding box regression loss, a classification loss and a confidence loss.

[0014] Preferably, the bounding box regression loss uses a GIoU loss, the classification loss uses a binary cross-entropy loss, and the confidence loss uses a squared error loss.

[0015] Compared with the prior art, the present application has the following advantages and technical effects: The present application realizes the depth interaction and complementarity of features between different modalities by adopting a generator architecture including a feature flow, a fusion flow and a reconstruction flow, extracting features by convolution and manba module, extracting deeper features by guided manba module, and combining a manba-based cross-modal interaction fusion module for shallow and deep fusion (wherein the result of shallow fusion is used to guide the extraction of deep features). This makes the generated fused image capable of retaining and highlighting key characteristics (such as thermal radiation information of infrared image and texture details of visible light image) from different modalities at the same time, significantly enriching the information content of the fused image.

[0016] Since the target detection module is integrated in the reconstruction flow and directly performs target detection on the fused image, the present application constructs an end-to-end "fusion-detection" integrated system. This design enables the image fusion process to be trained with the goal of optimizing the final detection performance, thereby ensuring that the fusion result is more suitable for downstream detection tasks. Therefore, in the face of complex scenes such as bad weather and occlusion, the present application can provide more stable and reliable target detection results.

[0017] This invention, through a serial processing architecture consisting of a feature stream, a fusion stream, and a reconstruction stream, and by applying a state-space model in each process, can effectively extract, interact with, and reconstruct image features at multiple levels. This not only helps capture global structural information but also finely preserves local details such as the edges and textures of the target, thus laying a solid foundation for high-quality image fusion and high-precision target detection.

[0018] This invention leverages the potential advantages of state-space models in handling long sequence dependencies and applies them to the core steps of feature extraction and fusion in multimodal images. This invention provides a feasible technical path for efficiently processing high-resolution multimodal image data, helping to improve processing efficiency while ensuring performance. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a generative multimodal image fusion detection method based on a state-space model according to an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0022] Example 1 This invention incorporates a meticulously designed bimodal interaction module within the generator, focusing on extracting deep local information between different modalities, thus overcoming the shortcomings of existing technologies in local feature mining. This module enables more precise capture of key details in infrared and visible light images, such as the edges and textures of targets, thereby enhancing the local feature representation capability of the fused image.

[0023] This invention proposes a generative multimodal image fusion detection method based on a state-space model, which adopts an advanced architecture and module design to achieve efficient and accurate target detection.

[0024] like Figure 1 As shown, this embodiment provides a generative multimodal image fusion detection method based on a state-space model, including: Step 1: Acquire source images of different modalities in the intelligent transportation road environment perception scenario, including visible light images and infrared images.

[0025] Specifically, a training dataset is prepared, selecting intelligent transportation road environment perception as the specific application scenario. Two different modal datasets are prepared for the same road scene at the same time and from the same perspective: RGB visible light images acquired by an onboard visible light camera, and thermal infrared images acquired by an onboard infrared thermal imager. The datasets are labeled to record detailed information such as the location and category of target objects in each image, providing supervisory signals for subsequent target detection.

[0026] Step 2: Input the source image into a pre-trained network model, which includes a generator and a discriminator; Specifically, the generator produces a high-quality fused image by extracting and fusing features from images of different modalities. The discriminator evaluates the realism of the generated image, avoiding focusing solely on a single type of source image, thereby acquiring more comprehensive information from the source images and improving the fidelity of the fused image.

[0027] The generator includes a feature stream, a fusion stream, and a reconstruction stream; In the feature stream, convolutional modules and Mamba modules are used to extract features from the source image to obtain low-level features. Then, guided Mamba modules are used to extract deeper features from the extracted low-level features. The Mamba modules utilize the linear complexity characteristics of the state-space model to achieve global perception modeling of image features. Furthermore, in the feature stream, after extracting the low-level features, the obtained low-level features and the shallowly fused features are concatenated, and then the feature map is downsampled using a downsampling module.

[0028] Specifically, in the feature stream, this embodiment uses two input images. and Initial features are extracted through convolutional layers. Then, the Mamba module further extracts low-level feature information from the initial features. and Next, the low-level features and the shallow fusion output features are concatenated, and the feature map is downsampled using the downsampling module to obtain the features. and The downsampled feature map is fed into a guided Mamba module to extract more localized, deeper feature information. and This allows for a more refined representation of image features.

[0029] In the fusion stream, a Mamba-based cross-modal interactive fusion module is used to perform shallow and deep fusion on the low-level features. The result of shallow fusion is used to guide the extraction of deep features. The Mamba-based cross-modal interactive fusion module utilizes the linear complexity characteristics of the state-space model to achieve global perceptual modeling of image features. Furthermore, the guided Mamba module includes two state-space model modules with a symmetrical structure design. It performs asymmetrical interactive processing on the input infrared feature map and visible light feature map. In the infrared feature map branch, the infrared feature plays a dominant role and the visible light feature plays an auxiliary role. In the visible light feature map branch, the visible light feature plays a dominant role and the infrared feature plays an auxiliary role.

[0030] Specifically, in the guided Mamba module, this embodiment first processes the input features. and Layer normalization is performed to ensure the consistency of feature distribution. Next, the normalized features are transformed using a linear layer to adapt to subsequent processing steps. Then, in this embodiment, the transformed features are divided into two parts along the channel for independent processing. One part of the features is first processed for local spatial information through a convolutional layer, and then a 2D selective scanning mechanism (SS2D) is used to capture long-range spatial dependencies, as shown in formula (1), to obtain... and .

[0031] (1) in, These are the features after 2D selective scanning of the visible light branch. The features are obtained after 2D selective scanning of the infrared branch. SS2D represents 2D selective scanning, and Conv represents a convolutional layer. Split To divide along the channel, Linear For linear mapping, LN is layer normalization.

[0032] Another part of the features are transformed nonlinearly through the SiLU activation function to enhance the expressive power of the features, as shown in Equation (2), to obtain and .

[0033] (2) These two feature sets are then enhanced through bi-branch cross-multiplication and linear mapping. Finally, to preserve valuable original information, the enhanced features are integrated with the original features via skip connections to obtain the final output. This allows for the enhancement of bimodal feature expression capabilities while maintaining information integrity.

[0034] (3) Through this guided Mamba module design, the guided Mamba module can more deeply understand and extract deep information from the input features, thereby improving the model's performance in multimodal image processing tasks.

[0035] Furthermore, in the shallow fusion stage of the fusion stream, a Mamba-based cross-modal interactive fusion module is used to fuse the low-level features to obtain preliminary fused features.

[0036] Furthermore, in the deep fusion stage of the fusion stream, a Mamba-based cross-modal interactive fusion module is used. Before inputting into the cross-modal interactive fusion module, the shallow fusion result and the deep feature extraction result are concatenated to form two fusion paths. The two fusion paths interact with two feature streams respectively before being input into the cross-modal interactive fusion module.

[0037] Specifically, in the fusion stream, this embodiment employs a Mamba-based cross-modal interactive fusion module to perform layer-by-layer fusion of the feature information extracted from the feature stream. Specifically, in the shallow fusion stage, this embodiment introduces a Mamba-based cross-modal interactive fusion module, inputting the feature information from the feature stream into this module, and using the results of shallow fusion to guide the extraction of deeper features, thereby obtaining preliminary fused features. In the deep fusion stage, this embodiment still uses the Mamba-based cross-modal interactive fusion module. However, unlike shallow fusion, this embodiment applies the concept of residual structures before inputting the data into this module. Specifically, this embodiment concatenates the results of shallow fusion and deep feature extraction separately, forming two fusion paths. These two paths then interact with two feature streams respectively, and finally input these two fusion paths into the Mamba-based cross-modal interactive fusion module. In this way, not only is effective fusion of shallow and deep features achieved, but the feature streams are also used to optimize the processing of the fusion paths, resulting in more refined fused features. It can contain more comprehensive image information.

[0038] Specifically, in the Mamba-based cross-modal interactive fusion module, taking the shallow fusion stage as an example, this module consists of two symmetrical processing paths. Each path includes layer normalization, linear transformation, feature segmentation, parallel branch processing, and a cross-path feature fusion unit. The input features are first normalized, and then mapped to a high-dimensional space through a linear transformation. Next, the channel dimension is divided into two parts through a channel partitioning operation, which are then input into two parallel branches for independent processing. For branch A, the input features are first processed... and Local spatial features are extracted through convolutional layers, and then global information is integrated using 2D selective scanning (SS2D). and The formula is shown in (4).

[0039] (4) in, and These are the features after 2D selective scanning of the visible light branch and the features after 2D selective scanning of the infrared branch, respectively.

[0040] Branch B employs the SiLU activation function to preserve the non-linear distribution of features and reduces computational overhead through skip connections. Subsequently, the output features of the two branches are interacted through element-wise multiplication to enhance the semantic correlation between channels. Then, the outputs of these two processing paths are transformed through a linear layer and combined with the original input features. and Perform residual connections to obtain output features. and As shown in Equation (5). This design not only helps to form a stable gradient backpropagation channel, thereby improving the stability of training, but also improves the efficiency of training while maintaining model performance. In this way, the model can learn and fuse key information from different modalities more effectively, thereby achieving better performance in various tasks.

[0041] (5) Final output features The original input channel structure is preserved, while spatial and channel enhancement information across paths is integrated.

[0042] (6) in, The output is the result of fusing the shallow infrared branch and the visible light branch. and These are the fused outputs of the infrared branch and the visible light branch, respectively.

[0043] In the reconstruction stream, an upsampling module is used to upsample the fused features to generate a fused image; a target detection module is integrated into the reconstruction stream to perform target detection on the fused image.

[0044] Furthermore, in the reconstruction stream, the upsampling module gradually expands the size of the fused feature map by combining convolution operations and a state-space model. The Mamba module utilizes the linear complexity of the state-space model to achieve global perceptual modeling of image features.

[0045] Furthermore, the target detection module is a YOLOv5 detection module.

[0046] Specifically, after reconstructing the flow and completing further feature fusion, the model uses multiple upsampling modules to progressively restore the low-resolution feature map to a higher resolution. Each upsampling layer gradually increases the size of the fused feature map through a combination of convolution and Mamba modules until the desired output resolution is reached. The final part of the model contains a final fusion step and a convolutional layer that converts the fused multi-channel feature map into a single-channel output, resulting in the final fused image. .

[0047] Finally, a YOLOv5 detection module was introduced into the reconstructed network to ensure that the images input to the YOLOv5 detection module contain more complete scene information, thereby obtaining more robust detection results.

[0048] Step 3: Design the loss function; Furthermore, the method also includes designing a loss function, which includes bounding box regression loss, classification loss, and confidence loss.

[0049] Furthermore, the bounding box regression loss uses GIoU loss, the classification loss uses binary cross-entropy loss, and the confidence loss uses squared error loss.

[0050] The loss function used in this embodiment is the bounding box regression loss (… ), classification loss ( ) and confidence loss ( The total loss formula is: (7) in, , and These represent the GIoU loss for regression, the binary cross-entropy loss for classification, and the squared error loss for confidence, respectively. Parameters , and As a hyperparameter, it is used to adjust the weight of each loss term in the entire optimization process.

[0051] The model uses GIoU (Generalized Intersection over Union) loss to measure the similarity between the predicted and ground truth bounding boxes. It considers not only the overlapping area of ​​the bounding boxes but also their center positions and areas, thus more effectively guiding bounding box regression. By minimizing the GIoU loss, the model can more accurately locate the position of the target object in the image. The classification loss uses binary cross-entropy loss to handle multi-class classification problems. It measures the accuracy of the model's prediction of the class of each target object in the image. The confidence loss uses the Mean Squared Error Loss to measure the accuracy of the model's confidence predictions regarding the presence of an object in the bounding box. It helps the model learn which bounding boxes are more likely to contain the target object, thereby improving detection accuracy.

[0052] Step 4: Train the network model; The model is trained iteratively on the training set, with a loss function value calculated for each batch, and the network weights updated based on these values. Simultaneously, this embodiment periodically evaluates the model performance on the validation set to monitor overfitting and underfitting, and adjusts hyperparameters such as the learning rate and batch size accordingly.

[0053] After training, this embodiment performs a final evaluation of the model on the test set, analyzes the quality of the fused images, and uses appropriate metrics to measure model performance. If the evaluation results are unsatisfactory, this embodiment needs to go back and adjust the network architecture, loss function, or training process. Through hyperparameter tuning, this embodiment further optimizes the model performance until a satisfactory level is achieved.

[0054] Step 5: Determine the detection model.

[0055] After the fusion network is trained, the network model parameters are fixed, and this network model is determined as the final detection network model. When it is necessary to perform target detection on infrared and visible light images, the source image is directly input into the final fusion network to obtain the detection result.

[0056] In the guided Mamba module, one input is responsible for generating projection and time-scale parameters, while the other input is the sequence to be processed. This module is used to further extract deep feature information from the source image.

[0057] The guided Mamba module consists of two Mamba modules with a symmetrical structure. For the input infrared and visible light feature maps, this embodiment performs interactive processing between these two modalities. Although both branches use bimodal interaction modules, the interaction mechanism differs for each branch. In the infrared image branch, infrared feature information plays a dominant role, while visible light feature information serves as an auxiliary; conversely, in the visible light image branch, visible light feature information is dominant, and infrared feature information is auxiliary. This allows for deep feature extraction using the bimodal interaction module. The Mamba module utilizes the linear complexity of the state-space model to achieve global perceptual modeling of image features.

[0058] To extract and retain more information from the source image before inputting this module, this embodiment modifies the shallow features obtained from the first module. and The initial fusion was performed, and the result is expressed as follows: In this embodiment, when extracting features using the guided Mamba module, the feature stream is further guided by the fusion stream. The input to the guided Mamba module is the result of splicing the source image stream and the fusion stream channel.

[0059] Compared with existing similar products or methods, this invention demonstrates significant advantages in multimodal image fusion tasks. By combining serial and parallel structures, this invention can not only efficiently extract global structural information but also finely capture detailed features such as texture and edges, thereby generating fused images with rich details and better visual effects. The introduction of the Transformer mechanism and striped convolutional blocks further enhances the ability to model global information and edge feature details, effectively solving the problems of insufficient long-range dependency modeling and detail loss in traditional methods. The dual discriminator design and the application of multivariate loss functions significantly improve the fidelity and overall quality of the fused image, while enhancing the model's adaptability and generalization ability to different image types and complex scenes. Comparative experimental results on the FLIR dataset are shown in Table 1, and comparative experimental results on the LLVIP dataset are shown in Table 2. The experimental results in Tables 1 and 2 show that this invention outperforms existing state-of-the-art methods on multiple datasets and evaluation metrics, fully demonstrating its superiority.

[0060] Table 1 Table 2 The beneficial effects of this embodiment: This embodiment proposes a generative multimodal image fusion detection method based on a state-space model. The method extracts deep local information between different modalities by designing a guided Mamba module, and achieves deep feature fusion using a Mamba-based cross-modal interactive fusion module.

[0061] This embodiment improves the state space model by using the cross-modal interactive fusion module and the guided Mamba module of Mamba. It expands the two-dimensional features into a sequence by rows and columns, and regards each position as a hidden state that evolves with spatial steps. It completes the global information aggregation in one go through linear recursion. The parameters and resolution are linear, the memory usage is constant, it can be expanded in parallel, supports long sequence gradient stability, adapts to high-resolution real-time fusion detection, and takes into account the global receptive field, efficient hardware execution and low power consumption deployment.

[0062] Furthermore, by combining the YOLOv5-L detection module in this embodiment, the present invention not only improves the quality of the fused image, but also significantly enhances the accuracy and robustness of target detection. Experimental results on multiple datasets demonstrate its superiority.

[0063] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A generative multimodal image fusion detection method based on a state-space model, characterized in that, Includes the following steps: Acquire source images of different modalities in the intelligent transportation road environment perception scenario, wherein the source images include visible light images and infrared images; The source image is input into a pre-trained network model, which includes a generator and a discriminator; The generator includes a feature stream, a fusion stream, and a reconstruction stream; In the feature stream, convolutional modules and Mamba modules are used to extract features from the source image to obtain low-level features. Then, guided Mamba modules are used to extract deeper features from the extracted low-level features. The Mamba modules utilize the linear complexity characteristics of the state-space model to achieve global perception modeling of image features. In the fusion stream, a Mamba-based cross-modal interactive fusion module is used to perform shallow and deep fusion on the low-level features. The result of shallow fusion is used to guide the extraction of deep features. The Mamba-based cross-modal interactive fusion module utilizes the linear complexity characteristics of the state-space model to achieve global perceptual modeling of image features. In the reconstruction stream, an upsampling module is used to upsample the fused features to generate a fused image; a target detection module is integrated into the reconstruction stream to perform target detection on the fused image.

2. The method according to claim 1, characterized in that, In the feature stream, after extracting the low-level features, the obtained low-level features and the shallowly fused features are concatenated, and then the feature map is downsampled using a downsampling module.

3. The method according to claim 1, characterized in that, The guided Mamba module includes two state-space model modules with a symmetrical structure design. It performs asymmetrical interactive processing on the input infrared feature map and visible light feature map. In the infrared feature map branch, the infrared feature plays a dominant role and the visible light feature plays an auxiliary role. In the visible light feature map branch, the visible light feature plays a dominant role and the infrared feature plays an auxiliary role.

4. The method according to claim 1, characterized in that, In the shallow fusion stage of the fusion stream, the low-level features are fused using a Mamba-based cross-modal interactive fusion module to obtain preliminary fused features. The Mamba module utilizes the linear complexity characteristics of the state-space model to achieve global perceptual modeling of image features.

5. The method according to claim 4, characterized in that, In the deep fusion stage of the fusion stream, a Mamba-based cross-modal interactive fusion module is used. Before being input into the cross-modal interactive fusion module, the shallow fusion result and the deep feature extraction result are concatenated to form two fusion paths. The two fusion paths interact with two feature streams respectively before being input into the cross-modal interactive fusion module.

6. The method according to claim 1, characterized in that, The Mamba-based cross-modal interactive fusion module includes two state-space model modules with a symmetrical structure design. It performs asymmetrical interactive processing on the input infrared feature map and visible light feature map. In the infrared feature map branch, infrared features play a dominant role and visible light features play an auxiliary role. In the visible light feature map branch, visible light features play a dominant role and infrared features play an auxiliary role. Finally, the features output from the two branches are concatenated to obtain the final output of the Mamba-based cross-modal interactive fusion module.

7. The method according to claim 1, characterized in that, In the reconstruction stream, the upsampling module gradually expands the size of the fused feature map by combining convolution operations and the Mamba module. The Mamba module utilizes the linear complexity of the state-space model to achieve global perceptual modeling of image features.

8. The method according to claim 1, characterized in that, The target detection module is a YOLOv5 detection module.

9. The method according to claim 1, characterized in that, The method also includes designing a loss function, which includes bounding box regression loss, classification loss, and confidence loss.

10. The method according to claim 9, characterized in that, The bounding box regression loss uses GIoU loss, the classification loss uses binary cross-entropy loss, and the confidence loss uses squared error loss.

Images