Multispectral target detection method and apparatus, system, storage medium

Abstract

The application discloses a multispectral target detection method and device, system and storage medium, comprising the following steps: S1, acquiring an infrared image and a visible light image of a target; S2, extracting structural information of the infrared image and texture information of the visible light image through a double-flow backbone network; S3, fusing the structural information of the infrared image and the texture information of the visible light image by using a cross-modal window self-attention fusion module; and S4, inputting the features after cross-modal fusion to the neck of a detection network for information aggregation, and then realizing target detection based on multi-scale information through a multi-scale detection head. The technical scheme of the application improves the detection accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to a multispectral target detection method, device, system, and storage medium. Background Technology

[0002] For intelligent traffic monitoring and autonomous driving tasks, most current mainstream multispectral image target detection methods are based on convolutional neural networks (CNNs) or Transformers for feature extraction and multimodal fusion. CNN methods typically employ hand-designed fusion strategies, stitching or weighting features from infrared and visible light images at different levels. However, due to the limited receptive field of convolutional kernels, they are insufficient in modeling the contextual relationships of distant targets and struggle to effectively capture deep complementary information between different modalities. Furthermore, some studies have introduced Transformer structures to leverage their global modeling capabilities for cross-modal fusion. While this has yielded some progress in modeling ability, the need for Transformers to segment images into fixed-size blocks can lead to information loss and feature inconsistencies. Simultaneously, the computational complexity of Transformers results in significant computational overhead when dealing with high-resolution images, limiting deployment efficiency in practical applications. Therefore, how to reduce computational burden and improve feature consistency and robustness while ensuring cross-modal fusion effectiveness remains a critical issue that urgently needs to be addressed in the field of multispectral target detection. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a multispectral target detection method, device, system, and storage medium.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A multispectral target detection method, comprising: Step S1: Acquire infrared and visible light images of the target; Step S2: Extract the structural information of the infrared image and the texture information of the visible light image respectively through a dual-stream backbone network; Step S3: Perform cross-modal window self-attention fusion on the structural information of the infrared image and the texture information of the visible light image; Step S4: Input the cross-modal fusion features into the neck of the detection network for information aggregation, and then achieve target detection based on multi-scale information through a multi-scale detection head.

[0005] As a preferred option, the dual-stream backbone network is based on the YOLOv5 model, with its backbone modified into two branches to extract structural information from infrared images and texture information from visible light images, respectively.

[0006] Preferably, in step S3, the cross-modal window self-attention fusion module constructs a cross-modal global window self-attention mechanism based on SwinTransformer to achieve joint modeling of the features after stitching together the structural information of the infrared image and the texture information of the visible light image.

[0007] The present invention also provides a multispectral target detection device, comprising: The first processing module is used to acquire infrared and visible light images of the target; The second processing module is used to extract the structural information of the infrared image and the texture information of the visible light image through the dual-stream backbone network, respectively. The third processing module is used to perform cross-modal window self-attention fusion of structural information from infrared images and texture information from visible light images. The fourth processing module is used to input the cross-modal fusion features into the detection network to achieve target detection based on multi-scale information.

[0008] As a preferred option, the dual-stream backbone network is based on the YOLOv5 model, with its backbone modified into two branches to extract structural information from infrared images and texture information from visible light images, respectively.

[0009] Preferably, the third processing module constructs a cross-modal global window self-attention mechanism based on Swin Transformer to achieve joint modeling of the features after stitching together the structural information of infrared images and the texture information of visible light images.

[0010] The present invention also provides a multispectral target detection system, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multispectral target detection method when run by the processor.

[0011] The present invention also provides a storage medium storing a computer program, which executes a multispectral target detection method when running.

[0012] This invention employs a dual-stream feature extraction structure and borrows the architectural design principles of the Swing Transformer in its fusion module to achieve efficient and accurate cross-modal feature fusion. The YOLOv5 model is extended and modified into a dual-stream backbone network to extract features from visible light and infrared images. Combined with a cross-modal fusion module featuring a sliding window attention mechanism and hierarchical feature modeling capabilities, it effectively maintains the spatial continuity and causal structure of the image, improving the representational quality of intermodal feature alignment and fusion. Furthermore, the cross-modal window self-attention fusion module utilizes a global window for attention calculation, significantly reducing computational overhead while maintaining detection accuracy, demonstrating good robustness and practicality. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a flowchart of the multispectral target detection method according to an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Example 1: like Figure 1 As shown, this embodiment of the invention provides a multispectral target detection method, including: Step S1: Acquire infrared and visible light images of the target; Step S2: Extract the structural information of the infrared image and the texture information of the visible light image respectively through a dual-stream backbone network; Step S3: Perform cross-modal window self-attention fusion on the structural information of the infrared image and the texture information of the visible light image; Step S4: Input the cross-modal fusion features into the detection network to achieve target detection based on multi-scale information.

[0018] As one embodiment of the present invention, in step S2, the dual-stream backbone network structure is as follows: Figure 1 As shown, Figure 1 The upper and lower boxes contain the single-modal feature extraction networks for visible light and infrared light, respectively. Both single-modal feature extraction networks are constructed using the same convolutional and C3 modules as the YOLOv5 backbone. Each backbone network outputs five feature maps at different scales.

[0019] In one embodiment of the present invention, in step S3, cross-modal window self-attention fusion is performed on the features extracted in step S2. Specifically, the present invention sets up three intermediate cross-modal window self-attention fusion modules as feature fusion modules, fusing features from two modalities and inputting the fused features back into the single-modal feature extraction network to guide the single-modal feature extraction process. The present invention performs feature fusion operations on the features at the last three scales of the network, thus balancing efficiency and effectiveness. The feature fusion module is based on the Swing Transformer architecture and uses a cross-modal sliding window mechanism to implement feature partitioning and attention calculation. This approach not only effectively models intra-modal and inter-modal correlations but also maintains the spatial structural integrity of the features. Simultaneously, the calculation method of the cross-modal global window self-attention mechanism significantly reduces computational overhead, thereby achieving an efficient and accurate cross-modal deep feature fusion method.

[0020] The specific implementation process of cross-modal feature fusion in step S3 using the cross-modal window self-attention fusion module is as follows: First, the features extracted by the two-stream CNN network are represented as follows: ,in, Representing the real number field indicates that each element of the feature tensor is a real value. This represents the visible light features extracted from the backbone network. This represents the infrared features extracted from the backbone network. These represent the height, width, and number of channels of the input feature, respectively. Indicates the sequence number of different scales; step S3 only occurs in... First, the two modal features are fused. Second, the two input modal features are concatenated dimensionally to obtain the fused features. : .

[0021] Then, the obtained fused features are input into the cross-modal window self-attention fusion module. This fusion module utilizes the cross-modal global window self-attention mechanism to perform cross-modal feature fusion. The specific steps are as follows: the above fused features First, it will go through a Layers and a window-based multi-head self-attention module The first-level output is obtained. The calculation process is as follows: in, Representation layer normalization is used to improve the convergence and stability of network training.

[0022] Then the first-stage output is passed through a... Layers and The first layer, after passing through a residual connection, yields the second-stage output. : in, It is a multilayer perceptron structure used to perform nonlinear mapping and channel dimension transformation on normalized features.

[0023] After that, after another Layer and shift window cross attention module After obtaining the third level output : at last, After a Layers and The layer yields the final output of the fusion module. : The following is a detailed introduction. Module: This module performs window segmentation and reshaping of the image, and calculates attention. The input features for this module are: First, the blocks are slicing and reshaping are performed. The features after slicing and reshaping are as follows: The calculation process for window slicing and reshaping is as follows: in, and As an intermediate variable, and Indicates the width and height of the window. and Indicates the number of windows cut in the row and column directions. To input the height and width of the tensor, The function represents the reshaping function, and the relationship between the variables in the formula is shown in the following equation: The obtained features are then input into the attention module for attention calculation.

[0024] In the formula, , and Let represent the learnable linear mapping matrices, These are the query matrix, key matrix, and value matrix, respectively. This represents the dimension of the key vector. This represents the output features after attention weighting, where... .

[0025] Finally, the obtained features are reshaped back to their original size and fed back into the visible light and infrared single-mode feature extraction branches to enhance single-mode information, correct the relationship between the two modes, and facilitate subsequent scale fusion. Furthermore, Modules and The difference is The module adds a ShiftedWindow mechanism, which involves a fixed offset of the feature map during window division, causing the window position to be misaligned between adjacent layers, thereby allowing information interaction between different windows across modalities.

[0026] In one embodiment of the present invention, in step S4, after the fusion module performs cross-modal fusion processing on the bimodal features at each scale, it inputs the output fused features into the neck module and the detection head of the detection network to achieve effective integration of multi-scale information and target detection. The detection network of the present invention adopts a three-stage architecture, including a CSPDarknet backbone network, an FPN-PAN neck network, and a multi-scale detection head. The backbone network is used for multimodal feature extraction, the neck network is used for the fusion and enhancement of multimodal and multi-scale features, and the detection head performs target classification and localization based on multi-scale features. These three components are connected in series to complete feature extraction, fusion enhancement, and target detection at different scales.

[0027] The technical solution of this invention firstly employs a dual-stream backbone network to extract structural information from infrared images and texture information from visible light images. Through collaborative modeling of structural flow and detail flow, complementary features between modalities are effectively preserved and enhanced, improving the detection model's adaptability to complex environments (such as low light, occlusion, and low contrast scenes). Secondly, the cross-modal window self-attention fusion module, drawing inspiration from the Swing Transformer architecture, constructs a cross-modal global window self-attention mechanism. This mechanism concatenates features from both modalities and jointly models them, capturing both intra-modal contextual information and deep inter-modal relationships. Simultaneously, the sliding window mechanism avoids the causal structure disruption problem caused by fixed blocks in traditional Transformers, maintaining image spatial continuity and semantic consistency. Furthermore, the global window self-attention mechanism used in the cross-modal window self-attention fusion module significantly reduces computational cost compared to pixel-based global self-attention mechanisms, greatly improving computational efficiency while maintaining fusion accuracy. This module was applied to the multi-scale detection branch (80×80, 40×40, 20×20), further enhancing the ability to perceive targets of different sizes.

[0028] This invention achieves an effective balance between fusion effect, detection accuracy and computational efficiency, and has stronger robustness and practical application value, especially in multispectral small target detection and complex environments.

[0029] Example 2: This invention also provides a multispectral target detection device, comprising: The first processing module is used to acquire infrared and visible light images of the target; The second processing module is used to extract the structural information of the infrared image and the texture information of the visible light image through the dual-stream backbone network, respectively. The third processing module is used to perform cross-modal window self-attention fusion of structural information from infrared images and texture information from visible light images; The fourth processing module is used to input the cross-modal fusion features into the detection network to achieve target detection based on multi-scale information.

[0030] As one embodiment of the present invention, the dual-stream backbone network is based on the YOLOv5 model, and its backbone network is modified into two branches to extract the structural information of infrared images and the texture information of visible light images, respectively.

[0031] As one embodiment of the present invention, the third processing module constructs a cross-modal global window self-attention mechanism based on Swing Transformer, thereby realizing joint modeling of the features after splicing the structural information of infrared images and the texture information of visible light images.

[0032] Example 3: This invention also provides a multispectral target detection system, including: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a multispectral target detection method when run by the processor.

[0033] Example 4: This invention also provides a storage medium storing a computer program that executes a multispectral target detection method during runtime.

[0034] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multispectral target detection method, characterized in that, include: Step S1: Acquire infrared and visible light images of the target; Step S2: Extract the structural information of the infrared image and the texture information of the visible light image respectively through a dual-stream backbone network; Step S3: The structural information of the infrared image and the texture information of the visible light image are fused using a cross-modal window self-attention fusion module based on SwinTransformer. Step S4: Input the cross-modal fusion features into the neck of the detection network for information aggregation, and then use the multi-scale detection head to achieve target detection based on multi-scale information.

2. The multispectral target detection method as described in claim 1, characterized in that, The dual-stream backbone network is based on the YOLOv5 model, with its backbone modified into two branches to extract structural information from infrared images and texture information from visible light images, respectively.

3. The multispectral target detection method as described in claim 2, characterized in that, In step S3, the cross-modal window self-attention fusion module constructs a cross-modal global window self-attention mechanism based on Swin Transformer to jointly model the features after stitching together the structural information of the infrared image and the texture information of the visible light image.

4. A multispectral target detection device, characterized in that, include: The first processing module is used to acquire infrared and visible light images of the target; The second processing module is used to extract the structural information of the infrared image and the texture information of the visible light image through the dual-stream backbone network, respectively. The third processing module is used to perform cross-modal window self-attention fusion of structural information from infrared images and texture information from visible light images; The fourth processing module is used to input the cross-modal fusion features into the detection network to achieve target detection based on multi-scale information.

5. The multispectral target detection device as described in claim 4, characterized in that, The dual-stream backbone network is based on the YOLOv5 model, with its backbone modified into two branches to extract structural information from infrared images and texture information from visible light images, respectively.

6. The multispectral target detection device as described in claim 5, characterized in that, The third processing module uses a cross-modal window self-attention fusion module, employing a cross-modal global window self-attention mechanism based on the Swin Transformer, to jointly model the features resulting from the splicing of structural information from infrared images and texture information from visible light images.

7. A multispectral target detection system, characterized in that, include: A memory and a processor, wherein the memory stores a computer program executed by the processor, the computer program performing the multispectral target detection method as described in any one of claims 1 to 3 when executed by the processor.

8. A storage medium, characterized in that, The storage medium stores a computer program, which executes the multispectral target detection method as described in any one of claims 1 to 3 when it runs.

Images