Lightweight aerial multispectral target detection method

Abstract

The present invention belongs to the field of image processing technology, and in particular relates to a lightweight aviation multispectral target detection method. The method comprises: S1: acquiring a visible light image and an infrared image of the same shooting scene; S2: constructing a multispectral target detection network, which includes a multi-scale extraction network, a neck network, and a head network; S3: inputting the visible light image and the infrared image into the multispectral target detection network for training to obtain a multispectral target detection model; S4: inputting the visible light image and the infrared image to be detected in the same shooting scene into the multispectral target detection model for detection to obtain the target position and target category. The present invention has the advantages of low computational complexity and small video memory usage, and can meet the practical application requirements of lightweight aviation multispectral multi-target detection.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a lightweight aviation multispectral target detection method. Background Art

[0002] Target detection algorithms can quickly extract information from images or videos, accurately locate targets of interest, and determine their categories. However, the application of single-modal target detection is subject to environmental influences and has certain limitations. For example, in conditions of insufficient lighting, such as at night or in bad weather, target detectors based on visible light images have difficulty extracting effective features, which affects detection performance. Relying solely on infrared images, target detectors lack color and texture information and have difficulty accurately determining target categories. To address this problem, multispectral target detection significantly improves the performance of target detectors in all-weather target detection by fusing features from images of different modalities. It also demonstrates excellent robustness and stability in different scenarios, and has important research significance and application value.

[0003] In multispectral target detection, how to effectively fuse different modal information is a key issue. Based on the fusion position, multispectral target detection algorithms can be divided into pixel-level fusion, feature-level fusion, and decision-level fusion. Pixel-level fusion fuses images of different modalities at the raw data layer. Pixel-level fusion requires additional fusion steps and cannot achieve end-to-end detection. Decision-level fusion uses two models to detect visible light images and infrared images respectively, and filters the detection results through methods such as non-maximum suppression. The operation is simple, but lacks information interaction between different modalities. At the same time, it greatly increases the computational complexity. Feature-level fusion fuses feature maps at different stages of the target detection model by designing various fusion modules, making full use of the multi-scale features of different modalities and achieving higher detection accuracy and efficiency. Summary of the Invention

[0004] In view of this, the present invention aims to provide a lightweight aerial multispectral target detection method to solve the problems of insufficient existing multimodal feature fusion, lack of semantic information and global receptive field in small-scale fusion feature maps, etc. The present invention is aimed at the practical application needs of all-weather, multi-scene and high-precision target detection in aerial remote sensing, and realizes the purpose of feature interaction between different modalities and feature mining and automatic fusion detection within each modality. The present invention has the advantages of low computational complexity and small video memory occupancy, and can meet the practical application needs of lightweight aerial multispectral multi-target detection.

[0005] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0006] A lightweight aerial multispectral target detection method specifically includes the following steps:

[0007] S1: Acquire visible light images and infrared images of the same shooting scene;

[0008] S2: Construct a multispectral target detection network, which includes a multi-scale extraction network, a neck network, and a head network;

[0009] The multi-scale extraction network is used to extract, interact and fuse information from visible light images and infrared images. The fused features are input into the head network through the neck network.

[0010] S3: Input the visible light image and infrared image into the multispectral target detection network for training to obtain a multispectral target detection model;

[0011] S4: Input the visible light image to be detected and the infrared image to be detected in the same shooting scene into the multispectral target detection model for detection to obtain the target position and target category.

[0012] Furthermore, the multi-scale extraction network includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a seventh feature extraction module, an eighth feature extraction module, a ninth feature extraction module, a tenth feature extraction module, a first efficient cross-attention fusion module, a second efficient cross-attention fusion module, a third efficient cross-attention fusion module, a fourth efficient cross-attention fusion module and an SPPF module; wherein, the infrared image is processed by the first feature extraction module and the second feature extraction module to obtain a feature map A1; the visible light image is processed by the third feature extraction module and the fourth feature extraction module to obtain a feature map A2; the feature map A1 and the feature map A2 are input into the first efficient cross-attention fusion module for processing, and the feature map A3 and the feature map A4 are obtained accordingly; the feature map A3 is input into the fifth feature extraction module for processing to obtain Feature map A5, input feature map A4 into the sixth feature extraction module for processing to obtain feature map A6, input feature map A5 and feature map A6 into the second efficient cross-attention fusion module for processing, and correspondingly obtain feature map A7 and feature map A8; input feature map A7 into the seventh feature extraction module for processing to obtain feature map A9, input feature map A8 into the eighth feature extraction module for processing to obtain feature map A10, input feature map A9 and feature map A10 into the third efficient cross-attention fusion module for processing, and correspondingly obtain feature map A11 and feature map A12; input feature map A11 into the ninth feature extraction module for processing to obtain feature map A13, input feature map A12 into the tenth feature extraction module for processing to obtain feature map A14, input feature map A13 and feature map A14 into the fourth efficient cross-attention fusion module for processing, and correspondingly obtain feature map A15 and feature map A16;

[0013] Add feature map A3 and feature map A4 to obtain the first fusion feature; add feature map A7 and feature map A8 to obtain the second fusion feature; add feature map A11 and feature map A12 to obtain the third fusion feature; add feature map A15 and feature map A16 to obtain feature map A17; input feature map A17 into the SPPF module for processing to obtain the fourth fusion feature.

[0014] Furthermore, the first feature extraction module and the third feature extraction module have the same network structure, and the first feature extraction module includes a first CBS module and a second CBS module having the same network structure;

[0015] The network structures of the second feature extraction module, the fourth feature extraction module, the fifth feature extraction module, the sixth feature extraction module, the seventh feature extraction module, the eighth feature extraction module, the ninth feature extraction module and the tenth feature extraction module are the same. The second feature extraction module includes a third CBS module and a first C2F module connected in sequence.

[0016] Furthermore, the network structures of the first efficient cross-attention fusion module, the second efficient cross-attention fusion module, the third efficient cross-attention fusion module and the fourth efficient cross-attention fusion module are the same. The first efficient cross-attention fusion module includes the fourth CBS module, the fifth CBS module, the first downsampling layer, the second downsampling layer, the third downsampling layer, the first Conv module, the second Conv module, the third Conv module, the fourth Conv module, the fifth Conv module, the sixth Conv module, the visible light cross-attention mechanism, the infrared cross-attention mechanism, the first upsampling layer, the second upsampling layer, and the first MLP module; wherein, the feature map A1 is processed by the first downsampling layer and the first Conv module to obtain the feature map B1; the feature map A2 is processed by the second downsampling layer and the second Conv module to obtain the feature map B2; the feature map A1 and the feature map A2 are spliced ​​to obtain the feature map B3, and the feature map B3 is processed by the fourth CBS module, the third downsampling layer and the third Conv module The feature map B4 is processed in rows to obtain feature map B4; the feature map B4 is split to obtain feature map B5 and feature map B6; the feature map B1 and feature map B5 are processed by the visible light cross attention mechanism and the first upsampling layer to obtain a visible light attention matrix; the feature map B2 and feature map B6 are processed by the infrared cross attention mechanism and the second upsampling layer to obtain an infrared attention matrix; the feature map A1 is input to the fourth Conv module for processing to obtain feature map B7; the feature map A2 is input to the fifth Conv module for processing to obtain feature map B8; the feature map B7 and the visible light attention matrix are added to obtain feature map B9; the feature map B8 and the infrared attention matrix are added to obtain feature map B10; the feature map B9 and feature map B10 are spliced ​​to obtain feature map B11; the feature map B11 is input to the sixth Conv module for processing to obtain feature map B12; the feature map A1 and feature map A2 are spliced ​​and processed by CBS in the channel dimension to obtain feature map F f , the feature map F fInput the feature map B12 to the fifth CBS module for processing to obtain feature map B13, add the feature map B12 and the feature map B13 to obtain feature map B14, input the feature map B14 to the first MLP module for processing to obtain feature map B15, split the feature map B15 into feature map B17 and feature map B18, and add the feature map B17 and the feature map A1 to obtain feature map A3, add the feature map B18 and the feature map A2 to obtain feature map A4, and add the feature map A3 and the feature map A4 to obtain the first fusion feature.

[0017] Furthermore, the network structures of the first CBS module, the second CBS module, the third CBS module, the fourth CBS module and the fifth CBS module are the same. The first CBS module includes a 2D convolution layer, a batch normalization layer and a SiLU activation function connected in sequence, and the convolution kernel of the 2D convolution layer is 3×3 and the stride is 2;

[0018] The network structures of the first downsampling layer, the second downsampling layer, and the third downsampling layer are the same. The first downsampling layer includes a maximum pooling layer and an average pooling layer. After the feature map A1 is processed by the maximum pooling layer and the average pooling layer respectively, the processing results of the two are multiplied by their respective assigned weights and added together. The added result is input into the first Conv module for processing. The weight assigned to the branch where the maximum pooling layer is located is 0.5, and the weight assigned to the branch where the average pooling layer is located is 0.5.

[0019] The feature scaling factor of the downsampling operation in the first efficient cross-attention fusion module is set to 8, the feature scaling factor of the downsampling operation in the second efficient cross-attention fusion module is set to 4, the feature scaling factor of the downsampling operation in the third efficient cross-attention fusion module is set to 2, and the feature scaling factor of the downsampling operation in the fourth efficient cross-attention fusion module is set to 1.

[0020] Furthermore, the neck network includes a second C2F module, a third C2F module, a fourth C2F module, a fifth C2F module, a sixth C2F module and a seventh C2F module. The fourth fusion feature is upsampled and concatenated with the third fusion feature to obtain a feature map A18. The feature map A18 is input to the second C2F module for processing to obtain a feature map A19. The feature map A19 is upsampled and concatenated with the second fusion feature to obtain a feature map A20. The feature map A20 is input to the third C2F module for processing to obtain a feature map A21. The feature map A21 is upsampled and concatenated with the first fusion feature to obtain a feature map A22. The feature map A22 is input to the fourth C2F module for processing to obtain a feature map A23; after downsampling, the feature map A23 is spliced ​​with the feature map A21 to obtain a feature map A24, and the feature map A24 is input to the fifth C2F module for processing to obtain a scale feature map P3. The scale feature map P3 is downsampled and spliced ​​with the feature map A19 to obtain a feature map A25. The feature map A25 is input to the sixth C2F module for processing to obtain a scale feature map P4. The scale feature map P4 is downsampled and spliced ​​with the feature map A17 to obtain a feature map A26. The feature map A26 is input to the seventh C2F module for processing to obtain a scale feature map P5.

[0021] Furthermore, the head network includes a SIEM module, a first detection head, a second detection head and a third detection head. The scale feature map P3, the scale feature map P4 and the scale feature map P5 are input into the SIEM module for processing to obtain a small-scale feature map. The small-scale feature map is input into the first detection head for processing, the scale feature map P4 is input into the second detection head for processing, and the scale feature map P5 is input into the third detection head for processing. The target position and target category are obtained according to the processing results of the first detection head, the second detection head and the third detection head.

[0022] Furthermore, the first detection head corresponds to the P3 detection layer, the second detection head corresponds to the P4 detection layer, and the third detection head corresponds to the P5 detection layer.

[0023] Furthermore, the SIEM module includes a fourth downsampling layer, a fifth downsampling layer, a sixth CBS module, a seventh CBS module, a seventh Conv module, an attention mechanism, a third upsampling layer, an eighth Conv module, a ninth Conv module and a second MLP module, wherein the scale feature map P3 is input to the fourth downsampling layer for processing to obtain a feature map C1, the scale feature map P4 is input to the fifth downsampling layer for processing to obtain a feature map C2, the scale feature map P5 is input to the sixth CBS module for processing to obtain a feature map C3, the feature maps C1, C2 and C3 are spliced ​​to obtain a feature map C4, The feature map C4 is processed by the seventh CBS module, the seventh Conv module and the attention mechanism to obtain the attention matrix, which is input into the third upsampling layer for upsampling to obtain the feature map C5, and the scale feature map P3 is input into the eighth Conv module for processing to obtain the feature map C6. The feature map C5 and the feature map C6 are added to obtain the feature map C7, and the feature map C7 is input into the ninth Conv module for processing to obtain the feature map C8. The feature map C8 and the scale feature map P3 are added to obtain the feature map C9, and the feature map C9 is input into the second MLP module for processing to obtain the small-scale feature map .

[0024] Furthermore, the second MLP module has the same network structure as the first MLP module. The first MLP module includes an eighth CBS module and a ninth CBS module. After the feature map B14 is input into the MLP module, it is processed by the eighth CBS module and the ninth CBS module to obtain a feature map D1. After the feature map D1 is added to the feature map B14, a feature map B15 is obtained. The channel expansion rate between the eighth CBS module and the ninth CBS module is set to 2.

[0025] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0026] (1) The present invention creates the lightweight aerial multispectral target detection method. Considering the sufficiency of feature fusion and the semantic information and global receptive field of small-scale fusion features, the present invention designs a lightweight multispectral target detection model based on the YOLOv8 architecture. In the backbone network, the C2F module is used as the basic unit to construct two independent networks, which are respectively used to extract the multi-scale feature map of visible light and the multi-scale feature map of infrared image, and adopts an efficient cross-attention fusion module to fuse the multimodal feature maps of the same stage. The present invention introduces a feature scaling method for local information enhancement (setting the feature scaling factor of the downsampling operation in each efficient cross-attention fusion module), which effectively reduces the computational complexity and video memory usage while reducing the information loss caused by the feature scaling process. After the feature aggregation network, the present invention also designs a SIEM module (small-scale information enhancement module) to fuse the multi-scale feature maps and model long-range dependencies, providing semantic information and global receptive field for the small-scale branches.

[0027] (2) The present invention creates a lightweight aerial multispectral target detection method and proposes a new multimodal feature fusion module, namely, an efficient cross-attention fusion module. This module consists of a cross-attention mechanism and a feature scaling method for local information enhancement. The cross-attention mechanism can effectively extract, interact, and fuse multimodal features, while the feature scaling method for local information enhancement not only reduces the negative impact caused by the downsampling-upsampling process, but also significantly reduces the computational complexity of the fusion module.

[0028] (3) The lightweight aerial multispectral target detection method created by the present invention, the SIEM module (small-scale information enhancement module) is designed to fuse multi-scale feature maps and calculate the self-attention mechanism, aiming to enhance the semantic information and global perception ability of small-scale branches. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0030] Figure 1 A schematic diagram of a flow chart of a lightweight aviation multispectral target detection method according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram of the network structure of the multispectral target detection model described in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the network structure of the first efficient cross-attention fusion module described in an embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the network structure of the SIEM module described in the embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram of the detection results of the present invention on the DroneVehicle dataset described in the embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram of the detection results of the present invention on the LLVIP dataset described in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0037] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0038] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0039] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0040] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0041] like Figure 1 As shown, the present invention proposes a lightweight aerial multispectral target detection method, which specifically includes the following steps: S1: obtaining visible light images and infrared images under the same shooting scene; S2: constructing a multispectral target detection network, which includes a multi-scale extraction network, a neck network and a head network; the multi-scale extraction network is used to extract, interact and fuse information from visible light images and infrared images, and the fused features are input to the head network through the neck network; S3: inputting the visible light image and the infrared image into the multispectral target detection network for training to obtain a multispectral target detection model; S4: inputting the visible light image and the infrared image to be detected under the same shooting scene into the multispectral target detection model for detection to obtain the target position and target category.

[0042] It should be noted that the present invention first reads the visible light image and the infrared image, and the shooting scenes of the visible light image and the infrared image are the same; the visible light image and the infrared image are respectively input into two independent multi-scale feature extraction networks, and the information extraction, interaction and fusion between different modal feature maps are realized through an efficient cross-attention fusion module; the multi-scale fused feature map is input into the neck network, and the semantic information and positioning information contained in the visible light image and the infrared image are respectively transmitted along the top-down and bottom-up paths; the decoupled detection head is used to predict the three-scale feature maps (P3, P4, P5) generated by the neck network, and output the target position and target category.

[0043] In some embodiments, as Figure 2As shown, the multi-scale extraction network includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a seventh feature extraction module, an eighth feature extraction module, a ninth feature extraction module, a tenth feature extraction module, a first efficient cross-attention fusion module, a second efficient cross-attention fusion module, a third efficient cross-attention fusion module, a fourth efficient cross-attention fusion module and an SPPF module; wherein the infrared image is processed by the first feature extraction module and the second feature extraction module to obtain a feature map A1; the visible light image is processed by the third feature extraction module and the fourth feature extraction module to obtain a feature map A2; the feature map A1 and the feature map A2 are input to the first efficient cross-attention fusion module for processing, and the feature map A3 and the feature map A4 are obtained accordingly; the feature map A3 is input to the fifth feature extraction module for processing to obtain a feature map A1. Feature map A5, feature map A4 is input to the sixth feature extraction module for processing to obtain feature map A6, feature map A5 and feature map A6 are input to the second efficient cross attention fusion module for processing, and feature map A7 and feature map A8 are obtained correspondingly; feature map A7 is input to the seventh feature extraction module for processing to obtain feature map A9, feature map A8 is input to the eighth feature extraction module for processing to obtain feature map A10, feature map A9 and feature map A10 are input to the third efficient cross attention fusion module for processing, and feature map A11 and feature map A12 are obtained correspondingly; feature map A11 is input to the ninth feature extraction module for processing to obtain feature map A13, feature map A12 is input to the tenth feature extraction module for processing to obtain feature map A14, feature map A13 and feature map A14 are input to the fourth efficient cross attention fusion module for processing, and feature map A15 and feature map A16 are obtained correspondingly;

[0044] Add feature map A3 and feature map A4 to obtain the first fusion feature; add feature map A7 and feature map A8 to obtain the second fusion feature; add feature map A11 and feature map A12 to obtain the third fusion feature; add feature map A15 and feature map A16 to obtain feature map A17; input feature map A17 into the SPPF module for processing to obtain the fourth fusion feature.

[0045] It should be noted that the visible light image and infrared image are processed separately by two independent feature extraction modules and then simultaneously by the first efficient cross-attention fusion module to obtain the first visible light feature (feature map A3), the first infrared feature (feature map A4), and the first fused feature. The first visible light feature and the first infrared feature are each processed separately by independent feature extraction modules and simultaneously by the second efficient cross-attention fusion module to obtain the second visible light feature (feature map A7), the second infrared feature (feature map A8), and the second fused feature. The second visible light feature and the second infrared feature are each processed separately by independent feature extraction modules and simultaneously by the third efficient cross-attention fusion module to obtain the third visible light feature (feature map A11), the third infrared feature (feature map A12), and the third fused feature. The third visible light feature and the third infrared feature are each processed separately by independent feature extraction modules and simultaneously by the fourth efficient cross-attention fusion module and the SPPF module to obtain the fourth fused feature. The first, second, third, and fourth fused features are all input into the neck network. The SPPF module is a common structure in the YOLO series of models and is used to extract multi-scale information from the feature maps.

[0046] In some embodiments, the first feature extraction module and the third feature extraction module have the same network structure, and the first feature extraction module includes a first CBS module and a second CBS module having the same network structure;

[0047] The network structure of the second feature extraction module, the fourth feature extraction module, the fifth feature extraction module, the sixth feature extraction module, the seventh feature extraction module, the eighth feature extraction module, the ninth feature extraction module and the tenth feature extraction module are the same, and the second feature extraction module includes a third CBS module and a first C2F module connected in sequence;

[0048] It should be noted that the first CBS module, the second CBS module and the third CBS module all include a 2D convolution layer, a batch normalization layer and a SiLU activation function connected in sequence. The convolution kernel of the 2D convolution layer is 3×3 and the stride is 2.

[0049] In some embodiments, the network structures of the first efficient cross-attention fusion module, the second efficient cross-attention fusion module, the third efficient cross-attention fusion module and the fourth efficient cross-attention fusion module are the same, and the first efficient cross-attention fusion module includes a fourth CBS module, a fifth CBS module, a first downsampling layer, a second downsampling layer, a third downsampling layer, a first Conv module, a second Conv module, a third Conv module, a fourth Conv module, a fifth Conv module, a sixth Conv module, a visible light cross-attention mechanism, an infrared cross-attention mechanism, a first upsampling layer, a second upsampling layer, and a first MLP module; wherein, the feature map A1 is processed by the first downsampling layer and the first Conv module to obtain a feature map B1; the feature map A2 is processed by the second downsampling layer and the second Conv module to obtain a feature map B2; the feature map A1 and the feature map A2 are spliced ​​to obtain a feature map B3, and the feature map B3 is processed by the fourth CBS module, the third downsampling layer and the third Conv module Processing is performed to obtain feature map B4; feature map B4 is split to obtain feature map B5 and feature map B6; feature map B1 and feature map B5 are processed by the visible light cross attention mechanism and the first upsampling layer to obtain a visible light attention matrix; feature map B2 and feature map B6 are processed by the infrared cross attention mechanism and the second upsampling layer to obtain an infrared attention matrix; feature map A1 is input to the fourth Conv module for processing to obtain feature map B7; feature map A2 is input to the fifth Conv module for processing to obtain feature map B8; feature map B7 and the visible light attention matrix are added to obtain feature map B9; feature map B8 and the infrared attention matrix are added to obtain feature map B10; feature map B9 and feature map B10 are spliced ​​to obtain feature map B11; feature map B11 is input to the sixth Conv module for processing to obtain feature map B12; feature map A1 and feature map A2 are spliced ​​and processed by CBS in the channel dimension to obtain feature map F f , the feature map F f Input the feature map B12 to the fifth CBS module for processing to obtain feature map B13, add the feature map B12 and the feature map B13 to obtain feature map B14, input the feature map B14 to the first MLP module for processing to obtain feature map B15, split the feature map B15 into feature map B17 and feature map B18, and add the feature map B17 and the feature map A1 to obtain feature map A3, add the feature map B18 and the feature map A2 to obtain feature map A4, and add the feature map A3 and the feature map A4 to obtain the first fusion feature.

[0050] It should be noted that the convolution kernels of the first Conv module, the second Conv module, the third Conv module, and the sixth Conv module are 1×1, and the convolution kernels of the fourth Conv module and the fifth Conv module are 3×3.

[0051] Furthermore, the visible light features and infrared features are processed through the downsampling layer and the Conv module, respectively, to obtain visible light downsampled features (such as feature map B1) and infrared downsampled features (such as feature map B2). The visible light features and infrared features are concatenated and then passed through the CBS module to obtain a primary fused feature map (feature map F). The primary fused feature map is then processed through the downsampling layer, the Conv module, and a split operation to obtain primary fused visible light downsampled features (feature map B5) and primary fused infrared downsampled features (feature map B6). The visible light downsampled features and primary fused visible light downsampled features are processed through a visible light cross-attention mechanism and an upsampling layer, while the infrared downsampled features and primary fused infrared downsampled features are processed through an infrared cross-attention mechanism and an upsampling layer to obtain the visible light attention matrix and infrared attention matrix, respectively. The sum of the convolution operation's visible light features and the visible light attention matrix is ​​concatenated and convolved with the sum of the convolution operation's infrared features and the added attention matrix, and then added to the primary fusion feature map that has undergone the downsampling layer and convolution operation. After passing through the first MLP module, it is split into the visible light feature map and infrared feature map of this stage. The split visible light feature map and infrared feature map are added to the input visible light feature map and infrared feature map respectively, and the visible light features and infrared features of this layer are output. The visible light features and infrared features of this layer are added together to output the fusion features of this layer.

[0052] In some embodiments, the network structures of the first CBS module, the second CBS module, the third CBS module, the fourth CBS module, and the fifth CBS module are the same, the first CBS module includes a 2D convolution layer, a batch normalization layer, and a SiLU activation function connected in sequence, and the convolution kernel of the 2D convolution layer is 3×3 and the stride is 2;

[0053] The network structures of the first downsampling layer, the second downsampling layer, and the third downsampling layer are the same. The first downsampling layer includes a maximum pooling layer and an average pooling layer. After the feature map A1 is processed by the maximum pooling layer and the average pooling layer respectively, the processing results of the two are multiplied by their respective assigned weights and added together. The added result is input into the first Conv module for processing. The weight assigned to the branch where the maximum pooling layer is located is 0.5, and the weight assigned to the branch where the average pooling layer is located is 0.5.

[0054] The feature scaling factor of the downsampling operation in the first efficient cross-attention fusion module is set to 8, the feature scaling factor of the downsampling operation in the second efficient cross-attention fusion module is set to 4, the feature scaling factor of the downsampling operation in the third efficient cross-attention fusion module is set to 2, and the feature scaling factor of the downsampling operation in the fourth efficient cross-attention fusion module is set to 1.

[0055] In some embodiments, the neck network includes a second C2F module, a third C2F module, a fourth C2F module, a fifth C2F module, a sixth C2F module, and a seventh C2F module. The fourth fusion feature is upsampled and then spliced ​​with the third fusion feature to obtain a feature map A18. The feature map A18 is input to the second C2F module for processing to obtain a feature map A19. The feature map A19 is upsampled and then spliced ​​with the second fusion feature to obtain a feature map A20. The feature map A20 is input to the third C2F module for processing to obtain a feature map A21. The feature map A21 is upsampled and then spliced ​​with the first fusion feature to obtain a feature map A22. The feature map A22 is input to the fourth C2F module for processing to obtain a feature map A23; after downsampling, the feature map A23 is spliced ​​with the feature map A21 to obtain a feature map A24, and the feature map A24 is input to the fifth C2F module for processing to obtain a scale feature map P3. The scale feature map P3 is downsampled and spliced ​​with the feature map A19 to obtain a feature map A25. The feature map A25 is input to the sixth C2F module for processing to obtain a scale feature map P4. The scale feature map P4 is downsampled and spliced ​​with the feature map A17 to obtain a feature map A26. The feature map A26 is input to the seventh C2F module for processing to obtain a scale feature map P5.

[0056] It should be noted that the first, second, third, and fourth fused features are input to the neck network, respectively. The fourth fused feature is upsampled and concatenated with the third fused feature, then passes through the C2F layer to obtain the first-scale fused feature (e.g., feature map A19). This is then upsampled and concatenated with the second fused feature, and then passes through the C2F layer to obtain the second-scale fused feature (e.g., feature map A21). This is then upsampled and concatenated with the first fused feature, and then passes through the C2F layer and downsampled. This concatenated with the second-scale fused feature, and then passes through the C2F layer to obtain the P3-scale feature map. This is then downsampled and concatenated with the first-scale fused feature, and then passes through the C2F layer to obtain the P4-scale feature map. This is then downsampled and concatenated with the fourth fused feature, and then passes through the C2F layer to obtain the P5-scale feature map. The P3, P4, and P5-scale feature maps are simultaneously input to the small-scale information enhancement module (SIEM module) to obtain small-scale feature maps.

[0057] like Figure 3 As shown, and Represents the visible light feature map and the infrared feature map respectively. By performing splicing and 3×3 convolution operations in the channel dimension, the information carried by the two modalities is adaptively integrated together to obtain a simple fusion feature map :

[0058] ;

[0059] Among them, CBS represents the processing function of the CBS module (convolution kernel is 3×3).

[0060] Due to the computational complexity and spatial dimension of self-attention The square of the length of is proportional to the three feature maps, and the three feature maps are downsampled to obtain

[0061] ;

[0062] in, Represents the feature scaling factor, and Down represents the downsampling operation.

[0063] Specifically, the downsampling operation Consists of maximum pooling and average pooling:

[0064] ;

[0065] in, and is the regulating factor, Represents the input feature map, here we will and Both are set to 0.5.

[0066] The calculation process of cross attention is as follows: the downsampled feature map is respectively calculated through three 1×1 convolutions to calculate the query matrix of visible light , infrared interrogation matrix , the fused key-value matrix and , .

[0067] ;

[0068] Then, adjust , , and , and calculate the size of and Relative to Visible light attention matrix and infrared attention matrix :

[0069] ;

[0070] ;

[0071] Among them, the symbol represents matrix multiplication, is the number of channels of the feature map, and SoftMax is the SoftMax function.

[0072] In order to reduce the information loss during feature scaling, and Perform size transformation and upsample the two matrices using 3×3 convolution from and Extract local information from and Combined, , .

[0073] ;

[0074] In obtaining the feature maps of the two modalities relative to the fusion feature map After the information is obtained, it is spliced ​​in the channel dimension and , and use 1x1 convolution for linear projection. In addition, we introduce The residual path of is used to alleviate the model degradation problem, and the 1×1 convolution is used to Adjust the number of channels. .

[0075] ;

[0076] in, is the channel dimension splicing operation, It is a 1x1 convolutional layer.

[0077] In the feedforward network, the MLP module consists of two 1×1 convolutions and a residual connection to achieve information exchange between different modalities. Considering the computational cost, the channel expansion rate between the two convolutions is set to 2:

[0078]

[0079] Divide along the channel dimension , and add them to the original two modal feature maps to obtain the visible feature map for feature extraction in the next stage and infrared signatures Finally, the two feature maps are added together to obtain the fusion feature map of this stage. , .

[0080] ;

[0081] ;

[0082] in, To evenly divide the feature map in the channel dimension.

[0083] In some embodiments, the head network includes a SIEM module, a first detection head, a second detection head, and a third detection head. The scale feature map P3, the scale feature map P4, and the scale feature map P5 are input into the SIEM module for processing to obtain a small-scale feature map. The small-scale feature map is input into the first detection head for processing, the scale feature map P4 is input into the second detection head for processing, and the scale feature map P5 is input into the third detection head for processing. The target position and target category are obtained according to the processing results of the first detection head, the second detection head, and the third detection head.

[0084] In some embodiments, the first detection head corresponds to the P3 detection layer, the second detection head corresponds to the P4 detection layer, and the third detection head corresponds to the P5 detection layer.

[0085] It should be noted that the three detection heads are arranged in parallel and correspond to the output feature maps of each scale. The detection heads are used to obtain prediction information at different scales from the multi-scale fusion features, generate target categories and target coordinate positions, and mark the target attributes and areas in the infrared image and visible light image.

[0086] In some embodiments, the SIEM module includes a fourth downsampling layer, a fifth downsampling layer, a sixth CBS module, a seventh CBS module, a seventh Conv module, an attention mechanism, a third upsampling layer, an eighth Conv module, a ninth Conv module, and a second MLP module, wherein the scale feature map P3 is input to the fourth downsampling layer for processing to obtain a feature map C1, the scale feature map P4 is input to the fifth downsampling layer for processing to obtain a feature map C2, the scale feature map P5 is input to the sixth CBS module for processing to obtain a feature map C3, and the feature maps C1, C2, and C3 are spliced ​​to obtain a feature map C4. , the feature map C4 is processed by the seventh CBS module, the seventh Conv module and the attention mechanism to obtain the attention matrix, the attention matrix is ​​input to the third upsampling layer for upsampling operation, and the feature map C5 is obtained. The scale feature map P3 is input to the eighth Conv module for processing to obtain the feature map C6, the feature map C5 and the feature map C6 are added to obtain the feature map C7, the feature map C7 is input to the ninth Conv module for processing to obtain the feature map C8, the feature map C8 and the scale feature map P3 are added to obtain the feature map C9, the feature map C9 is input to the second MLP module for processing to obtain the small-scale feature map .

[0087] In the SIEM module (small-scale information enhancement module), the input scale feature map P3 and scale feature map P4 are downsampled with feature scaling factors of 4 and 2 respectively, and the scale feature map P5 is input to the sixth CBS module for processing. After the three processing results are spliced, the number of channels is adjusted through the seventh CBS module and the seventh Conv module, and then the scaled dot product attention is calculated to obtain the attention matrix. The attention matrix is ​​upsampled and added to the P3 scale feature map processed by the eighth Conv module. After processing by the ninth Conv module, it is added to the P3 scale feature map and linearly projected and calculated by the second MLP module for the feedforward network to obtain the small-scale feature map. .

[0088] Furthermore, the seventh Conv module is a 1×1 convolution, and the eighth and ninth Conv modules are 3×3 convolutions.

[0089] In some embodiments, the second MLP module has the same network structure as the first MLP module, the first MLP module includes an eighth CBS module and a ninth CBS module, the feature map B14 is input into the MLP module, and then processed by the eighth CBS module and the ninth CBS module to obtain a feature map D1, and the feature map D1 is added to the feature map B14 to obtain a feature map B15; the channel expansion rate between the eighth CBS module and the ninth CBS module is set to 2.

[0090] It should be noted that the MLP module consists of two 1×1 convolutional layers and a residual connection, and the channel expansion rate between the two convolutional layers is set to 2. The feature scaling factors of the downsampling operations (consisting of maximum pooling layers and average pooling layers) in these four efficient cross-attention fusion modules are set to 8, 4, 2, and 1, respectively, thereby controlling the size of the downsampled feature map at each stage to 1 / 32 of the input image. The network structure of the eighth and ninth CBS modules is the same, both including sequentially connected convolutional layers, batch normalization layers, and SiLU activation functions. The convolution kernels of the convolutional layers contained in the eighth and ninth CBS modules are both 1×1.

[0091] like Figure 4 As shown, the multi-scale feature map after selecting the path aggregation network , and As the input of SIEM module, similar feature scaling method is adopted to reduce computational complexity. and The downsampling operation with feature scaling factors of 4 and 2 was applied respectively, and A 1x1 convolution is performed:

[0092] ;

[0093] , and By splicing in the channel dimension and adjusting the number of channels using 1×1 convolution, a fused feature map is obtained. .

[0094] ;

[0095] Then, calculate the scaled dot product attention of DP and get the attention matrix :

[0096]

[0097]

[0098] In order to reduce the information loss caused by the downsampling-upsampling process, is upsampled in advance and added with the small-scale local information extracted by 3×3 convolution:

[0099] ;

[0100] in, is the upsampling operation.

[0101] Finally, yes Perform linear projection and feedforward network calculations to obtain , used to replace the original Serves as the input to the small-scale branch of the detection head.

[0102] ;

[0103] .

[0104] Example 1

[0105] The present invention proposes a lightweight aerial multispectral target detection method, which specifically includes the following steps:

[0106] Step 1: Input a visible light image and an infrared image with a size of H×W respectively.

[0107] In step 2, the visible light image and infrared image are input into two independent multi-scale feature extraction networks respectively, and the information extraction, interaction and fusion between the feature maps of different modalities are realized through the efficient cross-attention fusion module.

[0108] This paper optimizes and improves the YOLOv8 framework. The entire multispectral target detection network is optimized using a stochastic gradient descent algorithm with a learning rate of 0.01, a momentum of 0.937, a weight decay of 0.0005, and an epoch of 100. It uses an early stopping mechanism and does not employ pre-trained weights. Each image undergoes a horizontal flip with a probability of 0.5, an HSV transformation, and mosaic enhancement to increase diversity. The same data augmentation is applied to both visible and infrared images, ensuring that the enhanced image pairs fed into the multispectral target detection network remain aligned. The DroneVehicle and LLVIP datasets were used for training and testing on two NVIDIA RTX3090 GPUs, with a batch size of 16.

[0109] In step 3, the multi-scale fusion feature map is input into the neck network to transfer semantic information and positioning information along the top-down and bottom-up paths respectively.

[0110] The top-down path refers to the branch where the upsampling process is located, which gradually transfers the semantic information of the high-level feature map to the low-level feature map; the bottom-up path refers to the branch where the downsampling process is located, which gradually transfers the positioning information of the low-level feature map to the high-level feature map.

[0111] In step 4, the decoupled detection head is used to predict the three-scale feature maps (P3, P4, and P5) generated by the neck network, and output the target location and target category.

[0112] To verify the effectiveness of this invention, we compared the detection results of our method with those of existing methods. For object detection evaluation, we introduced several common metrics, such as precision, recall, mAP50, and mAP, as criteria for model performance. We also used model parameter counts, mAP50, and mAP to measure the model's detection speed. The effectiveness and superiority of the algorithm were verified from multiple perspectives: detection accuracy, model size, and real-time performance.

[0113] Experimental results of different methods on the DroneVehicle dataset are shown in Table 1. The proposed method achieved 83.3% mAP50 and 63.4% mAP, respectively. With fewer parameters, it significantly surpassed two simple fusion methods, YOLOv8n-Add and YOLOv8n-Cat. Compared to the deeper and wider YOLOv8s-Add, it improved by 1.0 mAP50 and 0.8 mAP, respectively, demonstrating that YOLO-FT (the multispectral object detection model designed by the proposed method) can effectively fuse features from different modalities under the guidance of global information. Compared to the ICA Fusion method, the proposed multispectral object detection model improved by 0.5 and 2.5 mAP, respectively, while reducing the number of parameters by 12.95M. Figure 5 The following figure shows the detection results of the present invention on the DroneVehicle dataset. It can be found that YOLO-FT has fewer missed detections and false detections.

[0114] Table 1

[0115]

[0116] Table 2 shows the experimental results of different methods on the LLVIP dataset. Our method achieved 97.9% mAP50 and 68.1% mAP, respectively, surpassing the 0.9 mAP50 and 1.0 mAP achieved by YOLOv8n in the infrared modality. Our method achieved the same mAP50 as LRAF-Net, but with a 1.8 mAP improvement. This method achieved the best detection accuracy with the least number of parameters, demonstrating that the predicted boxes of the multispectral object detection model have a higher degree of overlap with the ground-truth boxes. Figure 6 As a result of using the present invention to perform detection on the LLVIP dataset, it can be found that YOLO-FT has fewer missed detections and false detections.

[0117] Table 2

[0118]

[0119] In summary, the present invention improves the YOLOv8 network architecture and proposes a lightweight multispectral target detection model, which is mainly composed of an efficient cross-attention module and a small-scale information enhancement module. The cross-attention mechanism fully extracts, interacts and fuses multimodal features by calculating the attention of different modal feature maps to simple fusion feature maps. The feature scaling method of local information enhancement reduces the computational complexity of the efficient cross-attention module and reduces the negative impact caused by the downsampling-upsampling process. The SIEM module captures the long-range dependencies of multi-scale fusion feature maps, provides rich semantic information and global perception capabilities for small-scale branches, and can effectively improve the all-weather, multi-scene, and high-precision target detection capabilities of aerial remote sensing.

[0120] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0121] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A lightweight aviation multispectral target detection method, characterized by: The specific steps include: S1: Acquire visible light images and infrared images of the same shooting scene; S2: constructing a multispectral target detection network, wherein the multispectral target detection network includes a multi-scale extraction network, a neck network, and a head network; The multi-scale extraction network is used to extract, interact and fuse information from visible light images and infrared images, and the fused features are input into the head network via the neck network; S3: Input the visible light image and infrared image into the multispectral target detection network for training to obtain a multispectral target detection model; S4: Input the visible light image and infrared image to be detected in the same shooting scene into the multispectral target detection model for detection to obtain the target position and target category; The multi-scale extraction network includes a first feature extraction module, a second feature extraction module, a third feature extraction module, a fourth feature extraction module, a fifth feature extraction module, a sixth feature extraction module, a seventh feature extraction module, an eighth feature extraction module, a ninth feature extraction module, a tenth feature extraction module, a first efficient cross-attention fusion module, a second efficient cross-attention fusion module, a third efficient cross-attention fusion module, a fourth efficient cross-attention fusion module and an SPPF module; wherein, the infrared image is processed by the first feature extraction module and the second feature extraction module to obtain a feature map A1; the visible light image is processed by the third feature extraction module and the fourth feature extraction module to obtain a feature map A2; the feature map A1 and the feature map A2 are input into the first efficient cross-attention fusion module for processing, and the feature map A3 and the feature map A4 are obtained accordingly; the feature map A3 is input into the fifth feature extraction module for processing to obtain a feature map A5, input the feature map A4 to the sixth feature extraction module for processing to obtain a feature map A6, input the feature map A5 and the feature map A6 to the second efficient cross-attention fusion module for processing, and correspondingly obtain the feature map A7 and the feature map A8; input the feature map A7 to the seventh feature extraction module for processing to obtain the feature map A9, input the feature map A8 to the eighth feature extraction module for processing to obtain the feature map A10, input the feature map A9 and the feature map A10 to the third efficient cross-attention fusion module for processing, and correspondingly obtain the feature map A11 and the feature map A12; input the feature map A11 to the ninth feature extraction module for processing to obtain the feature map A13, input the feature map A12 to the tenth feature extraction module for processing to obtain the feature map A14, input the feature map A13 and the feature map A14 to the fourth efficient cross-attention fusion module for processing, and correspondingly obtain the feature map A15 and the feature map A16; Add feature map A3 and feature map A4 to obtain the first fusion feature; add feature map A7 and feature map A8 to obtain the second fusion feature; add feature map A11 and feature map A12 to obtain the third fusion feature; add feature map A15 and feature map A16 to obtain feature map A17; input feature map A17 into the SPPF module for processing to obtain the fourth fusion feature; The first feature extraction module and the third feature extraction module have the same network structure, and the first feature extraction module includes a first CBS module and a second CBS module with the same network structure; The network structure of the second feature extraction module, the fourth feature extraction module, the fifth feature extraction module, the sixth feature extraction module, the seventh feature extraction module, the eighth feature extraction module, the ninth feature extraction module and the tenth feature extraction module are the same, and the second feature extraction module includes a third CBS module and a first C2F module connected in sequence; The network structures of the first efficient cross-attention fusion module, the second efficient cross-attention fusion module, the third efficient cross-attention fusion module and the fourth efficient cross-attention fusion module are the same. The first efficient cross-attention fusion module includes the fourth CBS module, the fifth CBS module, the first downsampling layer, the second downsampling layer, the third downsampling layer, the first Conv module, the second Conv module, the third Conv module, the fourth Conv module, the fifth Conv module, the sixth Conv module, the visible light cross-attention mechanism, the infrared cross-attention mechanism, the first upsampling layer, the second upsampling layer, and the first MLP module; wherein, the feature map A1 is processed by the first downsampling layer and the first Conv module to obtain the feature map B1; the feature map A2 is processed by the second downsampling layer and the second Conv module to obtain the feature map B2; the feature map A1 and the feature map A2 are spliced ​​to obtain the feature map B3, and the feature map B3 is processed by the fourth CBS module, the third downsampling layer and the third Conv module Processing to obtain feature map B4; split feature map B4 to obtain feature map B5 and feature map B6, process feature map B1 and feature map B5 through the visible light cross attention mechanism and the first upsampling layer to obtain a visible light attention matrix, process feature map B2 and feature map B6 through the infrared cross attention mechanism and the second upsampling layer to obtain an infrared attention matrix; input feature map A1 to the fourth Conv module for processing to obtain feature map B7, input feature map A2 to the fifth Conv module for processing to obtain feature map B8, add feature map B7 and the visible light attention matrix to obtain feature map B9, add feature map B8 and the infrared attention matrix to obtain feature map B10, splice feature map B9 and feature map B10 to obtain feature map B11, input feature map B11 to the sixth Conv module for processing to obtain feature map B12, splice feature map A1 and feature map A2 in the channel dimension and perform CBS processing to obtain feature map F f , the feature map F f Input the feature map B12 to the fifth CBS module for processing to obtain feature map B13, add the feature map B12 and the feature map B13 to obtain feature map B14, input the feature map B14 to the first MLP module for processing to obtain feature map B15, split the feature map B15 into feature map B17 and feature map B18, and add the feature map B17 and the feature map A1 to obtain feature map A3, add the feature map B18 and the feature map A2 to obtain feature map A4, and add the feature map A3 and the feature map A4 to obtain the first fusion feature.

2. The lightweight aviation multispectral target detection method according to claim 1, characterized in that: The network structures of the first CBS module, the second CBS module, the third CBS module, the fourth CBS module and the fifth CBS module are the same. The first CBS module includes a 2D convolution layer, a batch normalization layer and a SiLU activation function connected in sequence, and the convolution kernel of the 2D convolution layer is 3×3 and the stride is 2; The network structures of the first downsampling layer, the second downsampling layer, and the third downsampling layer are the same, wherein the first downsampling layer includes a maximum pooling layer and an average pooling layer. After the feature map A1 is processed by the maximum pooling layer and the average pooling layer, the processing results of the two are multiplied by their respective assigned weights and added together, and the added result is input into the first Conv module for processing. The weight assigned to the branch where the maximum pooling layer is located is 0.5, and the weight assigned to the branch where the average pooling layer is located is 0.5; The feature scaling factor of the downsampling operation in the first efficient cross-attention fusion module is set to 8, the feature scaling factor of the downsampling operation in the second efficient cross-attention fusion module is set to 4, the feature scaling factor of the downsampling operation in the third efficient cross-attention fusion module is set to 2, and the feature scaling factor of the downsampling operation in the fourth efficient cross-attention fusion module is set to 1.

3. The lightweight aviation multispectral target detection method according to claim 1, characterized in that: The neck network includes a second C2F module, a third C2F module, a fourth C2F module, a fifth C2F module, a sixth C2F module and a seventh C2F module. The fourth fusion feature is upsampled and concatenated with the third fusion feature to obtain a feature map A18. The feature map A18 is input to the second C2F module for processing to obtain a feature map A19. The feature map A19 is upsampled and concatenated with the second fusion feature to obtain a feature map A20. The feature map A20 is input to the third C2F module for processing to obtain a feature map A21. The feature map A21 is upsampled and concatenated with the first fusion feature to obtain a feature map A22. The feature map A 22 is input to the fourth C2F module for processing to obtain a feature map A23; the feature map A23 is downsampled and spliced ​​with the feature map A21 to obtain a feature map A24, and the feature map A24 is input to the fifth C2F module for processing to obtain a scale feature map P3, and the scale feature map P3 is downsampled and spliced ​​with the feature map A19 to obtain a feature map A25, and the feature map A25 is input to the sixth C2F module for processing to obtain a scale feature map P4, and the scale feature map P4 is downsampled and spliced ​​with the feature map A17 to obtain a feature map A26, and the feature map A26 is input to the seventh C2F module for processing to obtain a scale feature map P5.

4. The lightweight aviation multispectral target detection method according to claim 3, characterized in that: The head network includes a SIEM module, a first detection head, a second detection head and a third detection head. The scale feature map P3, the scale feature map P4 and the scale feature map P5 are input into the SIEM module for processing to obtain a small-scale feature map. The small-scale feature map is input into the first detection head for processing, the scale feature map P4 is input into the second detection head for processing, and the scale feature map P5 is input into the third detection head for processing. The target position and target category are obtained according to the processing results of the first detection head, the second detection head and the third detection head.

5. The lightweight aviation multispectral target detection method according to claim 4, characterized in that: The first detection head corresponds to the P3 detection layer, the second detection head corresponds to the P4 detection layer, and the third detection head corresponds to the P5 detection layer.

6. The lightweight aviation multispectral target detection method according to claim 4, characterized in that: The SIEM module includes a fourth downsampling layer, a fifth downsampling layer, a sixth CBS module, a seventh CBS module, a seventh Conv module, an attention mechanism, a third upsampling layer, an eighth Conv module, a ninth Conv module and a second MLP module, wherein the scale feature map P3 is input to the fourth downsampling layer for processing to obtain a feature map C1, the scale feature map P4 is input to the fifth downsampling layer for processing to obtain a feature map C2, the scale feature map P5 is input to the sixth CBS module for processing to obtain a feature map C3, the feature maps C1, C2 and C3 are spliced ​​to obtain a feature map C4, and the feature map C4 is processed by the seventh CBS module, the seventh Conv module and the attention mechanism to obtain an attention matrix, which is input into the third upsampling layer for upsampling to obtain a feature map C5, and the scale feature map P3 is input into the eighth Conv module for processing to obtain a feature map C6, and the feature map C5 is added to the feature map C6 to obtain a feature map C7, and the feature map C7 is input into the ninth Conv module for processing to obtain a feature map C8, and the feature map C8 is added to the scale feature map P3 to obtain a feature map C9, and the feature map C9 is input into the second MLP module for processing to obtain a small-scale feature map .

7. The lightweight aviation multispectral target detection method according to claim 6, characterized in that: The second MLP module has the same network structure as the first MLP module. The first MLP module includes the eighth CBS module and the ninth CBS module. After the feature map B14 is input into the MLP module, it is processed by the eighth CBS module and the ninth CBS module to obtain the feature map D1. After adding the feature map D1 to the feature map B14, the feature map B15 is obtained. The channel expansion rate between the eighth CBS module and the ninth CBS module is set to 2.

Images