Mask enhancement-based pixel-level multispectral fusion small target detection method and system

Abstract

The invention discloses a pixel-level multispectral fusion small target detection method and system based on mask enhancement. The method comprises the following steps: acquiring remote sensing image data to be processed, wherein the data comprises a multispectral image pair of a visible light spectrum and an infrared spectrum; splitting the multispectral modal image pair and inputting the split multispectral modal image pair into a feature fusion module at the same time to obtain respective masks, then obtaining feature information after mask processing according to the respective masks, performing feature fusion on the feature information and the split original image to obtain a first fusion result of a single modal feature, and then performing channel splicing and merging to obtain a second fusion result of the single modal feature; obtaining a result after the second fusion; inputting a result after the second fusion into an SE module, screening out effective information features, inhibiting redundant features, and obtaining an image after pixel-level fusion; and inputting the image after pixel-level fusion into a lightweight detection network to obtain a detection result. According to the invention, the detection precision of small targets in multi-modal remote sensing image target detection is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image target detection, and in particular relates to a pixel-level multi-spectral fusion small target detection method and system based on mask enhancement. Background Art

[0002] In recent years, small target detection technology has been widely used in civil and military fields. With the development of multi-spectral modal equipment, it has become a trend to improve the accuracy of small target detection by using multiple spectral modal fusion technology, such as visible light spectrum, infrared spectrum and ultraviolet spectrum. The information complementarity between multiple modal data can improve the detection accuracy, but the existing multi-spectral modal fusion has the problems of large number of parameters, computational redundancy and poor fusion effect, which leads to poor detection performance when the model is deployed at the edge.

[0003] Multimodal algorithms can be divided into pixel-level fusion, feature-level fusion, and decision-level fusion according to the level of feature fusion. Feature-level fusion has more parameters than pixel-level fusion. For example Figure 1 This is a block diagram of the feature-level fusion method of the prior art. Feature-level fusion refers to fusion at the feature representation level where data is converted into feature descriptors. This fusion method has two feature extraction backbone networks and three head networks. The total number of model parameters reaches 122 million, and the real-time frame rate is only 6FPS, which does not meet the requirements for edge device deployment.

[0004] This limits its application in resource-constrained mobile devices or real-time systems. Edge deployment mainly considers lightweight pixel-level fusion methods. However, existing pixel-level fusion methods have the problem of low fusion quality. Data of different modalities need to be converted to a unified feature space or representation when fused, which will result in the loss of some original feature information. In addition, the existence of duplicate information in different modalities will increase the computational cost and interfere with the model's extraction and understanding of key information, reducing the fusion effect.

[0005] Faced with many challenges, traditional fusion methods cannot meet the needs of real-time target detection. Developing fusion methods with high fusion quality and lightweight deployment has become an important research field. Summary of the invention

[0006] In view of the problems in the prior art that the fusion technology has poor fusion quality, high number of parameters, and is not suitable for edge device deployment, the purpose of the present invention is to overcome the above technical defects and provide a pixel-level multi-spectral fusion small target detection method and system based on mask enhancement.

[0007] In view of this, the present invention proposes a pixel-level multi-spectral fusion small target detection method based on mask enhancement, comprising:

[0008] Step (1): obtaining remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum;

[0009] Step (2): split the multispectral modal image pair and input them into the feature fusion module at the same time to obtain their respective masks, and then obtain the feature information after mask processing according to their respective masks, and perform feature fusion with the split original image to obtain the first fusion result of the single modality feature, and then perform channel splicing and merging to obtain the result after the second fusion;

[0010] Step 3) Input the result after the second fusion into the SE module, filter out the effective information features, suppress the redundant features, and obtain the pixel-level fused image;

[0011] Step 4) Input the pixel-level fused image into the lightweight detection network to obtain the detection result.

[0012] Preferably, the feature fusion module of step 2) includes two mask generation modules with the same structure, wherein the input of the first mask generation module is the feature map x of the visible light spectrum. rgb_ori , the output is the mask x of the visible light spectrum mask_rgb , the input of the second mask generation module is the feature map x of the infrared spectrum ir_ori , the output is the infrared spectrum mask x mask_ir Each of the mask generation modules includes: two convolution layers with a convolution kernel size of 3, a ReLU activation function and a Sigmoid activation function, wherein,

[0013] The first convolutional layer is used to perform preliminary feature extraction on the input data;

[0014] ReLU activation function is used to introduce nonlinear characteristics so that the network can better express features;

[0015] The second convolutional layer is used to transform the previously extracted features;

[0016] Sigmoid activation function is used to map the mask value to between 0 and 1.

[0017] Preferably, the feature fusion module in step 2) includes two multiplication layers with the same structure, which are used to filter the pixel elements in the original image according to the mask, wherein:

[0018] Mask x for the visible light spectrum mask_rgb Characteristic diagram of visible light spectrum x rgb_ori Perform element-wise multiplication, infrared spectrum mask x mask_ir Characteristic graph of infrared spectrum x ir_oriPerform element-by-element multiplication to obtain the corresponding masked feature information x rgb_masked and x ir_masked .

[0019] Preferably, the feature fusion module of step 2) includes a first fusion network, and the first fusion network includes: two addition layers with the same structure and a convolution layer with a convolution kernel size of 3; wherein,

[0020] The first addition layer is used to convert x rgb_masked With x rgb_ori Perform element-by-element addition; the first convolution layer is used to convolve the output of the first addition layer to obtain the first fusion result x of the visible light modality feature out_rgb ;

[0021] The second addition layer is used to add x ir_masked With x ir_ori Perform element-by-element addition; the second convolution layer is used to convolve the output of the second addition layer to obtain the first fusion result x of the infrared light modality feature out_ir .

[0022] Preferably, the feature fusion module in step 2) includes channel splicing and merging, which is used to out_rgb and x out_ir Splice along the channel with dimension 1 to get the result x after the second fusion out_cat .

[0023] Preferably, the SE module of step 3) includes a gate mechanism consisting of a global average pooling layer and two fully connected layers; wherein,

[0024] The global average pooling layer is used to compress the global spatial information of each channel;

[0025] The gate mechanism is used to provide a specific weight for each channel, and the weight is used to reflect the importance of the channel. By weighted filtering out redundant information and strengthening effective information, the pixel-level fused image I is obtained. f .

[0026] Preferably, the lightweight detection network in step 4) is YOLO v 8n lightweight detection network, input is the fused image I f , the output is a recognition result map with confidence and category labels.

[0027] Preferably, the lightweight detection network includes a backbone network and a head network.

[0028] On the other hand, the present invention provides a pixel-level multi-spectral fusion small target detection system based on mask enhancement, comprising:

[0029] An acquisition module, used for acquiring remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum;

[0030] The secondary fusion module is used to split the multispectral modality image pair and input them into the feature fusion module at the same time to obtain their respective masks, and then obtain the feature information after mask processing according to their respective masks, and perform feature fusion with the split original image to obtain the first fusion result of the single modality feature, and then perform channel splicing and merging to obtain the result after the second fusion;

[0031] The screening and suppression module is used to input the result after the second fusion into the SE module, screen out the effective information features, suppress the redundant features, and obtain the pixel-level fused image;

[0032] The detection output module is used to input the pixel-level fused image into the lightweight detection network to obtain the detection result.

[0033] Compared with the prior art, the advantages of the present invention are:

[0034] 1. Traditional pixel-level fusion methods use 1×1 convolution to generate masks. Simple 1×1 convolution is not enough to capture complex spatial relationships and will lose some original spatial structure information, thus affecting the quality of the mask. In contrast, our method uses two 3×3 convolution layers and a ReLU activation function in the mask generation module in the feature fusion module to generate masks, which can learn richer spatial features.

[0035] 2. In terms of mask application, traditional methods multiply the original image features by 0.5 before applying the generated mask. This scaling may affect the expressiveness of the features and the accuracy of subsequent processing. In contrast, our method directly processes the original feature map without additional scaling steps, preserving the original information of the features. In addition, our method uses the Sigmoid function in the feature fusion module to limit the mask value between 0 and 1, which helps to more finely control the importance of the features.

[0036] 3. There is some repeated or similar information between different modalities, such as background information in the image, repeated textures, etc. If the data of these two modalities are simply spliced ​​or fused, a lot of redundant information will be introduced. This redundant information will not only increase the computational cost, but may also interfere with the model's extraction and understanding of key information, reducing the fusion effect. Our method performs feature processing on each modality separately, screens out effective features, completes the first fusion, and uses the SE module to screen the feature results after the second fusion again, thereby obtaining a high-quality fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a block diagram of the existing feature-level fusion method with a huge number of parameters;

[0038] Figure 2 It is a block diagram of the pixel-level multi-spectral fusion small target detection method based on mask enhancement of the present invention;

[0039] Figure 3 This is the block diagram of the traditional pixel-level fusion method;

[0040] Figure 4 It is a block diagram of the pixel-level multi-spectral fusion method based on mask enhancement of the present invention;

[0041] Figure 5 It is a visual comparison chart of the fusion effect between the present invention and the traditional pixel-level fusion method. DETAILED DESCRIPTION

[0042] The method of the present invention comprises:

[0043] Acquiring remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum;

[0044] The multispectral modality image pair is split and input into the feature fusion module at the same time to obtain their respective masks, and then the feature information after mask processing is obtained according to their respective masks, and the feature fusion is performed with the split original image to obtain the first fusion result of the single modality feature, and then the channel splicing and merging are performed to obtain the result after the second fusion;

[0045] The result of the second fusion is input into the SE module to filter out effective information features, suppress redundant features, and obtain a pixel-level fused image;

[0046] The pixel-level fused image is input into the lightweight detection network to obtain the detection result.

[0047] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0048] Example 1

[0049] Reference Figure 2 Embodiment 1 of the present invention proposes a pixel-level multi-spectral fusion small target detection method based on mask enhancement, and the specific steps are as follows:

[0050] Step (1): read multispectral modality image pairs;

[0051] The multispectral modalities in step (1) refer to infrared and visible light spectra. Remote sensing images are collected by drones and contain a variety of shooting conditions and scenes, covering different viewing angles, altitudes, and lighting conditions.

[0052] Step (2): split the image pairs and send them to the mask generation module respectively;

[0053] The mask generation module in step (2) uses two convolutional layers with a kernel size of 3, a ReLU activation function, and a Sigmoid activation function to generate the mask. First, the first convolutional layer is used to perform preliminary feature extraction on the input data; secondly, the ReLU activation function is used to introduce nonlinear characteristics so that the network can better express the features; then, the second convolutional layer is used to further transform the previously extracted features; finally, the Sigmoid activation function is used to map the mask value to between 0 and 1 to more finely control the importance of the feature. Original feature map x rgb_ori and x ir_ori After being processed by the mask generation module, the generated mask x is obtained mask_rgb and x mask_ir .

[0054] x mask_rgb =σ Sig (Conv 3×3 (σ ReLU (Conv 3×3 (x rgb_ori ))))

[0055] x mask_ir =σ Sig (Conv 3×3 (σ ReLU (Conv 3×3 (x ir_ori ))))

[0056] Among them, σ Sig is the Sigmoid activation function; σ ReLU is the ReLU activation function; Conv 3×3 It is a convolutional layer with a kernel size of 3.

[0057] Step (3): Apply the generated mask to the image input by S2 to obtain the feature information after mask processing;

[0058] The specific implementation of the mask application in step (3) is to generate the mask x mask_rgb With the original feature x rgb_ori Perform element-by-element multiplication to achieve the original feature x according to the mask value rgb_ori The pixel elements in are filtered and processed by this module to obtain x rgb_masked , and similarly we can get x ir_masked .

[0059] Step (4): perform feature fusion on the feature information after mask processing and the original information input by S2, and obtain the first fusion result of the single modality feature through the feature fusion module;

[0060] The specific implementation of the feature fusion module in step (4) consists of two parts. First, x rgb_masked With x rgb_ori Element-by-element addition is performed to fuse the masked features with the original feature information. Secondly, the added result is sent to the convolution layer with a convolution kernel size of 3 to obtain x out_rgb . Similarly, we can get x out_ir That is, the first fusion result of each modality feature is obtained through the feature fusion module.

[0061] x out_rgb =Conv 3×3 (x rgb_ori +x rgb_ori × mask_rgb )

[0062] x out_ir =Conv 3×3 (x ir_ori +x ir_ori × mask_ir )

[0063] Step (5): perform channel splicing and merging on the results of different modal feature processing obtained in S4 to obtain the result after the second fusion;

[0064] The specific implementation method of the second fusion in step (5) is to convert x out_rgb and x out_ir Splicing is performed along the channel with dimension 1, and the feature processing results of the two different modes are fused to obtain x out_cat .

[0065]

[0066] Step (6): Send the result of S5 to the SE module to filter out effective information features, suppress redundant features, and obtain the final pixel-level fused image;

[0067] The SE module in step (6) is used to fuse step (5) to obtain x out_tat The SE module first compresses the global spatial information of each channel through the global average pooling layer, and then provides a specific weight for each channel through a gate mechanism composed of two fully connected layers. The weight is used to reflect the importance of the channel. Then, redundant information is filtered out through weighting and effective information is enhanced. Finally, the fused image I is obtained. f , used for subsequent target detection tasks.

[0068] I f =SE(x out_cat )

[0069] Step (7): Send the fused image obtained in S6 to the YOLOv8n lightweight detection network to obtain the detection result.

[0070] In step (7), the YOLOv8n lightweight detection network includes a YOLOv8 backbone network and a head network, where n represents the total depth and total width of the network, which are set to 0.33 and 0.25 to ensure the network is lightweight. The input of the detection network is the fused image I f , the output is a recognition result map with confidence and category labels.

[0071] like Figure 3 The figure shows a block diagram of a pixel-level fusion method. Pixel-level fusion refers to fusion on raw data by combining data information from different sensors in the data preprocessing stage. This fusion method has fewer parameters, but the mask generation and application are simple, which will lead to the loss of modal features. And in the final fusion, only simple splicing will lead to poor fusion quality.

[0072] like Figure 4 The figure shows a block diagram of the pixel-level multi-spectral fusion method based on mask enhancement of the present invention, corresponding to steps (2) to (6).

[0073] like Figure 5 The figure shows a visual comparison of the fusion effect between the present invention and the traditional pixel-level fusion method, indicating that the mask-enhanced pixel-level multispectral fusion method designed by the present invention can generate a fusion feature map with higher quality.

[0074] The comparison of the detection results of the present invention and the traditional pixel-level fusion method is shown in Table 1. By comparing the accuracy index mAP, it is shown that the pixel-level multispectral fusion small target detection method based on mask enhancement designed by the present invention can improve the small target detection and recognition accuracy.

[0075] Table 1 Comparison of pixel-level fusion detection results

[0076]

[0077] Table 2 shows the comparison of the total parameters of the method of the present invention and the parameters of the MGMF feature-level fusion method. By comparing the parameters, it is shown that the pixel-level multi-spectral fusion small target detection method based on mask enhancement designed by the present invention is lighter and more suitable for edge device deployment.

[0078] Table 2 Total parameters of the method of the present invention Figure 1 Comparison of parameters of feature-level fusion methods

[0079]

[0080] Example 2

[0081] Embodiment 2 of the present invention proposes a pixel-level multi-spectral fusion small target detection system based on mask enhancement, which is implemented based on the method of embodiment 1. The system includes:

[0082] An acquisition module, used for acquiring remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum;

[0083] The secondary fusion module is used to split the multispectral modality image pair and input them into the feature fusion module at the same time to obtain their respective masks, and then obtain the feature information after mask processing according to their respective masks, and perform feature fusion with the split original image to obtain the first fusion result of the single modality feature, and then perform channel splicing and merging to obtain the result after the second fusion;

[0084] The screening and suppression module is used to input the result after the second fusion into the SE module, screen out the effective information features, suppress the redundant features, and obtain the pixel-level fused image;

[0085] The detection output module is used to input the pixel-level fused image into the lightweight detection network to obtain the detection result.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention is described in detail with reference to the embodiments, it should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present invention does not depart from the spirit and scope of the technical solutions of the present invention and should be included in the scope of the claims of the present invention.

Claims

1. A pixel-level multispectral fusion small target detection method based on mask enhancement, comprising: Step (1): obtaining remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum; Step (2): split the multispectral modal image pair and input them into the feature fusion module at the same time to obtain their respective masks, and then obtain the feature information after mask processing according to their respective masks, and perform feature fusion with the split original image to obtain the first fusion result of the single modality feature, and then perform channel splicing and merging to obtain the result after the second fusion; Step 3) Input the result after the second fusion into the SE module, filter out the effective information features, suppress the redundant features, and obtain the pixel-level fused image; Step 4) Input the pixel-level fused image into the lightweight detection network to obtain the detection result.

2. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 1, characterized in that: The feature fusion module of step 2) includes two mask generation modules with the same structure, wherein the input of the first mask generation module is the feature map x of the visible light spectrum. rgb_ori , the output is the mask x of the visible light spectrum mask_rgb , the input of the second mask generation module is the feature map x of the infrared spectrum ir_ori , the output is the infrared spectrum mask x mask_ir Each of the mask generation modules includes: two convolution layers with a convolution kernel size of 3, a ReLU activation function and a Sigmoid activation function, wherein, The first convolutional layer is used to perform preliminary feature extraction on the input data; ReLU activation function is used to introduce nonlinear characteristics so that the network can better express features; The second convolutional layer is used to transform the previously extracted features; Sigmoid activation function is used to map the mask value to between 0 and 1.

3. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 2 is characterized in that: The feature fusion module of step 2) includes two multiplication layers with the same structure, which are used to filter the pixel elements in the original image according to the mask, wherein: Mask x for the visible light spectrum mask_rgb Characteristic diagram of visible light spectrum x rgb_ori Perform element-wise multiplication, infrared spectrum mask x mask_ir Characteristic graph of infrared spectrum x ir_ori Perform element-by-element multiplication to obtain the corresponding masked feature information x rgb_masked and x ir_masked .

4. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 3 is characterized in that: The feature fusion module of step 2) includes a first fusion network, and the first fusion network includes: two addition layers with the same structure and a convolution layer with a convolution kernel size of 3; wherein, The first addition layer is used to convert x rgb_masked With x rgb_ori Perform element-by-element addition; the first convolution layer is used to convolve the output of the first addition layer to obtain the first fusion result x of the visible light modality feature out_rgb ; The second addition layer is used to add x ir_masked With x ir_ori Perform element-by-element addition; the second convolution layer is used to convolve the output of the second addition layer to obtain the first fusion result x of the infrared light modality feature out_ir .

5. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 4 is characterized in that: The feature fusion module in step 2) includes channel splicing and merging, which is used to out_rgb and x out_ir Splice along the channel with dimension 1 to get the result x after the second fusion out_cat .

6. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 5, characterized in that: The SE module of step 3) includes a gate mechanism consisting of a global average pooling layer and two fully connected layers; wherein, The global average pooling layer is used to compress the global spatial information of each channel; The gate mechanism is used to provide a specific weight for each channel, and the weight is used to reflect the importance of the channel. By weighted filtering out redundant information and strengthening effective information, the pixel-level fused image I is obtained. f .

7. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 6, characterized in that: The lightweight detection network of step 4) is a YOLOv8n lightweight detection network, and the input is the fused image I f , the output is a recognition result map with confidence and category labels.

8. The method for detecting small targets by pixel-level multi-spectral fusion based on mask enhancement according to claim 7, characterized in that: The lightweight detection network includes a backbone network and a head network.

9. A pixel-level multispectral fusion small target detection system based on mask enhancement, characterized in that: include: An acquisition module, used for acquiring remote sensing image data to be processed, wherein the remote sensing image data includes a multispectral image pair of a visible light spectrum and an infrared spectrum; The secondary fusion module is used to split the multispectral modality image pair and input them into the feature fusion module at the same time to obtain their respective masks, and then obtain the feature information after mask processing according to their respective masks, and perform feature fusion with the split original image to obtain the first fusion result of the single modality feature, and then perform channel splicing and merging to obtain the result after the second fusion; The screening and suppression module is used to input the result after the second fusion into the SE module, screen out the effective information features, suppress the redundant features, and obtain the pixel-level fused image; and The detection output module is used to input the pixel-level fused image into the lightweight detection network to obtain the detection result.

Images