A cross-modal fusion target detection method based on drones

Abstract

The present invention discloses a cross-modal fusion target detection method based on drones, which fuses and complements visible light image features with infrared image features, and adaptively adjusts the modal fusion weight according to the light intensity, further solving the limitations of single modality detection of visible light or infrared images and improving the accuracy of fusion target detection. vis With infrared image I inf Each modality is fed into the corresponding branch network for feature extraction. In the backbone of the BTF (Balancing TransFusion) model, the TransFusion structure fuses and complements the features of the two modalities. Balancing units then adaptively adjust the fusion weights to form a fused feature map. The fused feature map is then fused at multiple scales to produce the final detection result.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a cross-modal fusion target detection method based on drones. Background Art

[0002] Drone target detection technology uses a drone's onboard camera to analyze images captured by the drone through image processing techniques, thereby detecting and tracking targets. This technology is a key component of drone intelligence, improving drone perception and autonomy, and providing fundamental technical support for drone applications in various fields.

[0003] However, drone-based target detection technology also faces some difficulties and challenges. For example, the images acquired by drones have complex backgrounds, changing perspectives, and uneven lighting, which can easily lead to poor image quality and introduce noise interference to target detection. At the same time, the targets in drone images have large scale variations and irregular shapes, which can easily make them difficult to locate and identify. In terms of imaging hardware, due to the imaging principles of the visible light sensor hardware itself, it is difficult to achieve high-precision detection results using only visible light images. Infrared images only rely on the target's thermal radiation ability. At the same time, they have better smoke penetration ability, are not affected by lighting conditions, and are suitable for all-weather operation. Currently, single-modality target detection technology cannot support the fusion detection of two modalities at the same time. Therefore, multimodal fusion technology is used to fuse the images of these two different modalities and design cross-modal fusion target detection to fully utilize the effective information of the two modalities, thereby improving the reliability and stability of the algorithm.

[0004] Currently, research on target detection based on the fusion of visible light and infrared images is still in its infancy. Essentially, it is a multimodal image fusion task, with fusion methods mainly divided into early fusion, late fusion, and hybrid fusion, which are suitable for different application scenarios. Since numerous studies have demonstrated that CNNs have powerful representation learning capabilities in single-modality tasks, especially image modalities, current cross-modal fusion target detection methods mostly use convolutional neural network structures, combined with manually designed fusion strategies to guide fusion. However, approaches based on manually designed fusion strategies cannot fully integrate effective information across modalities. Furthermore, previous studies have mostly focused on single scenes under low light conditions, without considering the impact of lighting factors on cross-modal fusion.

[0005] The main problems with existing object detection methods based on visible light and infrared image fusion are: First, the use of manually designed fusion strategies may lead to inadequate cross-modal information fusion. Second, different lighting conditions have different impacts on the weight of cross-modal information fusion, and lighting factors need to be considered when guiding cross-modal information fusion. Summary of the Invention

[0006] The present invention provides a cross-modal fusion target detection method based on drones to solve the problems of insufficient fusion and illumination adaptive fusion weight adjustment that may be caused by manually designed cross-modal fusion strategies.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A target detection method based on cross-modal fusion of drones, comprising the following steps:

[0009] Step 1: Convert the visible light image I vis With infrared image I inf Input the corresponding branch modes for feature extraction respectively;

[0010] Step 2: Input the intermediate features of the visible light branch and the infrared image branch into the TransFusion module to fuse the feature information, and then add the fused information to the corresponding branch. The features after information supplementation are sent to the corresponding Balancing Units structure for fusion weight balancing and then added to obtain the fused feature P;

[0011] Step 3: Repeat step 2 three times for the feature maps at different scales to obtain fused feature maps P3, P4, and P5 respectively;

[0012] Step 4: Send the fused feature map obtained in step 3 into the multi-scale fusion network for feature information complementation;

[0013] Step 5: Detect the feature map obtained in step 4 and output the result.

[0014] Furthermore, the step 1 includes the following steps:

[0015] The input visible light image and infrared image are both in RGB format. The visible light branch has the same structure as the infrared image branch. The feature extraction network includes the ConvBNSiLU layer and the C3 layer. The ConvBNSiLU layer is a combination of the convolution layer, the BN layer and the SiLU activation function. The C3 layer follows the ConvBNSiLU layer and includes a convolution block and a residual structure to accelerate feature extraction and reduce parameters.

[0016] Furthermore, the step 2 includes the following steps:

[0017] Perform cross-modal information fusion complementation and fusion weight adaptive adjustment,

[0018] I in =Contact(Seq(F R ), Seq(F I )),

[0019] I out =f TransFusion (I in ),

[0020] F' R =F R +UnSeq R (I out ),

[0021] F' I =F I +UnSeq I (I out ),

[0022] P=f Balancing Units (F′ R )+f Balancing Units (F′ I ).

[0023] Among them: I in Represents the input of the TransFusion module, Contact() represents the splicing operation, Seq() represents the serialization operation, F R With F I They represent the visible light image feature map and infrared image feature map input in the cross-modal fusion process, I out represents the total feature of cross-modal fusion, f TransFusion () represents the information fusion operation of the two modal feature maps, F' R represents the part of the total cross-modal fusion features corresponding to the visible light image features, F' I Indicates the part of the total cross-modal fusion features corresponding to the infrared image features, UnSeq R (),UnSeq I () respectively indicate that the splitting operation is performed first and then the deserialization is performed to obtain the feature maps corresponding to the visible light and infrared images, Balancing Units () performs adaptive balancing of fusion weights according to light intensity, where P is the fusion feature map obtained after performing cross-modal information fusion complementation and fusion weight adaptation at this scale;

[0024] Cross-modal information fusion complementary operation: First, the visible light image feature map F is obtained through two branch network convolution modules R and infrared image feature map F I , and then F R With F I After flattening the serialization, splicing is performed to form the input I of the TransFusion structure in After the TransFusion structure fusion features are split and deserialized, the fusion features are added to the original branches to obtain F'R With F' I , completing the cross-modal information complementation process;

[0025] Fusion weight adaptive adjustment operation: F' R With F' I Input the corresponding Balancing Units modules respectively to complete weight balancing and then add them together to obtain the fusion feature P;

[0026] The input size of the Balancing Units module is C×H×W, where C is the number of input feature map channels, W and H represent the width and height of the original image, respectively. The 1×1, C / 2 and 3×3, C / 2 of the ConvBNSiLU layer refer to the convolution kernel sizes of 1×1×C / 2 and 3×3×C / 2, respectively. p0 and p1 refer to the padding settings of 0 and 1 during the convolution process, respectively.

[0027] Furthermore, the step three includes the following steps:

[0028] Repeat step 2 three times, that is, send the feature maps of different scales in the two modal feature extraction process into the BTF structure for corresponding operations to obtain fused feature maps P3, P4, and P5 at different scales;

[0029] Furthermore, the step 4 includes the following steps:

[0030] The size of the input image to be detected is C×H×W, then the size of P3 is P4 size is P5 size is Where: C represents the number of image channels, W and H represent the width and height of the original image respectively, C1, C2 and C3 represent the number of channels of the corresponding fusion feature map respectively, and C3 is equal to twice C2, and C2 is equal to twice C1;

[0031] P5 obtains the feature map P5 by halving the number of channels through the ConvBNSiLU layer ′ , for P5 ′ Perform UPSAMPLE (up sampling) operation to transform the feature map into After resizing, channel-level contact fusion is performed with the P4 feature map to obtain The size feature map is passed through the ConvBNSiLU layer to change the number of channels to C1 to obtain the feature map P4 ′ , for feature map P4 ′ After performing UPSAMPLE operation and fusing with P3, the size is Feature map P3 ′ , P3 ′ After the residual module halves the number of channels, the feature map size is obtained PS, perform target detection on PS at this scale, and send PS to ConvBNSiLU layer to reduce the size by half to obtain a size of and compare the feature map with P4 ′ Perform channel-level contact fusion and continue to send the feature map into the residual block to obtain a feature map size of PM, perform target detection on PM at this scale, and send PM to ConvBNSiLU layer to obtain a size of The feature map is compared with P5 ′ After channel-level contact fusion, the size is obtained After the feature map is fed into the residual block, the feature map size is obtained. PL, and perform target detection on PL at this scale.

[0032] Furthermore, the step five includes the following steps:

[0033] Multiple anchor boxes of different scales are used to predict the feature map obtained in step 4. Each anchor box needs to calculate its matching degree with the target, that is, predict the category and position of the target. After predicting the category and position at each anchor box position, the non-maximum suppression algorithm (NMS) is used to delete overlapping detection results, retaining only the result with the highest confidence. Finally, the bounding box and category of the target are calculated based on the prediction results, and the detection result is output.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] In response to the problems of insufficient fusion and illumination adaptive fusion weight adjustment that may be caused by existing manually designed cross-modal fusion strategies, this paper proposes a cross-modal fusion target detection model based on drones, which realizes self-learning cross-modal fusion strategy and fusion weight adaptation under different lighting conditions, thereby improving the accuracy and robustness of cross-modal fusion target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of the present invention;

[0037] Figure 2 It is the backbone part of the dual-stream cross-modal fusion network of the present invention;

[0038] Figure 3 It is the TransFusion structure and Balancing Units module in the model of the present invention;

[0039] Figure 4 It is the multi-scale feature fusion network structure of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the embodiments.

[0041] The present invention provides a cross-modal fusion target detection method based on drones, which includes feature extraction, cross-modal information fusion and complementation, adaptive illumination information fusion weights, multi-scale feature fusion, and finally outputs the target detection result. Figure 1-4 The invention is further described in the following examples.

[0042] like Figure 1 As shown in FIG, a target detection method based on cross-modal fusion of UAV includes the following steps:

[0043] Step 1: Convert the visible light image I vis With infrared image I inf The corresponding branch modalities are input respectively for feature extraction; specifically: the input visible light image and infrared image are both in RGB format, where the visible light branch has the same structure as the infrared image branch, and the feature extraction network includes the ConvBNSiLU layer and the C3 layer, where the ConvBNSiLU layer is a combination of the convolution layer, the BN layer and the SiLU activation function. The C3 layer follows the ConvBNSiLU layer and includes a convolution block and a residual structure, which is used to accelerate feature extraction and reduce parameters.

[0044] Step 2: Input the intermediate features of the visible light branch and the infrared image branch into the TransFusion module to fuse the feature information, and then add the fused information to the corresponding branch. The features after information supplementation are sent to the corresponding Balancing Units structure for fusion weight balancing and then added to obtain the fused feature P;

[0045] Perform cross-modal information fusion complementation and fusion weight adaptive adjustment,

[0046] I in =Contact(Seq(F R ), Seq(F I )),

[0047] I out =f TransFusion (I in ),

[0048] F' R =F R +UnSeq R (I out ),

[0049] F' I =F I +UnSeq I(I out ),

[0050] P=f Balancing Units (F' R )+f Balancing Units (F' I ).

[0051] Among them: I in Represents the input of the TransFusion module, Contact() represents the splicing operation, Seq() represents the serialization operation, F R With F I They represent the visible light image feature map and infrared image feature map input in the cross-modal fusion process, I out represents the total feature of cross-modal fusion, f TransFusion () represents the information fusion operation of the two modal feature maps, F' R represents the part of the total cross-modal fusion features corresponding to the visible light image features, F I Indicates the part of the total cross-modal fusion features corresponding to the infrared image features, UnSeq R (),UnSeq I () respectively indicate that the splitting operation is performed first and then the deserialization is performed to obtain the feature maps corresponding to the visible light and infrared images, Balancing Units () performs adaptive balancing of fusion weights according to light intensity, where P is the fusion feature map obtained after performing cross-modal information fusion complementation and fusion weight adaptation at this scale;

[0052] Cross-modal information fusion complementary operation: First, the visible light image feature map F is obtained through two branch network convolution modules R and infrared image feature map F I , and then F R With F I After flattening the serialization, splicing is performed to form the input I of the TransFusion structure in After the TransFusion structure fusion features are split and deserialized, the fusion features are added to the original branches to obtain F' R With F' I , completing the cross-modal information complementation process;

[0053] Fusion weight adaptive adjustment operation: F' R With F' I Input the corresponding Balancing Units modules respectively to complete weight balancing and then add them together to obtain the fusion feature P;

[0054] like Figure 3As shown in Figure 1, the input size of the Balancing Units module is C×H×W, where C is the number of input feature map channels, W and H represent the width and height of the original image, respectively. The 1×1, C / 2 and 3×3, C / 2 of the ConvBNSiLU layer refer to the convolution kernel sizes of 1×1×C / 2 and 3×3×C / 2, respectively. p0 and p1 refer to the padding settings of 0 and 1 during the convolution process, respectively.

[0055] Step 3: Repeat step 2 three times for the feature maps at different scales, that is, send the feature maps of different scales in the two modal feature extraction processes into the BTF structure for corresponding operations, and obtain fused feature maps P3, P4, and P5 respectively;

[0056] Step 4: Send the fused feature map obtained in step 3 into the multi-scale fusion network for feature information complementation:

[0057] The size of the input image to be detected is C×H×W, then the size of P3 is P4 size is P5 size is Where: C represents the number of image channels, W and H represent the width and height of the original image respectively, C1, C2 and C3 represent the number of channels of the corresponding fusion feature map respectively, and C3 is equal to twice C2, and C2 is equal to twice C1;

[0058] P5 is passed through the ConvBNSiLU layer to halve the number of channels to obtain the feature map P5', and the UPSAMPLE (up-sampling) operation is performed on P5' to transform the feature map into After resizing, channel-level contact fusion is performed with the P4 feature map to obtain The feature map of size is obtained by passing the ConvBNSiLU layer to change the number of channels to C1 to obtain the feature map P4'. The UPSAMPLE operation is performed on the feature map P'4 and fused with P3 to obtain a size of Feature map P3 ′ , P3 ′ After the residual module halves the number of channels, the feature map size is obtained PS, perform target detection on PS at this scale, and send PS to ConvBNSiLU layer to reduce the size by half to obtain a size of and compare the feature map with P4 ′ Perform channel-level contact fusion and continue to send the feature map into the residual block to obtain a feature map size of PM, perform target detection on PM at this scale, and send PM to ConvBNSiLU layer to obtain a size of The feature map is compared with P5 ′After channel-level contact fusion, the size is obtained After the feature map is fed into the residual block, the feature map size is obtained. PL, and perform target detection on PL at this scale.

[0059] Step 5: Detect the feature map obtained in step 4 and output the result:

[0060] Multiple anchor boxes of different scales are used to predict the feature map obtained in step 4. Each anchor box needs to calculate its matching degree with the target, that is, predict the category and position of the target. After predicting the category and position at each anchor box position, the non-maximum suppression algorithm (NMS) is used to delete overlapping detection results, retaining only the result with the highest confidence. Finally, the bounding box and category of the target are calculated based on the prediction results, and the detection result is output.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A target detection method based on cross-modal fusion of drones, characterized by: The following steps are involved: Step 1: Convert the visible light image I vis With infrared image I inf Input the corresponding branch modes for feature extraction respectively; Step 2: Input the intermediate features of the visible light branch and the infrared image branch into the TransFusion module to fuse the feature information, and add the fused information to the corresponding branch. The features after information supplementation are sent to the corresponding BalancingUnits structure for fusion weight balancing and then added to obtain the fused feature P; Step 3: Repeat step 2 three times for the feature maps at different scales to obtain fused feature maps P3, P4, and P5 respectively; Step 4: Send the fused feature map obtained in step 3 into the multi-scale fusion network for feature information complementation; Step 5: Detect the feature map obtained in step 4 and output the result; The step 1 comprises the following steps: The input visible light image and infrared image are both in RGB format. The visible light branch has the same structure as the infrared image branch. The feature extraction network includes ConvBNSiLU layer and C3 layer. The ConvBNSiLU layer is a combination of convolution layer, BN layer and SiLU activation function. The C3 layer follows the ConvBNSiLU layer and includes convolution blocks and residual structure to accelerate feature extraction and reduce parameters. The second step comprises the following steps: Perform cross-modal information fusion complementation and fusion weight adaptive adjustment, I in =Contact(Seq(F R ),Seq(F I )), I out =f TransFusion (I in ), F’ R =F R +UnSeq R (I out ), F’ I =F I +UnSeq I (I out ), P=f Balancing Units (F’ R )+f Balancing Units (F’ I ). Among them: I in Represents the input of the TransFusion module, Contact() represents the splicing operation, Seq() represents the serialization operation, F R With F I They represent the visible light image feature map and infrared image feature map input in the cross-modal fusion process, I out represents the total feature of cross-modal fusion, f TransFusion () represents the information fusion operation of the two modal feature maps, F' R represents the part of the total cross-modal fusion features corresponding to the visible light image features, F' I Indicates the part of the total cross-modal fusion features corresponding to the infrared image features, UnSeq R (),UnSeq I () respectively indicate that the splitting operation is performed first and then the deserialization is performed to obtain the feature maps corresponding to the visible light and infrared images, Balancing Units () performs adaptive balancing of fusion weights according to light intensity, where P is the fusion feature map obtained after performing cross-modal information fusion complementation and fusion weight adaptation at this scale; Cross-modal information fusion complementary operation: First, the visible light image feature map F is obtained through two branch network convolution modules R and infrared image feature map F I , and then F R With F I After flattening the serialization, splicing is performed to form the input I of the TransFusion structure in After the TransFusion structure fusion features are split and deserialized, the fusion features are added to the original branches to obtain F' R With F' I , completing the cross-modal information complementation process; Fusion weight adaptive adjustment operation: F' R With F' I Input the corresponding Balancing Units modules respectively to complete weight balancing and then add them together to obtain the fusion feature P; The input size of the Balancing Units module is C×H×W, where C is the number of input feature map channels, W and H represent the width and height of the original image, respectively. The 1×1, C / 2 and 3×3, C / 2 of the ConvBNSiLU layer refer to the convolution kernel sizes of 1×1×C / 2 and 3×3×C / 2, respectively. p0 and p1 refer to the padding settings of 0 and 1 during the convolution process, respectively.

2. The target detection method based on cross-modal fusion of drones according to claim 1 is characterized in that: The step three comprises the following steps: Repeat step 2 three times, that is, send the feature maps of different scales in the two modal feature extraction processes into the BTF structure for corresponding operations to obtain fused feature maps P3, P4, and P5 at different scales.

3. The target detection method based on cross-modal fusion of drones according to claim 1 is characterized in that: The step 4 includes the following steps: The size of the input image to be detected is C×H×W, then the size of P3 is P4 size is P5 size is Where: C represents the number of image channels, W and H represent the width and height of the original image respectively, C1, C2 and C3 represent the number of channels of the corresponding fusion feature map respectively, and C3 is equal to twice C2, and C2 is equal to twice C1; P5 is obtained by halving the number of channels through the ConvBNSiLU layer to obtain the feature map P5′, and the UPSAMPLE upsampling operation is performed on P5′ to transform the feature map into After resizing, channel-level contact fusion is performed with the P4 feature map to obtain The feature map of size is obtained by passing the ConvBNSiLU layer to change the number of channels to C1 to obtain the feature map P4′. The UPSAMPLE operation is performed on the feature map P4′ and fused with P3 to obtain a size of The feature map P3′ is obtained by halving the number of channels in the residual module. PS, perform target detection on PS at this scale, and send PS to ConvBNSiLU layer to reduce the size by half to obtain a size of The feature map is fused with P4′ at the channel level, and the feature map is fed into the residual block to obtain a feature map of size PM, perform target detection on PM at this scale, and send PM to ConvBNSiLU layer to obtain a size of The feature map is fused with P5′ at the channel level to obtain a feature map of size After the feature map is fed into the residual block, the feature map size is obtained. PL, and perform target detection on PL at this scale.

4. The target detection method based on cross-modal fusion of drones according to claim 1 is characterized in that: The step five comprises the following steps: Multiple anchor boxes of different scales are used to predict the feature map obtained in step 4. Each anchor box needs to calculate its matching degree with the target, that is, predict the category and position of the target. After predicting the category and position at each anchor box position, the non-maximum suppression algorithm NMS is used to delete overlapping detection results and only retain the result with the highest confidence. Finally, the bounding box and category of the target are calculated based on the prediction results, and the detection result is output.

Images