Multi-scale feature fusion unmanned aerial vehicle target detection method for bad weather

Abstract

The invention discloses a multi-scale feature fusion unmanned aerial vehicle target detection method for bad weather, and the method comprises the steps: collecting multi-modal data, and inputting the multi-modal data into a target detection model; the target detection model comprises a first modal feature extraction network, a second modal feature extraction network, a multi-scale pyramid feature fusion network, a cross-modal fusion network and a target detection head network; performing feature extraction on the visible light image data by using a first modal feature extraction fusion network to obtain visible light multi-scale features; performing feature extraction on the thermal infrared image data by using a second modal feature extraction fusion network to obtain thermal infrared multi-scale features; processing the visible light multi-scale features and the thermal infrared multi-scale features by using a multi-scale pyramid feature fusion network to obtain a visible light feature map and a thermal infrared feature map; and fusing the visible light feature map and the thermal infrared feature map by using a cross-modal fusion network to obtain a fused feature map for target detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-scale feature fusion unmanned aerial vehicle target detection method for bad weather. Background Art

[0002] Drones are increasingly used in various complex environments, especially in military surveillance, disaster relief, agricultural monitoring and other fields. However, in practical applications, bad weather (such as fog, heavy rain, strong wind, snow, etc.) poses a severe challenge to the target detection capability of drones. Traditional target detection methods mostly rely on a single visual sensor, such as an infrared camera or a visible light camera, but the performance of these sensors is often significantly affected under bad weather conditions, resulting in poor detection results. Therefore, how to effectively fuse multimodal data from different sensors to improve the accuracy of drone target detection in bad weather is an urgent problem to be solved.

[0003] At present, at home and abroad, the research on multi-scale feature fusion UAV target detection methods for bad weather is constantly developing, and significant progress has been made in multimodal data fusion technology. Many studies focus on using convolutional neural networks (CNN) and attention mechanisms to fuse visible light images, infrared images, millimeter wave radar data, etc. to cope with bad weather conditions. Some scholars have also proposed a multimodal convolutional neural network (MMCNN) model, which can improve the target detection performance of UAVs in complex environments such as foggy days and nights. There are also research teams that have developed improved models based on target detection algorithms such as YOLO and Faster R-CNN, which improve the accuracy and robustness of detection by combining multimodal data. For example, some studies have input multimodal data into a deep fusion network to achieve effective detection of targets in rainy and foggy environments. For example, in the relevant technology, a patent application document with publication number CN118230117A proposes a camouflaged target detection method based on adaptive fusion of attention mechanism. This method uses the optical information and infrared information of the camouflaged target to achieve effective detection of camouflaged targets based on image feature-level fusion. The patent application document with publication number CN116311363A proposes a RGB-infrared multi-source image target detection method based on dynamic network feature fusion and YOLOv5. This method is based on the single-stage visible light YOLOv5 target detection model, designs a multi-source feature extraction module, and realizes multi-scale feature extraction of RGB-infrared images; inspired by the attention mechanism and the dynamic network of the expert model, a dynamic network multi-source feature fusion module is constructed to realize the dynamic fusion of multi-source features at the channel level, spatial level and contextual long-distance connections, thereby improving the robustness of the model.

[0004] However, research shows that it is very difficult to choose an appropriate fusion strategy when fusing multimodal data. This is mainly because multimodal data fusion involves differences in physical properties, difficulty in spatiotemporal alignment, differences in signal quality and noise, information redundancy and complementarity, complexity of fusion methods, computing resource requirements, and special requirements of application scenarios. These factors together constitute a highly complex challenge that requires in-depth research and optimization at multiple levels, including theory, algorithms, and practice.

[0005] Fusion methods include early fusion (feature-level fusion), mid-term fusion (network-level fusion) and late fusion (decision-level fusion) according to different application scenarios, among which: Early fusion refers to the direct fusion of original data at the data input stage. This method can utilize the low-level features of each modal data, but may cause information loss or poor fusion effect due to the heterogeneity of the data and differences in noise characteristics. Mid-term fusion refers to the fusion of feature maps in the middle layer of the neural network. This method can find the best fusion method through the learning ability of the network while retaining the features of each modality, but the network design is highly complex and requires a lot of experiments and parameter adjustments. Late fusion refers to the fusion of the final results after the independent processing of each modality data. This method can utilize the strengths of each modality, but due to the lack of interaction between different modalities, it may not be able to fully utilize the complementarity of multimodal data. Summary of the invention

[0006] The technical problem to be solved by the present invention is how to improve the accuracy and robustness of target detection in harsh environments.

[0007] The present invention solves the above technical problems by the following technical means:

[0008] A multi-scale feature fusion UAV target detection method for bad weather is proposed, the method includes:

[0009] Collecting multimodal data, wherein the multimodal data includes visible light image data and thermal infrared image data;

[0010] Inputting the multimodal data into a target detection model, wherein the target detection model includes a first modality feature extraction network, a second modality feature extraction network, a multi-scale pyramid feature fusion network, a cross-modality fusion network, and a target detection head network;

[0011] Using the first modality feature extraction fusion network to perform multi-scale feature extraction on visible light image data to obtain visible light multi-scale features;

[0012] The second modality feature extraction and fusion network is used to extract multi-scale features from thermal infrared image data to obtain thermal infrared multi-scale features;

[0013] The visible light multi-scale features and the thermal infrared multi-scale features are processed respectively by using a multi-scale pyramid feature fusion network to obtain a visible light feature map and a thermal infrared feature map;

[0014] The visible light feature map and the thermal infrared feature map are fused using a cross-modal fusion network to obtain a fused feature map;

[0015] The target detection head network is used to perform target detection on the fused feature map to obtain the target detection result.

[0016] Furthermore, the first modality feature extraction network and the second modality feature extraction network both adopt a hierarchical feature extraction network, and the hierarchical feature extraction network includes a low-level convolutional subnetwork and a high-level convolutional subnetwork connected in sequence;

[0017] The low-level convolutional subnetwork includes sequentially connected convolutional layers Conv1 and Conv2, the high-level convolutional subnetwork includes sequentially connected convolutional layers Conv3, Conv4 and Conv5, and the convolutional layer Conv2 is connected to the convolutional layer Conv3;

[0018] The low-level convolutional subnetwork and the high-level convolutional subnetwork output low-level feature maps and high-level feature maps respectively.

[0019] Furthermore, the hierarchical feature extraction network adopts a deep convolutional neural network.

[0020] Further, the multi-scale pyramid feature fusion network includes a first feature pyramid sub-network and a multi-scale feature fusion sub-network connected in sequence;

[0021] The first feature pyramid subnetwork is used to perform upsampling on the high-level feature map and combine it with the corresponding low-level feature map to obtain a multi-scale feature map corresponding to the visible light image data or the thermal infrared image data;

[0022] The multi-scale feature fusion sub-network is used to process and fuse the feature information at each scale in the multi-scale feature map to obtain the visible light feature map or the thermal infrared feature map.

[0023] Further, the multi-scale feature map includes a high-resolution feature map, a medium-resolution feature map and a low-resolution feature map, and the multi-scale feature fusion sub-network includes a first fusion layer, a second fusion layer and a third fusion layer;

[0024] The first fusion layer is used to perform convolution and ReLU activation on the high-resolution feature map to enhance local detail information, thereby obtaining a detail feature map;

[0025] The second fusion layer is used to perform convolution and ReLU activation on the medium resolution feature map to enhance the target structure information and obtain the target feature map;

[0026] The third fusion layer is used to perform convolution and ReLU activation on the low-resolution feature map to extract global semantic information and obtain a semantic feature map;

[0027] A weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

[0028] Further, performing a weighted sum operation on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map includes:

[0029] Assigning corresponding weights to the detail feature map, the target feature map, and the semantic feature map through an attention mechanism;

[0030] According to the corresponding weights, a weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

[0031] Furthermore, the assigning corresponding weights to the detail feature map, the target feature map, and the semantic feature map through an attention mechanism includes:

[0032] The channel attention mechanism and spatial attention mechanism are used to assign weights to different feature maps in the channel dimension and spatial dimension respectively.

[0033] Furthermore, the cross-modal fusion network includes a second feature pyramid subnetwork, an attention mechanism, a convolutional layer and a fully connected layer connected in sequence, the visible light feature map and the thermal infrared feature map are respectively used as inputs of the second feature pyramid subnetwork, and the fully connected layer outputs the fused feature map.

[0034] Furthermore, the target detection head network adopts the YOLOv8 target detection network and detects the position of the target object through an adaptive anchor frame generation strategy.

[0035] Furthermore, the method further comprises:

[0036] The target detection model is trained, and the loss function used in the training process is a joint loss function, which is expressed as follows:

[0037] L total =α·L cls +β·L loc +γ·L reg

[0038] Where, L cls is the classification loss function, L loc is the positioning loss function, L reg is the regularized loss function, and α, β, and γ are the corresponding weight coefficients respectively.

[0039] The advantages of the present invention are:

[0040] (1) The present invention can effectively fuse features from different scales by introducing a feature pyramid network into a multi-scale pyramid feature fusion network, ensuring that the network has strong adaptability when detecting targets of different sizes. In drone target detection, a small drone may only occupy a few pixels in the distance, while a large drone in the near distance occupies a larger area. Multi-scale feature fusion can accurately detect targets of these different scales at the same time, and the fused multi-scale features contain not only detail information but also semantic information, thereby enhancing the expressiveness of the features and improving the accuracy and robustness of detection.

[0041] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a multi-scale feature fusion UAV target detection method for bad weather conditions proposed by an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of the network structure of a target detection model in one embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of the structure of a feature extraction network in one embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of the structure of a multi-scale pyramid feature fusion network in one embodiment of the present invention;

[0046] Figure 5 It is a schematic diagram of the structure of a cross-modal fusion network in one embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1 to Figure 2 As shown, this embodiment proposes a multi-scale feature fusion UAV target detection method for bad weather, and the method includes the following steps:

[0049] S10, collecting multimodal data, where the multimodal data includes visible light image data and thermal infrared image data;

[0050] S20, inputting the multimodal data into a target detection model, wherein the target detection model includes a first modality feature extraction network, a second modality feature extraction network, a multi-scale pyramid feature fusion network, a cross-modality fusion network and a target detection head network;

[0051] S30, using the first modal feature extraction fusion network to perform multi-scale feature extraction on the visible light image data to obtain visible light multi-scale features;

[0052] S40, using the second modality feature extraction fusion network to perform multi-scale feature extraction on the thermal infrared image data to obtain thermal infrared multi-scale features;

[0053] S50, using a multi-scale pyramid feature fusion network to process the visible light multi-scale features and the thermal infrared multi-scale features respectively to obtain a visible light feature map and a thermal infrared feature map;

[0054] S60, using a cross-modal fusion network to fuse the visible light feature map and the thermal infrared feature map to obtain a fused feature map;

[0055] S70. Use the target detection head network to perform target detection on the fused feature map to obtain a target detection result.

[0056] This embodiment can effectively fuse features from different scales by introducing a feature pyramid network into a multi-scale pyramid feature fusion network, ensuring that the network has strong adaptability when detecting targets of different sizes. In drone target detection, a small drone may only occupy a few pixels in the distance, while a large drone in the near distance occupies a larger area. Multi-scale feature fusion can accurately detect targets of these different scales at the same time, and the fused multi-scale features contain not only detail information but also semantic information, thereby improving the expressiveness of the features and the accuracy and robustness of the detection.

[0057] Specifically, the step S10: collecting multimodal data, wherein the multimodal data includes visible light image data and thermal infrared image data, specifically includes:

[0058] Visible light camera and infrared camera are used to collect visible light image data and thermal infrared image data respectively.

[0059] In actual operation, each step of multimodal sensor configuration and data acquisition requires precise control to ensure the quality and consistency of the final data. This embodiment can lay a solid foundation for subsequent multimodal feature fusion and target detection by selecting sensors, performing precise calibration and synchronization, rationally planning the flight path, and strictly monitoring the data acquisition process.

[0060] Specifically:

[0061] (1) Multimodal sensor selection and configuration

[0062] In case of bad weather, a combination of visible light camera and infrared camera is used to configure a multimodal sensor, where the visible light camera is used to capture visual images and is suitable for target detection during the day and under good lighting conditions. The infrared camera can detect heat sources at night or in low visibility conditions (such as smoke, rain and snow), and is suitable for detecting heat-generating objects such as human bodies and vehicles.

[0063] Then, a calibration method including a checkerboard method (applicable to cameras) and synchronous calibration of camera images is used to perform precise calibration (for example, intrinsic and extrinsic calibration) between multimodal sensors to ensure accurate alignment of the data. Since the data of different sensors are usually in different coordinate systems, this embodiment converts the data of all sensors to the same world coordinate system through external parameter calibration.

[0064] Through the unified clock (such as GPS timestamp), the hardware time synchronization of data from different sensors is carried out to ensure that each sensor collects data at the same time point. Specifically, a time synchronization module (such as PX4 or RTK-GPS) is used to achieve accurate synchronization. The collected data is time-aligned to deal with the problems of time delay (latency) and inconsistent sampling frequency, and the software time synchronization is carried out to ensure the time synchronization of the data.

[0065] The image data and thermal imaging data are spatially aligned using feature point-based image registration, and the multimodal data are jointly aligned using SLAM (Simultaneous Localization and Mapping) technology.

[0066] Furthermore, during the data collection process, the quality of sensor data is monitored in real time, data anomalies (such as noise and distortion) are detected, and preliminary processing such as denoising and completion is performed. The collected multimodal data is stored in a storage device (such as a solid-state drive) and can be transmitted to the control station in real time through a wireless communication module (such as 5G, WiFi) for further analysis.

[0067] It should be noted that the storage device records the original multimodal data, and classifies and backs up the data according to task requirements to prevent data loss. This embodiment specifically uses wireless communication technology to transmit the collected data to a remote server for cloud storage and backup, and uses data encryption and redundant backup technology to ensure data security and integrity.

[0068] As a further preferred technical solution, after collecting the multimodal data, the method further includes: performing a preprocessing operation on the collected multimodal data, specifically including the following steps:

[0069] (1) Use statistical anomaly detection, filtering and denoising in signal processing technology to filter and clean the collected data to remove noise, redundant information and erroneous data.

[0070] (2) Convert the data from different sensors into a unified JPEG format. For infrared images, convert the thermal map into a grayscale image or a pseudo-color image for subsequent processing and fusion.

[0071] (3) Before the data is input into the target detection model, necessary preprocessing and enhancement are performed. The present invention uses image contrast enhancement and normalization processing of thermal imaging data.

[0072] (4) Perform geometric transformations such as flipping, rotating, scaling, and cropping on the image to expand the data set and enhance the generalization ability of the model. Adjust image parameters such as brightness, contrast, and saturation to simulate different lighting conditions and improve the robustness of the model. Add Gaussian noise, salt and pepper noise, etc. to the image to improve the robustness of the model in a noisy environment.

[0073] Specifically, after collecting and processing the historical multimodal data in the above manner, this embodiment divides the data set into a training set, a validation set, and a test set in a ratio (such as 70:15:15) for training and testing the target detection model to ensure the fairness and effectiveness of the model's training, validation, and testing.

[0074] It should be noted that the specific steps of data preprocessing in this embodiment involve data cleaning, format conversion, data enhancement, normalization and standardization, data segmentation and packaging, and data verification and inspection, to ensure that the preprocessed data can provide high-quality input for multimodal feature fusion and target detection. Through precise preprocessing, the training efficiency, detection accuracy and generalization ability of the model can be significantly improved.

[0075] As a further preferred technical solution, Figure 3 As shown, the first modality feature extraction network and the second modality feature extraction network both adopt a hierarchical feature extraction network, and the hierarchical feature extraction network includes a low-level convolutional subnetwork and a high-level convolutional subnetwork connected in sequence;

[0076] The low-level convolutional subnetwork includes sequentially connected convolutional layers Conv1 and Conv2, the high-level convolutional subnetwork includes sequentially connected convolutional layers Conv3, Conv4 and Conv5, and the convolutional layer Conv2 is connected to the convolutional layer Conv3;

[0077] The low-level convolutional subnetwork and the high-level convolutional subnetwork output low-level feature maps and high-level feature maps respectively.

[0078] Specifically, the feature extraction network adopts a convolutional neural network (CNN) or a deep convolutional neural network, in which the feature extraction network consists of 5 convolutional layers and a pooling layer to extract the basic features of the input image, including a low-level convolutional subnetwork (referring to the convolutional layers of the first few layers of the feature extraction network, which extract low-level features of the image, such as edges, textures, etc.) and a high-level convolutional subnetwork (referring to the last few layers of the feature extraction network, which mainly extract high-level semantic features, which are more abstract and can describe more complex patterns and objects). Through the DCNN hierarchical feature extraction, low-level to high-level features are gradually aggregated, ensuring that meaningful features can be extracted at different scales and complex backgrounds, thereby enhancing the robustness and accuracy of the model.

[0079] The feature extraction network extracts features layer by layer through 5 convolutional layers, ReLU activation functions, pooling layers, etc. The low-level convolutional subnetwork is the first two layers of these convolutional layers, including Conv1 and Conv2 multiple convolution and pooling operations, which are used to extract low-level features and extract more detailed features. The high-level convolutional subnetwork after the low-level convolutional subnetwork includes Conv3, Conv4, Conv5 multiple convolutional layers and pooling layers. The outputs of these layers usually have higher semantic information and represent more complex features.

[0080] The lower convolution layers are usually composed of smaller convolution kernels (such as 3×3 or 5*5). These layers can extract detailed features of the input image, such as edges, corners, textures, etc. These features are especially important for the detection of small targets because they are more obvious in high-resolution feature maps. In the high-level convolutional subnetwork, as the network depth increases, the receptive field of the convolution layer gradually expands, and the high-level feature extractor mainly captures global semantic information, such as the category, shape, and position of the object. These features are usually represented in low-resolution feature maps and are particularly important for detecting large targets and recognizing complex information in the background.

[0081] Furthermore, since visible light images contain rich detail information (such as edges and textures), the convolution kernel size in the first modal feature extraction network is set to 3×3. Small convolution kernels can better extract edge and texture features, while larger convolution kernels are used to extract more global patterns.

[0082] Since infrared images mainly reflect temperature distribution and have greater regional consistency but less detailed information, the convolution kernel size in the second modal feature extraction network is set to 5×5. Since infrared images are more dependent on global information such as temperature gradients, larger convolution kernels can better capture these features, while small convolution kernels are used to extract possible local hotspots or temperature differences.

[0083] Preferably, the optimization mechanism of this embodiment for bad weather can add specific filters or modules in the low-level convolutional layers to extract key features that can cope with weather conditions such as heavy fog, rain and snow.

[0084] Bad weather often leads to a decline in sensor data quality, such as blurred images or increased noise. This embodiment optimizes the feature extraction network so that it can still extract effective features under low visibility and complex weather conditions, thereby better suppressing background noise and reducing false alarm rates. At the same time, it ensures high sensitivity to the target and reduces the missed alarm rate, thereby improving detection accuracy in bad weather conditions.

[0085] Preferably, in the design of the convolution layer in the feature extraction network, this embodiment can introduce new convolution techniques such as grouped convolution, deformable convolution, and dilated convolution to enhance the network's adaptability to irregular targets or complex backgrounds. Among them, technologies such as deformable convolution allow the network to flexibly adjust the receptive field, enhance the detection capability of irregularly shaped targets, and improve spatial perception. Dilated convolution expands the receptive field by adjusting the sampling density, can effectively process sparse information, and helps to extract more representative features in bad weather.

[0086] Preferably, in order to improve the training stability of the network and prevent overfitting, this embodiment introduces Batch Normalization and Dropout layers after each convolutional layer in the feature extraction network.

[0087] As a further preferred technical solution, Figure 4 As shown, the multi-scale pyramid feature fusion network includes a first feature pyramid sub-network and a multi-scale feature fusion sub-network connected in sequence;

[0088] The first feature pyramid subnetwork is used to perform upsampling on the high-level feature map and combine it with the corresponding low-level feature map to obtain a multi-scale feature map corresponding to the visible light image data or the thermal infrared image data;

[0089] The multi-scale feature fusion sub-network is used to process and fuse the feature information at each scale in the multi-scale feature map to obtain the visible light feature map or the thermal infrared feature map.

[0090] Specifically, the first feature pyramid subnetwork (FPN) is used to fuse high-level feature maps with low-level feature maps. FPN achieves multi-scale feature fusion by upsampling and lateral connections of features at different layers. In FPN, the calculation of multi-scale features can usually be expressed as:

[0091] P i =Conv 1×1(C i )+Upsample(P i+1 )

[0092] Where P i represents the output of the feature pyramid layer at layer i, C i Represents the feature map extracted by the convolution of the corresponding layer, Conv 1×1 It represents a 1*1 convolution operation, which is used to reduce the number of channels. Upsample represents an upsampling operation, which is used to upsample the high-level feature map to the same spatial resolution as the low-level feature map.

[0093] It should be noted that this embodiment introduces a feature pyramid network (FPN) in the feature extraction network, which can effectively fuse features from different scales. This design ensures that the network has strong adaptability when detecting targets of different sizes and can cope with targets of different scales: by fusing low-level and high-level features, the network can process small targets and large targets at the same time. For example, in drone target detection, a small drone may only occupy a few pixels in the distance, while a large drone in the near distance occupies a larger area. Multi-scale feature fusion can accurately detect these targets of different scales at the same time. Moreover, the fused multi-scale features not only contain detail information (low-level features), but also semantic information (high-level features), thereby improving the expressiveness of the features and improving the accuracy and robustness of the detection.

[0094] It should be noted that FPN achieves the fusion of multi-scale features by upsampling the high-level feature maps of the network and combining them with the corresponding low-level feature maps, which can effectively utilize features of different scales. Since the high-level feature maps provide global semantic information and the low-level feature maps provide rich detail information, FPN can process small and large targets at the same time after fusing these features, thus improving the detection accuracy. FPN is particularly suitable for processing complex scenes in bad weather, because it can use local details and global semantic information at the same time, avoiding missed targets due to scale changes.

[0095] One of the design features of FPN is the top-down feature propagation mechanism, that is, high-level feature maps are fused with low-level feature maps step by step through upsampling. This design ensures that each layer can receive global information from the high-level while retaining the detailed information of the low-level. Through this top-down feature propagation and fusion, FPN is able to generate a set of multi-scale feature maps with rich semantic information, thereby enhancing the model's ability to detect objects of different sizes, especially in bad weather conditions.

[0096] It should be noted that this embodiment combines the deep convolutional neural network with the feature pyramid network, uses DCNN to extract features at different levels, captures multi-scale information, and then uses FPN to effectively fuse the multi-level features extracted by DCNN to ensure that the detection results take into account both details and global semantics.

[0097] As a further preferred technical solution, the multi-scale feature map includes a high-resolution feature map, a medium-resolution feature map and a low-resolution feature map, and the multi-scale feature fusion subnetwork includes a first fusion layer, a second fusion layer and a third fusion layer;

[0098] The first fusion layer is used to perform convolution and ReLU activation on the high-resolution feature map to enhance local detail information, thereby obtaining a detail feature map;

[0099] The second fusion layer is used to perform convolution and ReLU activation on the medium resolution feature map to enhance the target structure information and obtain the target feature map;

[0100] The third fusion layer is used to perform convolution and ReLU activation on the low-resolution feature map to extract global semantic information and obtain a semantic feature map;

[0101] A weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

[0102] Further, performing a weighted sum operation on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map includes:

[0103] Assigning corresponding weights to the detail feature map, the target feature map, and the semantic feature map through an attention mechanism;

[0104] According to the corresponding weights, a weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

[0105] This embodiment introduces a channel attention mechanism (such as the Squeeze-and-Excitation module of SENet) to assign weights to feature maps of different modalities in the channel dimension. By weighting the importance of each channel, the most representative features of different modalities can be highlighted. Spatial fusion is then performed in the spatial dimension, fusing the spatial distribution information of the feature map, and weighting the features at different positions through the spatial attention mechanism, especially distinguishing the target area from the background area, and enhancing the feature expression of the target area.

[0106] Specifically, suppose that the feature maps of different scales generated by FPN are P i, the attention mechanism assigns a weight w to each feature map i , the weighted feature map P i weighted It can be expressed as:

[0107] P i weighted =w i ·P i

[0108] Weight w i It can be dynamically calculated through the attention mechanism. A common way is to obtain the global feature vector F of the feature map through global average pooling (GAP) i , and then through a fully connected layer and Softmax function to get:

[0109]

[0110] Among them, F i Represents the global eigenvector F i Nonlinear transformation of θ (e.g., through a fully connected layer).

[0111] Finally, at the output of the feature extraction network, the weighted sum of feature maps of different scales is obtained:

[0112]

[0113] For example, the multi-scale feature maps output by the first feature pyramid network are P1 / 2 (high-resolution feature map), P2 / 3 (medium-resolution feature map) and P3 / 4 (low-resolution feature map), which are passed to different fusion layers of the multi-scale feature fusion subnetwork FFN. Each fusion layer performs convolution and activation operations to further process the feature information at each scale. The processed feature maps are integrated into a fused feature map through splicing or weighted sum operations. Then, suppose the feature maps output from FPN are P{1 / 2}, P{2 / 3}, P{3 / 4}, and the feature map outputs after each fusion layer are F1, F2, and F3 respectively. The convolution operation of these feature maps can be expressed as:

[0114] F i =ReLU(Conv(P i ))

[0115] A fusion operation (such as a weighted sum) can be expressed as:

[0116] P final =w1·F1+w2·F2+w3·F3

[0117] Among them, w1, w2, w3 are corresponding weighting coefficients used to balance the contribution of feature maps of different scales.

[0118] It should be noted that the multi-scale feature fusion sub-network integrates feature maps of different scales, allowing the model to focus on both detail features and global semantic information, thereby enhancing the ability to detect multi-scale targets. By introducing the weighted sum strategy, the network can dynamically adjust the weights of feature maps of different scales based on the feedback of training data. This mechanism enables the model to adapt to the features of different scenes and targets, further improving detection accuracy, significantly improving the model's target detection performance in complex environments, and improving the model's accuracy and robustness, making it perform well in complex conditions such as bad weather.

[0119] Furthermore, after obtaining the visible light feature map or thermal infrared feature map, the feature map is deeply processed through convolution operation to further extract the features most relevant to the detection task. After the convolution layer, the feature map may be compressed and classified through a fully connected layer to provide input for the final cross-modal fusion network.

[0120] As a further preferred technical solution, Figure 5 As shown, the cross-modal fusion network includes a second feature pyramid subnetwork, an attention mechanism, a convolutional layer and a fully connected layer connected in sequence, the visible light feature map and the thermal infrared feature map are respectively used as inputs of the second feature pyramid subnetwork, and the fully connected layer outputs the fused feature map.

[0121] As a further preferred technical solution, the target detection head network adopts the YOLOv8 target detection network and detects the position of the target object through an adaptive anchor frame generation strategy.

[0122] It should be noted that after the multimodal feature fusion is completed, the present embodiment has obtained fused multi-scale feature maps, which contain information extracted from different sensors and different scales, and can provide rich input data for target detection and positioning. These feature maps will be input into the YOLOv8 target detection network, which can complete the positioning and classification of the target in a single forward pass.

[0123] The target detection network usually generates a series of anchor boxes in the image. These anchor boxes are predefined rectangular boxes covering different areas of the image and are used to detect the location of the target object. In order to improve the detection accuracy under adverse weather conditions, this embodiment adopts an adaptive anchor box generation strategy: the adaptive anchor box generation strategy dynamically adjusts the size and proportion of the anchor box to better adapt to targets of different scales and shapes, thereby improving the accuracy and recall rate of target detection. This strategy mainly relies on the multi-scale information of the feature map and the actual distribution of the target for optimization. The specific implementation steps are:

[0124] (1) Initial anchor box generation: On the feature map of each scale, a set of anchor boxes with different sizes and ratios are generated.

[0125] Specifically, on the feature map of each scale, multiple basic anchor boxes are pre-set. The sizes and proportions of these anchor boxes are set according to experience or prior knowledge. The size and proportion of the initial anchor boxes usually cover a certain range to adapt to the possible target scales.

[0126] (2) Feature map analysis: Analyze the distribution of activated areas on the output feature map of the Feature Pyramid Network (FPN).

[0127] (3) Dynamic adjustment: Based on the analysis results, the size and proportion of the anchor box are dynamically adjusted so that the anchor box can better cover the high response area and match the actual shape of the target. Specifically:

[0128] 3-1) Dynamically adjust the size of the anchor box:

[0129] Adjustment based on feature map scale: On high-resolution feature maps, generate smaller anchor boxes to capture small targets; on low-resolution feature maps, generate larger anchor boxes to detect large targets. The formula for adjusting the size of the anchor box can be expressed as:

[0130] size anchor = k·size feature map

[0131] Among them, k is the preset scale factor, size feature map is the size of the feature map.

[0132] Adjustment based on feature distribution: By analyzing the activated areas (i.e., high response areas) in the feature map, the size of the anchor box can be dynamically adjusted to better cover these areas. For example, if the activation area of ​​a feature map is concentrated in a specific scale range, the size of the anchor box can be adjusted to a more suitable size based on this information.

[0133] 3-2) Dynamically adjust the anchor frame ratio:

[0134] The proportion adjustment of the anchor box is to adapt to objects of different shapes (such as objects with inconsistent aspect ratios). When generating the initial anchor box, several different proportions are usually preset (1:1, 1:2, 2:1). The adaptive strategy will adjust the proportion of the anchor box according to the actual shape distribution of the object so that it better matches the aspect ratio of the object.

[0135] Adjustment based on target shape: During the training process, the aspect ratio distribution of the target is statistically analyzed, and the proportion of the anchor box is dynamically adjusted based on this statistical information. The specific adjustment formula can be expressed as:

[0136]

[0137] Among them, W target and h target are the width and height of the target respectively.

[0138] (4) Update anchor frame: Use the adjusted anchor frame for target detection to improve detection accuracy.

[0139] This embodiment optimizes the matching degree between the anchor frame and the target by dynamically adjusting the size and proportion of the anchor frame through an adaptive anchor frame generation strategy. The size and proportion of the anchor frame are dynamically adjusted according to the content of the input image and the information of the multi-scale feature map, so that these anchor frames can better cover the potential target area. This adaptive anchor frame strategy helps to more accurately detect targets of different sizes, especially in complex environments, to ensure that targets of different sizes can be captured. These anchor frames will be matched with the actual target during the target detection process to locate the position of the target.

[0140] Specifically, the target detection head network is used to perform target classification and bounding box regression for each anchor box. These two processes are usually completed through two branch networks:

[0141] (1) Target Classification Network

[0142] Convolutional layer and fully connected layer: In each anchor box, features are extracted through the convolutional layer, and then the content in the anchor box is classified through the fully connected layer. The goal of classification is to identify whether there is a target object in the anchor box and the category of the object.

[0143] Softmax classifier: Softmax classifier is usually used to classify the features of each anchor box and output the probability value of belonging to each category. The category with the highest probability is the target category detected by the anchor box.

[0144] Utilization of multimodal features: By fusing multimodal features (optical, infrared, lidar, etc.), the classifier can distinguish objects of different categories more accurately, especially when the visual effects are greatly affected by weather.

[0145] (2) Bounding Box Regression Network

[0146] Regression model: The object detection network regresses the position of the anchor box to more accurately fit the bounding box of the actual target. The input of the regression model is the coordinates and size of the anchor box, and the output is the corrected bounding box parameters.

[0147] Multi-scale bounding box regression: Using feature maps from different scales, the regression model is able to accurately localize objects of different sizes. This multi-scale regression strategy helps localize objects in complex backgrounds.

[0148] Loss function optimization: During the training process, specific loss functions (such as Smooth L1 loss or IoU loss) are usually used to optimize the bounding box regression results. These loss functions help reduce the deviation between the anchor box and the actual target and improve the accuracy of positioning.

[0149] As a further preferred technical solution, the method further includes using a joint loss function as the loss function used in the training process of the target detection model, and the formula is expressed as:

[0150] L total =α·L cls +β·L loc +γ·L reg

[0151] Where, L cls is the classification loss function, usually using cross entropy loss; L loc For positioning loss function, smooth L1 loss or IoU loss is usually used; L reg is the regularization loss function such as L2 regularization; α, β, and γ are the corresponding weight coefficients respectively.

[0152] This embodiment plays a vital role in the multi-scale feature fusion UAV target detection method for bad weather by adopting a joint loss function. By combining classification loss, positioning loss and regularization loss, it ensures that the model can maintain high accuracy and high positioning accuracy in complex environments while preventing overfitting. The design innovation of the joint loss function lies in its flexibility and comprehensiveness, which enables the model to perform tasks robustly under various complex conditions.

[0153] By introducing weight coefficients α, β, and γ, the joint loss function can balance the contribution of each loss term to the total loss. This design allows the weight of the loss to be dynamically adjusted according to the needs of the actual application to better meet the requirements of the specific scenario. For example, in bad weather conditions, it may be necessary to pay more attention to positioning accuracy. In this case, the weight of the positioning loss β can be increased.

[0154] Furthermore, during the model training and validation process, a reasonable batch size is set to balance memory usage and model performance. L2 regularization technology is used to prevent model overfitting, and independent validation sets are used to evaluate model performance and monitor overfitting problems that may occur during training. An early stopping mechanism is introduced to terminate training early when the validation set performance no longer improves.

[0155] When testing and evaluating the model, this embodiment uses mAP (Mean Average Precision) to evaluate the average accuracy of the model under multiple thresholds, and uses FPS (Frames Per Second) to test the inference speed of the model to measure its performance in real-time applications. The confusion matrix is ​​used to analyze the types of classification errors, identify the weaknesses of the model, and check the samples that failed to be detected, and analyze the deficiencies of the model in specific scenarios, such as performance under occlusion or low light conditions.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0157] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0158] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-scale feature fusion UAV target detection method for bad weather, characterized by: The method comprises: Collecting multimodal data, wherein the multimodal data includes visible light image data and thermal infrared image data; Inputting the multimodal data into a target detection model, wherein the target detection model includes a first modality feature extraction network, a second modality feature extraction network, a multi-scale pyramid feature fusion network, a cross-modality fusion network, and a target detection head network; Using the first modality feature extraction fusion network to perform multi-scale feature extraction on visible light image data to obtain visible light multi-scale features; The second modality feature extraction and fusion network is used to extract multi-scale features from thermal infrared image data to obtain thermal infrared multi-scale features; The visible light multi-scale features and the thermal infrared multi-scale features are processed respectively by using a multi-scale pyramid feature fusion network to obtain a visible light feature map and a thermal infrared feature map; The visible light feature map and the thermal infrared feature map are fused using a cross-modal fusion network to obtain a fused feature map; The target detection head network is used to perform target detection on the fused feature map to obtain the target detection result.

2. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 1, characterized in that: The first modality feature extraction network and the second modality feature extraction network both adopt a hierarchical feature extraction network, and the hierarchical feature extraction network includes a low-level convolutional subnetwork and a high-level convolutional subnetwork connected in sequence; The low-level convolutional subnetwork includes sequentially connected convolutional layers Conv1 and Conv2, the high-level convolutional subnetwork includes sequentially connected convolutional layers Conv3, Conv4 and Conv5, and the convolutional layer Conv2 is connected to the convolutional layer Conv3; The low-level convolutional subnetwork and the high-level convolutional subnetwork output low-level feature maps and high-level feature maps respectively.

3. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 2, characterized in that: The hierarchical feature extraction network adopts a deep convolutional neural network.

4. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 1, characterized in that: The multi-scale pyramid feature fusion network includes a first feature pyramid sub-network and a multi-scale feature fusion sub-network connected in sequence; The first feature pyramid subnetwork is used to perform upsampling on the high-level feature map and combine it with the corresponding low-level feature map to obtain a multi-scale feature map corresponding to the visible light image data or the thermal infrared image data; The multi-scale feature fusion sub-network is used to process and fuse the feature information at each scale in the multi-scale feature map to obtain the visible light feature map or the thermal infrared feature map.

5. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 4 is characterized in that: The multi-scale feature map includes a high-resolution feature map, a medium-resolution feature map and a low-resolution feature map, and the multi-scale feature fusion sub-network includes a first fusion layer, a second fusion layer and a third fusion layer; The first fusion layer is used to perform convolution and ReLU activation on the high-resolution feature map to enhance local detail information, thereby obtaining a detail feature map; The second fusion layer is used to perform convolution and ReLU activation on the medium resolution feature map to enhance the target structure information and obtain the target feature map; The third fusion layer is used to perform convolution and ReLU activation on the low-resolution feature map to extract global semantic information and obtain a semantic feature map; A weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

6. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 1, characterized in that: The step of performing a weighted sum operation on the detail feature map, the target feature map, and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map includes: Assigning corresponding weights to the detail feature map, the target feature map, and the semantic feature map through an attention mechanism; According to the corresponding weights, a weighted sum operation is performed on the detail feature map, the target feature map and the semantic feature map to obtain a visible light feature map or a thermal infrared feature map.

7. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 6, characterized in that: The assigning corresponding weights to the detail feature map, the target feature map, and the semantic feature map through an attention mechanism includes: The channel attention mechanism and spatial attention mechanism are used to assign weights to different feature maps in the channel dimension and spatial dimension respectively.

8. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 1, characterized in that: The cross-modal fusion network includes a second feature pyramid sub-network, an attention mechanism, a convolutional layer and a fully connected layer connected in sequence. The visible light feature map and the thermal infrared feature map are respectively used as inputs of the second feature pyramid sub-network, and the fully connected layer outputs the fused feature map.

9. The multi-scale feature fusion UAV target detection method for bad weather as claimed in claim 1, characterized in that: The target detection head network adopts the YOLOv8 target detection network and detects the position of the target object through an adaptive anchor frame generation strategy.

10. The multi-scale feature fusion UAV target detection method for bad weather according to any one of claims 1 to 9, characterized in that: The method further comprises: The target detection model is trained, and the loss function used in the training process is a joint loss function, which is expressed as follows: L total =α·L cls +β·L loc +γ·L reg Where, L cls is the classification loss function, L loc is the positioning loss function, L reg is the regularized loss function, and α, β, and γ are the corresponding weight coefficients respectively.

Images