Unmanned aerial vehicle small target detection model training method, application method and related products

Abstract

The application discloses a UAV small target detection model training method, an application method and related products, relates to the small target detection technical field of artificial intelligence, and comprises the following steps: extracting the features of a satellite map by using Resnet18 to obtain a satellite feature map; gradually downsampling UAV image data to extract target features and obtain a plurality of visible light feature maps; fusing the smallest visible light feature map with the satellite feature map to obtain a background fusion feature map; extracting the up-sampling features of the background fusion feature map by using an up-sampling method, and then sequentially fusing the up-sampled features with corresponding visible light feature maps of the same level to obtain a plurality of fusion feature maps; taking the plurality of fusion feature maps as input, taking the target position information and target category information contained in the UAV image data as labels, training a target detection model, and obtaining a UAV small target detection model. The application can effectively reduce the calculation amount on the basis of ensuring accuracy.

Description

Technical Field

[0001] This application relates to the field of small target detection technology in artificial intelligence, and in particular to a method, device, medium and product for small target detection by unmanned aerial vehicles. Background Technology

[0002] In recent years, with the rapid advancements in artificial intelligence (AI) and drone technology, drone aerial photography has demonstrated its enormous application potential across multiple fields. Due to its high flexibility, wide coverage, and relatively low cost, drone aerial photography has become an indispensable tool in areas such as urban traffic management, public safety monitoring, disaster emergency response, environmental protection monitoring, and security for large-scale events. Particularly in urban traffic monitoring, drones can be rapidly deployed to key areas, transmitting high-definition images in real time, helping traffic management departments effectively monitor traffic flow, identify violations, and respond quickly to traffic accidents, thereby significantly improving urban traffic efficiency and safety.

[0003] While drone aerial photography technology has brought unprecedented vision and data acquisition capabilities, it still faces a series of technical challenges in practical applications, especially in the detection of small targets. Small targets typically refer to objects that occupy a small portion of an image and have low resolution, such as pedestrians, vehicle details, small obstacles, or specific event markers. Due to their small size, varied shapes, and lack of texture details, coupled with factors such as changing perspectives, varying lighting conditions, and complex backgrounds that may occur during drone photography, accurate detection of small targets is an extremely challenging task.

[0004] Most existing technologies extract feature information from multiple modalities and then fuse them. However, during the fusion process, each modality is extracted through a backbone network, resulting in a large computational load. Summary of the Invention

[0005] The purpose of this application is to provide a training method, application method and related products for a small target detection model for unmanned aerial vehicles (UAVs), which can effectively reduce the amount of computation while ensuring accuracy.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] Firstly, this application provides a method for training a small target detection model for unmanned aerial vehicles (UAVs), including:

[0008] Acquire aerial photography data; the aerial photography data includes: drone aerial image data and corresponding camera angles and drone positions; the drone aerial image data includes: drone image data and target location information and target category information contained in the drone image data;

[0009] Based on the camera angle and the drone position, a satellite map corresponding to the drone image data is obtained, and features of the satellite map are extracted using ResNet18 to obtain a satellite feature map.

[0010] The UAV image data is downsampled stepwise to extract target features and obtain several visible light feature maps;

[0011] The smallest visible light feature map and the satellite feature map are fused using a Transformer to obtain a background fused feature map;

[0012] After extracting the upsampled features of the background fusion feature map using the upsampling method, it is sequentially fused and upsampled with the visible light feature map of the same level to obtain several fusion feature maps;

[0013] Using several fused feature maps as input, and the target location information and target category information contained in the UAV image data as labels, a target detection model is trained to obtain a UAV small target detection model.

[0014] Optionally, the UAV image data is progressively downsampled to extract target features, resulting in several visible light feature maps, specifically including:

[0015] By using several multi-scale feature extraction modules, the UAV image data is progressively downsampled to extract target features and obtain several visible light feature maps;

[0016] The feature extraction process of each of the multi-scale feature extraction modules specifically includes:

[0017] The UAV image data is downsampled using a 3×3 convolution to obtain the first feature map;

[0018] Using the receptive field adaptive module, a receptive field adaptive feature map is obtained based on the first feature map;

[0019] Using the spatial association module, a spatial association module feature map is obtained based on the first feature map;

[0020] The receptive field adaptive feature map and the spatial association module feature map are multiplied according to the corresponding positions of the elements to obtain the second feature map;

[0021] After adding the second feature map to the receptive field adaptive feature map, a visible light feature map is obtained by 1×1 convolution.

[0022] Optionally, the step of using the receptive field adaptive module to obtain a receptive field adaptive feature map based on the first feature map specifically includes:

[0023] The first feature map is copied five times to obtain the first copied feature map, the second copied feature map, the third copied feature map, the fourth copied feature map, and the fifth copied feature map;

[0024] The first, second, third, and fourth replicated feature maps are processed by parallel 3×3, 5×5, 9×9, and 13×13 large-kernel dilated convolutions to obtain receptive field feature maps.

[0025] After concatenating the receptive field feature map and the first feature map along the channel dimension, a multi-receptive field feature map is obtained by performing a 3×3 convolution.

[0026] Using the channel attention method, different weights are assigned to each receptive field channel of the multiple receptive field feature map to obtain a weighted receptive field feature map;

[0027] The receptive field feature map is reduced in dimensionality using a 1×1 convolution to obtain an adaptive receptive field feature map.

[0028] Optionally, the smallest visible light feature map and the satellite feature map are fused using a Transformer to obtain a background fused feature map, specifically including:

[0029] The smallest visible light feature map is stacked with the satellite feature map according to the channel to obtain a stacked feature map;

[0030] The stacked feature maps are subjected to 3×3 convolution and 5×5 average pooling to obtain redundant feature maps.

[0031] The redundant feature maps are fused using a Transformer to obtain a fused feature map.

[0032] The fused feature map is convolved with 1×1 to obtain the background fused feature map.

[0033] Optionally, the step of training a target detection model using several fused feature maps as input and the target location information and target category information contained in the UAV image data as labels to obtain a UAV small target detection model specifically includes:

[0034] For each of the fused feature maps:

[0035] The background fusion feature map is convolved with 3×3 to obtain the background convolution feature map;

[0036] The fused feature map is then convolved with a 3×3 convolution to obtain a fused convolutional feature map;

[0037] The background convolutional feature map and the fused convolutional feature map are multiplied by the corresponding intra-channel matrices to obtain the multiplied feature map;

[0038] The activation feature map is obtained by multiplying the multiplied feature map and the fused feature map at corresponding positions using the ReLU activation function.

[0039] The activated feature map is concatenated with the fused feature map to obtain a concatenated feature map;

[0040] The stitched feature map is convolved with a 1×1 convolution to obtain a background-foreground enhancement feature map;

[0041] Using the background foreground enhancement feature map as input, and the target location information and target category information contained in the UAV image data as labels, a target detection model is trained to obtain a UAV small target detection model.

[0042] Optionally, the loss function used when training the object detection model includes: classification loss, confidence loss, and localization loss.

[0043] Secondly, this application provides a method for applying a small target detection model for unmanned aerial vehicles (UAVs), including:

[0044] Acquire aerial data to be detected; the aerial data to be detected includes: first UAV image data and the corresponding first camera angle and first UAV position;

[0045] Based on the first camera angle and the first UAV position, a satellite map corresponding to the first UAV image data is obtained, and features of the satellite map are extracted using ResNet18 to obtain a first satellite feature map.

[0046] The first UAV image data is downsampled stepwise to extract target features and obtain several first visible light feature maps;

[0047] The smallest first visible light feature map and the first satellite feature map are fused using a Transformer to obtain the first background fused feature map;

[0048] After extracting the upsampled features of the first background fusion feature map using the upsampling method, it is sequentially fused and upsampled with the first visible light feature map of the same level to obtain several first fusion feature maps;

[0049] Several of the first fused feature maps are input into the UAV small target detection model to obtain the target location information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained by any of the above-described UAV small target detection model training methods.

[0050] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV small target detection model training method or the UAV small target detection model application method described above.

[0051] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the UAV small target detection model training method or the UAV small target detection model application method described above.

[0052] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the UAV small target detection model training method or the UAV small target detection model application method described above.

[0053] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0054] This application provides a training method, application method, and related products for a small target detection model for unmanned aerial vehicles (UAVs). The training method includes: acquiring aerial photography data; the aerial photography data includes: UAV aerial image data and corresponding camera angles and UAV positions; the UAV aerial image data includes: UAV image data and target position information and target category information contained in the UAV image data; acquiring a satellite map corresponding to the UAV image data based on the camera angles and the UAV position, and extracting features from the satellite map using ResNet18 to obtain a satellite feature map; progressively downsampling the UAV image data to extract target features, obtaining several visible light feature maps; fusing the smallest visible light feature map with the satellite feature map using a Transformer to obtain a background fusion feature map; extracting upsampled features from the background fusion feature map using an upsampling method, and then sequentially fusing and upsampling it with the corresponding visible light feature maps of the same level to obtain several fusion feature maps; using the several fusion feature maps as input, and the target position information and target category information contained in the UAV image data as labels, training a target detection model to obtain a small target detection model for UAVs. In this application, the visible light RGB mode is used as the primary mode, while other modes are extracted using a lightweight backbone network, ResNet18, which minimizes computational cost. Furthermore, during the fusion process, the smallest feature maps in the backbone network structure are fused, significantly reducing computational cost while effectively extracting background information. Therefore, this application effectively reduces computational cost while maintaining accuracy. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is an application environment diagram of a training method and application method for a small target detection model of an unmanned aerial vehicle (UAV) according to an embodiment of this application.

[0057] Figure 2 This is a flowchart illustrating a method for training a small target detection model for unmanned aerial vehicles (UAVs) according to an embodiment of this application.

[0058] Figure 3 This is a schematic diagram of the overall network structure provided in an embodiment of this application.

[0059] Figure 4 This is a schematic diagram of a multi-scale feature extraction module provided in an embodiment of this application.

[0060] Figure 5 Please see the schematic diagram of the receptive field adaptive module provided in one embodiment of this application.

[0061] Figure 6 This is a schematic diagram of a fusion upsampling module provided in an embodiment of this application.

[0062] Figure 7 This is a schematic diagram of a background and foreground enhancement module provided in an embodiment of this application.

[0063] Figure 8 This is a flowchart illustrating an application method for a small target detection model of an unmanned aerial vehicle (UAV) according to an embodiment of this application.

[0064] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] Related technologies utilize multimodal fusion methods to enrich image information of small targets. By integrating data from different sensors or imaging modalities (such as satellite imagery and infrared), the insufficiency of information from a single modality can be compensated for, thereby significantly improving the detection accuracy of small targets. However, to achieve this goal, it is necessary to develop effective multimodal fusion methods to ensure that data from different modalities can be accurately and efficiently integrated to maximize detection performance. This requires algorithms not only to possess strong feature extraction capabilities but also to intelligently handle the complementarity and redundancy between modalities, avoiding noise interference during the information fusion process.

[0067] This application aligns UAV images with satellite map images. It utilizes convolutions of varying sizes to extract UAV image features with different receptive fields, preserving features with effective receptive fields. Then, ResNet18 is used to extract satellite map images. The extracted background features from the UAV images and the map image features are fused. Upsampling is then performed using UAV image features of different sizes. Finally, the fused background information and the upsampled information are interacted and fused to determine the foreground and background. The fused result is then input into a detection head to obtain the detection result. This application can fully utilize the background information from both satellites and UAVs to increase the detection accuracy of foreground targets.

[0068] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] This application provides a method for training and applying a small target detection model for unmanned aerial vehicles (UAVs), which can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0070] In one exemplary embodiment, such as Figure 2 As shown, a method for training a small target detection model for unmanned aerial vehicles (UAVs) is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1Taking server 104 as an example, the explanation includes the following steps S1 to S6. Its overall network structure is as follows: Figure 3 As shown, where:

[0071] S1. Acquire aerial photography data; the aerial photography data includes: drone aerial image data and corresponding camera angles and drone positions; the drone aerial image data includes: drone image data and target location information and target category information contained in the drone image data. This embodiment acquires a large amount of drone aerial image data.

[0072] S2. Based on the camera angle and the drone position, obtain the satellite map corresponding to the drone image data, and use ResNet18 to extract the features of the satellite map to obtain a satellite feature map.

[0073] In this embodiment, the corresponding UAV position is obtained through satellite positioning, and a satellite map is obtained by combining the visible light camera angle and positioning. The satellite map is then aligned with the visible light image using an algorithm (multimodal alignment). The aligned satellite map image is then processed by ResNet18 to extract features, resulting in a satellite feature map R.

[0074] The multimodal alignment method can be:

[0075] If it is infrared image data, the image data is aligned by training a convolutional neural network with two inputs (RGB and other multimodal data) and a single output (corresponding to the multimodal data after modal alignment).

[0076] If it is satellite imagery, then alignment is achieved by determining a function:

[0077] The area cropped from the satellite map can be determined by using the drone's altitude and position information, as well as the field of view information from the RGB camera. This allows for alignment between the satellite map and the RGB camera.

[0078] First, a large amount of alignment data is acquired, including the drone's altitude and position information, and area information from cropped satellite maps. Then, a neural network method is used to determine the function.

[0079] ;

[0080] ;

[0081] Here, H and W represent the area retained in the satellite map after cropping it according to the height H and width W, with the drone's location as the center. Indicates the location coordinates of the drone. Represents the horizontal coordinate. The vertical coordinate is represented by h, the drone's altitude is represented by h, and the camera's field of view is represented by k, which are generally fixed. A neural network is used to determine the cropping size to align the map image data.

[0082] S3. The UAV image data is downsampled step by step to extract target features and obtain several visible light feature maps.

[0083] In this embodiment, a large amount of UAV image data is continuously passed through multiple multi-scale feature extraction modules to progressively downsample and extract features of targets of different sizes, resulting in visible light feature maps C5, C4, C3, C2, and C1 of different sizes.

[0084] For the multi-scale feature extraction module, please refer to [link / reference]. Figure 4 In this embodiment, the UAV image data is downsampled using a 3×3 convolution. The resulting feature map is then duplicated and passed through the receptive field adaptation module (GSY) and the spatial correlation module (FFN) respectively to obtain a receptive field adaptation feature map and a spatial correlation module feature map. The two feature maps are then multiplied element-wise to obtain feature maps adapted to different receptive field sizes. These are then added to the receptive field adaptation feature map and followed by a 1×1 convolution. The FFN consists of a linear layer, a ReLU activation function, and another linear layer.

[0085] Expressed as a formula:

[0086] ;

[0087] ;

[0088] ;

[0089] ;

[0090] ;

[0091] Where x is the feature map input to the receptive field adaptation module, x1, x2, and x3 represent intermediate processed feature maps, and y represents the output feature map after multi-scale feature extraction. Conv3×3 represents a regular convolution with a 3×3 kernel. GSY represents the receptive field adaptation module. FFN represents the spatial association module. This indicates element-wise multiplication. Conv1×1 represents a one-dimensional convolution with a 1×1 kernel. This represents the linear function transformation module. This represents the rectifier function transformation module.

[0092] Specifically, the process of progressively downsampling the UAV image data to extract target features and obtain several visible light feature maps includes: using several multi-scale feature extraction modules to progressively downsampling the UAV image data to extract target features and obtain several visible light feature maps.

[0093] The feature extraction process of each of the multi-scale feature extraction modules specifically includes:

[0094] The UAV image data is downsampled by 3×3 convolution to obtain a first feature map; a receptive field adaptive feature map is obtained based on the first feature map using a receptive field adaptive module; a spatial association module feature map is obtained based on the first feature map using a spatial association module; the receptive field adaptive feature map and the spatial association module feature map are multiplied according to the corresponding positions of the elements to obtain a second feature map; the second feature map is added to the receptive field adaptive feature map, and then a visible light feature map is obtained by 1×1 convolution.

[0095] For the receptive field adaptation module, please refer to [link / reference]. Figure 5 In this embodiment, the first feature map is copied five times. Four of these copies are subjected to parallel large-kernel dilated convolutions (3×3, 5×5, 9×9, and 13×13) to obtain feature maps with different receptive field sizes. The resulting maps are then concatenated with the original first feature map along the channel dimension, followed by a 3×3 convolution to obtain a multi-receptive field feature map. This multi-receptive field feature map is then processed with channel attention (SE) to assign different weights to different channels (different receptive fields). Finally, a 1×1 convolution is used for dimensionality reduction, compressing the input channels to obtain an adaptive receptive field feature map.

[0096] The receptive field adaptive module can be expressed by the following formula:

[0097] ;

[0098] ;

[0099] Adaptive receptive field feature map:

[0100] ;

[0101] Where x is the feature map input to the receptive field adaptive module, y represents the output feature map after multi-scale feature extraction, DWC is dilated convolution, N×N represents the kernel size, SE represents channel attention, and Conv1×1 represents a 1×1 convolution. y1 represents the output result after the operation, y2 represents the output result of the channel attention module, and y3 represents the output result of the receptive field adaptation module.

[0102] In this embodiment, the receptive field adaptive module obtains a receptive field adaptive feature map based on the first feature map, specifically including:

[0103] The first feature map is copied five times to obtain a first copied feature map, a second copied feature map, a third copied feature map, a fourth copied feature map, and a fifth copied feature map. The first copied feature map, the second copied feature map, the third copied feature map, and the fourth copied feature map are then subjected to parallel 3×3, 5×5, 9×9, and 13×13 large kernel dilated convolutions, respectively, to obtain receptive field feature maps. The receptive field feature maps and the first feature map are concatenated along the channel dimension and then subjected to a 3×3 convolution to obtain a multiple receptive field feature map. Using a channel attention method, different weights are assigned to each receptive field channel of the multiple receptive field feature map to obtain a weighted receptive field feature map. The weighted receptive field feature map is then dimensionality-reduced using a 1×1 convolution to obtain an adaptive receptive field feature map.

[0104] S4. The smallest visible light feature map and the satellite feature map are fused using a Transformer to obtain a background fused feature map.

[0105] In this embodiment, the smallest visible light feature map C1 is passed together with the satellite feature map R through the background fusion module to obtain the background fusion feature map B1.

[0106] The specific steps of the background blending module include:

[0107] Assume the smallest visible light feature map C1 has n channels, and the satellite feature map R has m channels. The input satellite feature map R and the smallest visible light feature map C1 are stacked together by channel. Redundant information is then removed by passing them through 3×3 convolutions and 5×5 average pooling layers, respectively. Feature fusion is then performed using the Transformer method, followed by 1×1 convolutions to reduce the dimensionality to the same number of channels c as the smallest visible light feature map C1.

[0108] Expressed as a formula:

[0109] ;

[0110] ;

[0111] ;

[0112] ;

[0113] Where z1 and z2 represent the input visible light feature map and satellite feature map, and z3, z4, and z5 represent intermediate feature maps. This indicates that the feature maps are concatenated according to channels. Maxpooling represents maximum feature pooling. ATT represents the attention mechanism. H represents the height of the input feature map, W represents the width of the input feature map, m represents the number of channels in the satellite feature map R, and n represents the number of channels in the visible light feature map C1. R represents the size of the feature map. z6 represents the size of the output feature map.

[0114] ATT is expressed by the formula:

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] in, The input is a satellite feature map. , , The matrix represents the learnable matrix, and Softmax is an activation function. Represents the query matrix. Represents the key matrix, Value matrix, This represents the transpose of the key matrix. It is a learnable parameter used to balance the magnitude of the dot product and prevent the gradient of the Softmax function from vanishing.

[0120] In this embodiment, the smallest visible light feature map and the satellite feature map are fused using a Transformer method to obtain a background fused feature map, specifically including:

[0121] The smallest visible light feature map is stacked with the satellite feature map according to the channel to obtain a stacked feature map; the stacked feature map is subjected to 3×3 convolution and 5×5 average pooling to obtain a redundant feature map; the redundant feature map is fused using a Transformer to obtain a fused feature map; the fused feature map is subjected to 1×1 convolution to obtain a background fused feature map.

[0122] S5. After extracting the upsampled features of the background fusion feature map using the upsampling method, the features are sequentially fused and upsampled with the visible light feature map of the same level to obtain several fusion feature maps.

[0123] In this embodiment, the obtained background fusion feature map B1 is used to extract features through an upsampling method, and then fused with the visible light feature map C2 through a fusion upsampling module to obtain fusion feature map B2. The obtained fusion feature map B2 is then used to extract features through an upsampling method, and then fused with the visible light feature map C3 through a fusion upsampling module to obtain fusion feature map B3. The obtained fusion feature map B3 is then used to extract features through an upsampling method, and then fused with the visible light feature map C4 through a fusion upsampling module to obtain fusion feature map B4. Thus, multiple fusion feature maps B2, B3, and B4 are obtained.

[0124] Please see Figure 6 The specific steps of the fusion upsampling module include:

[0125] The input fused feature map is subjected to a 3×3 convolution, followed by bilinear interpolation for upsampling, resulting in a feature map twice the size. The input visible light feature map is subjected to a 1×1 convolution and a 5×5 dilated convolution, resulting in a feature map of the same size. The two feature maps are then concatenated by channel, followed by a 3×3 convolution to reduce the number of channels to their original size, yielding the fused feature map.

[0126] Expressed as a formula:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] Where w represents the fused feature map, and w0 represents the visible light feature map. H represents the height of the input feature map, and W represents the width of the input feature map. Conv1×1 represents a 1×1 convolution. Conv3×3 represents a 3×3 convolution. DWC5×5 represents a 5×5 dilated convolution. ReLU represents the activation function. Upsample represents bilinear interpolation upsampling. This indicates concatenation along the channel dimension. R represents the size of the feature map, and C represents the number of channels in the input feature map. w1, w2, and w3 represent intermediate feature maps. w4 represents the output feature map.

[0132] S6. Using several fused feature maps as input, and the target location information and target category information contained in the UAV image data as labels, train the target detection model to obtain the UAV small target detection model.

[0133] In this embodiment, fused feature maps B2, B3, and B4 are combined with the background fused feature map B1 to obtain background foreground enhanced feature maps D1, D2, and D3 through a background foreground enhancement module. D1, D2, and D3 are then used to predict results using the prediction head of a YOLO series network. Loss functions are calculated for both the predicted and actual results, and backpropagation is performed. Gradient descent is used for optimization to train the network. The network weights are then saved, and the network is loaded onto devices such as drones to achieve drone target detection.

[0134] Please see Figure 7 In this embodiment, the specific steps of the background and foreground enhancement module include: passing the input background fusion feature map B1 through a 3×3 convolution to obtain feature map u0'; passing another feature map B through a 3×3 convolution, then multiplying the matrices according to the corresponding channels, then passing it through the ReLU activation function, multiplying it at the corresponding positions of the input feature map B, and then concatenating it with feature map B; finally, passing it through a 1×1 convolution, and finally outputting the result.

[0135] Expressed as a formula:

[0136] ;

[0137] ;

[0138] ;

[0139] Where u0 represents the background fusion feature map, u1 represents the intermediate output of the 3×3 convolution, u2 represents the intermediate output of the ReLU rectified function, and u3 represents the output. Conv1×1 represents a 1×1 convolution. Conv3×3 represents a 3×3 convolution. This indicates matrix multiplication of the two-dimensional feature maps of the corresponding channels. · indicates element-wise multiplication. Cancat indicates a concatenation operation. ReLU is the rectified activation function.

[0140] In this embodiment, a target detection model is trained using several fused feature maps as input, and the target location information and target category information contained in the UAV image data as labels, to obtain a UAV small target detection model. Specifically, for each fused feature map:

[0141] The background fusion feature map is convolved with a 3×3 matrix to obtain a background convolutional feature map; the fusion feature map is convolved with a 3×3 matrix to obtain a fusion convolutional feature map; the background convolutional feature map and the fusion convolutional feature map are multiplied by the matrix in the corresponding channel to obtain a multiplied feature map; the multiplied feature map and the fusion feature map are multiplied at corresponding positions using the ReLU activation function to obtain an activation feature map; the activation feature map and the fusion feature map are concatenated to obtain a concatenated feature map; the concatenated feature map is convolved with a 1×1 matrix to obtain a background-foreground enhancement feature map; using the background-foreground enhancement feature map as input, and the target location information and target category information contained in the UAV image data as labels, a target detection model is trained to obtain a UAV small target detection model.

[0142] The loss function used when training an object detection model includes: classification loss. Confidence loss and positioning loss .

[0143] Classification loss measures the difference between the predicted class and the true class.

[0144] ;

[0145] Where i represents each sample. The value is the actual label value. To predict the probability of the category.

[0146] Confidence loss measures the degree of overlap between the predicted bounding box and the ground truth bounding box. Confidence loss:

[0147] ;

[0148] Where j represents each sample. The value is the actual label value. To predict the probability of the category.

[0149] Localization loss measures the positional difference between the predicted bounding box and the ground truth bounding box.

[0150] ;

[0151] in, , represents the ratio of the intersection to the union of the predicted bounding box and the ground truth bounding box. A represents the predicted bounding box, and B represents the ground truth bounding box. The area of ​​the intersection is represented by C, where C represents the area of ​​the smallest bounding rectangle between the predicted and ground truth boxes. IOU represents the intersection-union loss function, and GIOU is the loss function that considers overlap and positional relationships.

[0152] Finally, the overall loss function of the network is the classification loss. Confidence loss and positioning loss Weighted average:

[0153] in, , This represents the weight of each loss function.

[0154] This application uses average pooling to filter foreground features and then uses Transformer to fuse the two background features.

[0155] 1. The background / foreground module can utilize the fused background from satellite maps and RGB cameras. On one hand, it can identify targets within the RGB camera's field of view by comparing the differences between the background and foreground. This is similar to the original image - background = foreground (target). On the other hand, targets in the foreground can be enhanced by the background. For example, recognizing a parking space can make the algorithm associate it with a car, recognizing a football field can make the algorithm associate it with a goal on the football field, and so on.

[0156] 2. Multiple different convolution kernels are used to obtain feature maps of receptive fields of different sizes. Channel attention is used to strengthen the channel weights of the most suitable receptive field. Finally, 1×1 convolution is used to restore the tensor to its original size, thus preserving important information.

[0157] 3. The network uses a method of multiplication followed by addition: multiplication is equivalent to a weighting strategy, which can enhance the features of important parts; addition can retain the original important features and further improve them based on the original features.

[0158] Based on the same inventive concept, such as Figure 8 As shown in the embodiments of this application, a method for applying a small target detection model for unmanned aerial vehicles (UAVs) is also provided, including:

[0159] A1. Obtain the aerial photography data to be detected; the aerial photography data to be detected includes: first UAV image data and the corresponding first camera angle and first UAV position.

[0160] A2. Based on the first camera angle and the first UAV position, obtain the satellite map corresponding to the first UAV image data, and use ResNet18 to extract the features of the satellite map to obtain the first satellite feature map.

[0161] A3. The first UAV image data is downsampled step by step to extract target features and obtain several first visible light feature maps.

[0162] A4. The smallest first visible light feature map and the first satellite feature map are fused using a Transformer to obtain a first background fusion feature map.

[0163] A5. After extracting the upsampled features of the first background fusion feature map using the upsampling method, it is then fused and upsampled sequentially with the first visible light feature map of the same level to obtain several first fusion feature maps.

[0164] A6. Input several of the first fused feature maps into the UAV small target detection model to obtain the target location information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained by the UAV small target detection model training method described above.

[0165] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 9 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a training method for a UAV small target detection model or an application method for a UAV small target detection model.

[0166] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0167] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0168] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0169] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0170] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0171] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0172] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc.

[0173] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0174] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for a small target detection model for unmanned aerial vehicles (UAVs), characterized in that, include: Acquire aerial data; The aerial data includes: drone aerial image data and corresponding camera angles and drone positions; the drone aerial image data includes: drone image data and target location information and target category information contained in the drone image data; Based on the camera angle and the drone position, a satellite map corresponding to the drone image data is obtained, and features of the satellite map are extracted using ResNet18 to obtain a satellite feature map. The UAV image data is downsampled stepwise to extract target features and obtain several visible light feature maps; The smallest visible light feature map and the satellite feature map are fused using a Transformer to obtain a background fused feature map; After extracting the upsampled features of the background fusion feature map using the upsampling method, it is sequentially fused and upsampled with the visible light feature map of the same level to obtain several fusion feature maps; Using several fused feature maps as input, and the target location information and target category information contained in the UAV image data as labels, a target detection model is trained to obtain a UAV small target detection model. The UAV image data is progressively downsampled to extract target features, resulting in several visible light feature maps, specifically including: By using several multi-scale feature extraction modules, the UAV image data is progressively downsampled to extract target features and obtain several visible light feature maps; The feature extraction process of each of the multi-scale feature extraction modules specifically includes: The UAV image data is downsampled using a 3×3 convolution to obtain the first feature map; Using the receptive field adaptive module, a receptive field adaptive feature map is obtained based on the first feature map; Using the spatial association module, a spatial association module feature map is obtained based on the first feature map; the spatial association module includes three parts: a linear layer, a ReLU activation function, and a linear layer. The receptive field adaptive feature map and the spatial association module feature map are multiplied according to the corresponding positions of the elements to obtain the second feature map; After adding the second feature map to the receptive field adaptive feature map, a visible light feature map is obtained by 1×1 convolution; The process of using the receptive field adaptive module to obtain a receptive field adaptive feature map based on the first feature map specifically includes: The first feature map is copied five times to obtain the first copied feature map, the second copied feature map, the third copied feature map, the fourth copied feature map, and the fifth copied feature map; The first, second, third, and fourth replicated feature maps are processed by parallel 3×3, 5×5, 9×9, and 13×13 large-kernel dilated convolutions to obtain receptive field feature maps. After concatenating the receptive field feature map and the first feature map along the channel dimension, a multi-receptive field feature map is obtained by performing a 3×3 convolution. Using the channel attention method, different weights are assigned to each receptive field channel of the multiple receptive field feature map to obtain a weighted receptive field feature map; The receptive field feature map is reduced in dimensionality using a 1×1 convolution to obtain an adaptive receptive field feature map.

2. The UAV small target detection model training method according to claim 1, characterized in that, The smallest visible light feature map and the satellite feature map are fused using a Transformer method to obtain a background fused feature map, specifically including: The smallest visible light feature map is stacked with the satellite feature map according to the channel to obtain a stacked feature map; The stacked feature maps are subjected to 3×3 convolution and 5×5 average pooling to obtain redundant feature maps. The redundant feature maps are fused using a Transformer to obtain a fused feature map. The fused feature map is convolved with 1×1 to obtain the background fused feature map.

3. The UAV small target detection model training method according to claim 1, characterized in that, The step involves using several fused feature maps as input, and the target location information and target category information contained in the UAV image data as labels, to train a target detection model, thereby obtaining a UAV small target detection model, specifically including: For each of the fused feature maps: The background fusion feature map is convolved with 3×3 to obtain the background convolution feature map; The fused feature map is then convolved with a 3×3 convolution to obtain a fused convolutional feature map; The background convolutional feature map and the fused convolutional feature map are multiplied by the corresponding intra-channel matrices to obtain the multiplied feature map; The activation feature map is obtained by multiplying the multiplied feature map and the fused feature map at corresponding positions using the ReLU activation function. The activated feature map is concatenated with the fused feature map to obtain a concatenated feature map; The stitched feature map is convolved with a 1×1 convolution to obtain a background-foreground enhancement feature map; Using the background foreground enhancement feature map as input, and the target location information and target category information contained in the UAV image data as labels, a target detection model is trained to obtain a UAV small target detection model.

4. The UAV small target detection model training method according to claim 1, characterized in that, The loss functions used when training an object detection model include: classification loss, confidence loss, and localization loss.

5. A method for applying a small target detection model for unmanned aerial vehicles (UAVs), characterized in that, include: Acquire aerial data to be tested; The aerial data to be detected includes: first drone image data and the corresponding first camera angle and first drone position; Based on the first camera angle and the first UAV position, a satellite map corresponding to the first UAV image data is obtained, and features of the satellite map are extracted using ResNet18 to obtain a first satellite feature map. The first UAV image data is downsampled stepwise to extract target features and obtain several first visible light feature maps; The smallest first visible light feature map and the first satellite feature map are fused using a Transformer to obtain the first background fused feature map; After extracting the upsampled features of the first background fusion feature map using the upsampling method, it is sequentially fused and upsampled with the first visible light feature map of the same level to obtain several first fusion feature maps; Several first fused feature maps are input into the UAV small target detection model to obtain target location information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained by the UAV small target detection model training method according to any one of claims 1-4.

6. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the UAV small target detection model training method according to any one of claims 1-4 or the UAV small target detection model application method according to claim 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV small target detection model training method according to any one of claims 1-4 or the UAV small target detection model application method according to claim 5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the UAV small target detection model training method according to any one of claims 1-4 or the UAV small target detection model application method according to claim 5.

Images