Unmanned aerial vehicle small target detection model training method, application method and related product

Abstract

The invention discloses an unmanned aerial vehicle small target detection model training method, an application method and a related product, and relates to the technical field of small target detection of artificial intelligence, and the training method comprises the steps: extracting the features of a satellite map through Resnet18, and obtaining a satellite feature map; performing gradual down-sampling on the unmanned aerial vehicle image data, extracting target features, and obtaining a plurality of visible light feature maps; performing feature fusion on the minimum visible light feature map and the satellite feature map to obtain a background fusion feature map; extracting up-sampling features of the background fusion feature map by using an up-sampling method, and then performing fusion up-sampling on the up-sampling features and the visible light feature maps corresponding to the same level in sequence to obtain a plurality of fusion feature maps; and training a target detection model by taking the plurality of fused feature maps as input and taking target position information and target category information contained in the unmanned aerial vehicle image data as labels to obtain an unmanned aerial vehicle small target detection model. According to the method, the calculation amount can be effectively reduced on the basis of ensuring the accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of small target detection by artificial intelligence, and in particular to a method, device, medium and product for detecting small targets by drones. Background Art

[0002] In recent years, with the rapid advancement of artificial intelligence (AI) and drone technology, drone aerial photography has demonstrated tremendous potential in a wide range of fields. Due to its flexibility, wide coverage, and relatively low cost, drone aerial photography has become an indispensable tool in urban traffic management, public safety monitoring, disaster response, environmental protection monitoring, and large-scale event security. In urban traffic monitoring, drones can be rapidly deployed to key areas, transmitting high-definition images in real time. This allows traffic management departments to effectively monitor traffic flow, identify violations, and quickly respond to traffic incidents, significantly improving urban transportation efficiency and safety.

[0003] While drone aerial photography technology offers unprecedented vision and data acquisition capabilities, practical applications, particularly in the detection of small objects, still face a series of technical challenges. Small objects typically represent low-resolution objects that occupy a relatively small portion of an image, such as pedestrians, vehicle details, small obstacles, or specific event signs. The tiny size, varied shapes, and lack of texture detail of these objects, combined with the potential for perspective changes, varying lighting conditions, and complex backgrounds during drone photography, make accurate detection of these small objects a particularly challenging task.

[0004] Most existing technologies extract feature information from multiple modalities and then fuse them. However, during the fusion process, each modality is extracted through the backbone network, which results in a large amount of computation. Summary of the Invention

[0005] The purpose of this application is to provide a UAV small target detection model training method, application method and related products, which can effectively reduce the amount of calculation while ensuring accuracy.

[0006] To achieve the above objectives, this application provides the following solutions: In a first aspect, the present application provides a method for training a small target detection model for a drone, comprising: Acquire aerial photography data; the aerial photography data includes: drone aerial image data and corresponding camera angle and drone position; the drone aerial image data includes: drone image data and target location information and target category information contained in the drone image data; Obtain a satellite map corresponding to the drone image data based on the camera angle and the drone position, and use Resnet18 to extract features of the satellite map to obtain a satellite feature map; Stepwise downsampling of the drone image data is performed to extract target features to obtain a plurality of visible light feature maps; The smallest visible light feature map and the satellite feature map are fused using a Transformer method to obtain a background fusion feature map; After extracting the upsampled features of the background fusion feature map by using an upsampling method, the upsampled features are sequentially fused and upsampled with the visible light feature map corresponding to the same level to obtain a plurality of fusion feature maps; Taking the plurality of fused feature maps as input and the target position information and target category information contained in the drone image data as labels, a target detection model is trained to obtain a drone small target detection model.

[0007] Optionally, the drone image data is gradually downsampled to extract target features to obtain several visible light feature maps, specifically including: Using several multi-scale feature extraction modules, the UAV image data is gradually downsampled to extract target features to obtain several visible light feature maps; The feature extraction process of each multi-scale feature extraction module specifically includes: Performing 3×3 convolution downsampling on the drone image data to obtain a first feature map; Using a receptive field adaptation module, a receptive field adaptation feature map is obtained according to the first feature map; Using a spatial association module, a spatial association module feature map is obtained according to the first feature map; Multiplying the receptive field adaptive feature map and the spatial association module feature map according to corresponding positions of the elements to obtain a second feature map; After adding the second feature map to the receptive field adaptive feature map, a visible light feature map is obtained through 1×1 convolution.

[0008] Optionally, the utilizing the receptive field adaptive module to obtain the receptive field adaptive feature map according to the first feature map specifically includes: Copying the first characteristic map five times to obtain a first copied characteristic map, a second copied characteristic map, a third copied characteristic map, a fourth copied characteristic map, and a fifth copied characteristic map; The first copy feature map, the second copy feature map, the third copy feature map, and the fourth copy feature map are respectively subjected to parallel 3×3, 5×5, 9×9, and 13×13 large kernel dilated convolution to obtain receptive field feature maps; After concatenating the receptive field feature map and the first feature map in the channel dimension, a 3×3 convolution is performed to obtain a multiple receptive field feature map; Using the channel attention method, different weights are assigned to each receptive field channel of the multiple receptive field feature maps to obtain a weighted receptive field feature map; The weighted receptive field feature map is reduced in dimension by using a 1×1 convolution to obtain a receptive field adaptive feature map.

[0009] Optionally, the smallest visible light feature map and the satellite feature map are subjected to feature fusion in a Transformer manner to obtain a background fusion feature map, specifically including: Stacking the minimum visible light feature map and the satellite feature map according to channels to obtain a stacked feature map; Perform 3×3 convolution and 5×5 average pooling on the stacked feature map to obtain a redundant feature map; Perform feature fusion on the redundant feature map by using a Transformer to obtain a fused feature map; The fused feature map is subjected to 1×1 convolution to obtain a background fused feature map.

[0010] Optionally, the method of using the plurality of fused feature maps as input and the target position information and target category information contained in the drone image data as labels to train a target detection model to obtain a drone small target detection model specifically includes: For each of the fused feature maps: Performing a 3×3 convolution on the background fusion feature map to obtain a background convolution feature map; Performing a 3×3 convolution on the fused feature map to obtain a fused convolution feature map; Multiplying the background convolution feature map and the fused convolution feature map according to the corresponding channel matrix to obtain a multiplied feature map; The multiplied feature map and the fused feature map are multiplied at corresponding positions by a ReLU activation function to obtain an activated feature map; Splicing the activation feature map and the fusion feature map to obtain a spliced feature map; Perform 1×1 convolution on the spliced feature map to obtain a background foreground enhancement feature map; The background foreground enhancement feature map is used as input, and the target position information and target category information contained in the drone image data are used as labels to train a target detection model to obtain a drone small target detection model.

[0011] Optionally, the loss function when training the target detection model includes: classification loss, confidence loss, and positioning loss.

[0012] In a second aspect, the present application provides a method for applying a drone small target detection model, including: Acquire the aerial photography data to be detected; the aerial photography data to be detected includes: first drone image data and a corresponding first camera angle and first drone position; Obtaining a satellite map corresponding to the first drone image data based on the first camera angle and the first drone position, and extracting features of the satellite map using Resnet18 to obtain a first satellite feature map; gradually downsampling the first UAV image data to extract target features to obtain a plurality of first visible light feature maps; Perform feature fusion on the smallest first visible light feature map and the first satellite feature map in a Transformer manner to obtain a first background fusion feature map; After extracting upsampled features of the first background fusion feature map using an upsampling method, the upsampled features are sequentially fused and upsampled with the first visible light feature map corresponding to the same level to obtain a plurality of first fusion feature maps; Inputting several of the first fusion feature maps into a UAV small target detection model to obtain target position information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained by any of the UAV small target detection model training methods described above.

[0013] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the computer program to implement the drone small target detection model training method or drone small target detection model application method described in any one of the above.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the drone small target detection model training method or drone small target detection model application method described above.

[0015] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the drone small target detection model training method or drone small target detection model application method described in any one of the above.

[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects: The present application provides a training method, application method and related products for a small target detection model for a drone. The training method includes: obtaining aerial photography data; the aerial photography data includes: drone aerial image data and corresponding camera angles and drone positions; the drone aerial image data includes: drone image data and target position information and target category information contained in the drone image data; obtaining a satellite map corresponding to the drone image data based on the camera angles and the drone positions, and extracting features of the satellite map using Resnet18 to obtain a satellite feature map; gradually downsampling the drone image data to extract target features to obtain a plurality of visible light feature maps; fusing the smallest visible light feature map with the satellite feature map in a Transformer manner to obtain a background fusion feature map; extracting upsampled features of the background fusion feature map using an upsampling method, and then fusing and upsampling them with the visible light feature maps corresponding to the same level in sequence to obtain a plurality of fusion feature maps; and training a target detection model using the plurality of fusion feature maps as input and the target position information and target category information contained in the drone image data as labels to obtain a drone small target detection model. This application uses visible light RGB as the primary modality, while other modalities are extracted through a lightweight backbone network, ResNet18, which reduces computational complexity. Furthermore, during the fusion process, the smallest feature maps in the backbone network structure are fused, significantly reducing computational complexity and effectively extracting background information. Therefore, this application can effectively reduce computational complexity while maintaining accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is an application environment diagram of a UAV small target detection model training method and application method in one embodiment of the present application.

[0019] Figure 2 A flowchart of a method for training a small target detection model for a drone is provided in one embodiment of the present application.

[0020] Figure 3 A schematic diagram of the overall network structure provided in one embodiment of the present application.

[0021] Figure 4 A schematic diagram of a multi-scale feature extraction module provided in one embodiment of the present application.

[0022] Figure 5 A schematic diagram of a receptive field adaptation module provided in one embodiment of the present application is shown.

[0023] Figure 6 A schematic diagram of a fusion upsampling module provided in one embodiment of the present application.

[0024] Figure 7 A schematic diagram of a background and foreground enhancement module provided in one embodiment of the present application.

[0025] Figure 8 A flowchart of a method for applying a small target detection model for a drone is provided in one embodiment of the present application.

[0026] Figure 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] Related technologies utilize multimodal fusion methods to enrich the image information of small targets. By integrating data from different sensors or imaging modalities (such as satellite imagery and infrared), the lack of information from a single modality can be compensated, significantly improving the detection accuracy of small targets. However, achieving this goal requires the development of effective multimodal fusion methods to ensure accurate and efficient integration of data from different modalities to maximize detection performance. This requires algorithms that not only possess strong feature extraction capabilities but also intelligently handle the complementarity and redundancy between modalities to avoid noise interference during the information fusion process.

[0029] This application aligns drone images with satellite map images. It uses convolutions of various sizes to extract drone image features with different receptive fields, retaining features with effective receptive fields. Resnet18 is then used to extract satellite map images. The extracted background features of the drone image are fused with the image features of the map. Upsampling is then performed by combining drone image features of varying sizes. Finally, the fused background information is combined with the upsampled information for background and foreground interaction and fusion. The fused information is input to the detection head to obtain the detection results. This application can fully utilize the background information of satellites and drones to increase the detection accuracy of foreground targets.

[0030] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0031] The present invention provides a method for training and applying a small target detection model for a drone, which can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0032] In an exemplary embodiment, Figure 2 As shown, a method for training a small target detection model for a drone is provided. The method is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in FIG. 1 is taken as an example to illustrate the process, which includes the following steps S1 to S6. The overall network structure is as follows: Figure 3 As shown, where: S1. Acquire aerial photography data; the aerial photography data includes: drone aerial image data and corresponding camera angles and drone positions; the drone aerial image data includes: drone image data and target position information and target category information contained in the drone image data. This embodiment acquires a large amount of drone aerial image data.

[0033] S2. Obtain a satellite map corresponding to the drone image data based on the camera angle and the drone position, and use Resnet18 to extract features of the satellite map to obtain a satellite feature map.

[0034] In this embodiment, the corresponding drone position is obtained through satellite positioning, and the satellite map is obtained by combining the visible light camera angle and positioning. The satellite map is then aligned with the visible light image through an algorithm (multimodal alignment). The aligned satellite map image is extracted through Resnet18 to obtain the satellite feature map R.

[0035] Among them, the multimodal alignment method can be: If it is infrared image data, the image data is aligned by training a convolutional neural network with dual input (RGB and other multimodal data) and single output (corresponding to the multimodal data after modal alignment).

[0036] If it is a satellite image, it is aligned by determining the function: The area to be cropped from the satellite map can be determined using the drone's altitude and position information and the RGB camera's field of view, allowing the satellite map to be aligned with the RGB camera.

[0037] First, we acquire a large amount of aligned data, including the altitude and position information of the drone and the area information of the cropped satellite map. Then, we use the neural network method to determine the function: ; ; Among them, H and W represent the area of the satellite map that is retained by cropping it according to the height H and width W with the drone's position as the center. Indicates the position coordinates of the drone, represents the horizontal coordinate, The vertical coordinate is represented by h, the altitude of the drone, and k is the camera angle of view, which are generally fixed. The cropping size is determined by a neural network to achieve alignment of the map image data.

[0038] S3. Gradually downsample the drone image data to extract target features to obtain several visible light feature maps.

[0039] In this embodiment, a large amount of drone image data is continuously passed through multiple multi-scale feature extraction modules, and features of targets of different sizes are gradually downsampled to obtain visible light feature maps C5, C4, C3, C2, and C1 of different sizes.

[0040] Among them, the multi-scale feature extraction module can be found in Figure 4 In this embodiment, the drone image data is downsampled using a 3×3 convolution. The resulting feature map is then replicated twice and passed through the receptive field adaptation module GSY and the spatial correlation module FFN, respectively, to produce a receptive field adaptation feature map and a spatial correlation module feature map. The two feature maps are then multiplied element-wise to produce feature maps adapted to different receptive field sizes. This is then added to the receptive field adaptation feature map and then subjected to a 1×1 convolution. The FFN consists of three components: a linear layer, a ReLU activation function, and a linear layer.

[0041] Expressed as a formula: ; ; ; ; ; Where x is the feature map input to the receptive field adaptation module, x1, x2, and x3 represent the intermediate processed feature maps, and y represents the output feature map after multi-scale feature extraction. Conv3×3 represents a normal convolution with a convolution kernel of 3×3. GSY represents the receptive field adaptation module. FFN represents the spatial association module. Indicates element-wise multiplication. Conv1×1 indicates a one-dimensional convolution with a convolution kernel of 1×1. represents the linear function transformation module, Represents the rectification function transformation module.

[0042] The method comprises the following steps: stepwise downsampling the drone image data, extracting target features, and obtaining a plurality of visible light feature maps, specifically comprising: utilizing a plurality of multi-scale feature extraction modules to stepwise downsampling the drone image data, extracting target features, and obtaining a plurality of visible light feature maps.

[0043] The feature extraction process of each multi-scale feature extraction module specifically includes: Perform 3×3 convolution downsampling on the drone image data to obtain a first feature map; use a receptive field adaptive module to obtain a receptive field adaptive feature map based on the first feature map; use a spatial association module to obtain a spatial association module feature map based on the first feature map; multiply the receptive field adaptive feature map and the spatial association module feature map according to the corresponding positions of the elements to obtain a second feature map; add the second feature map and the receptive field adaptive feature map, and then obtain a visible light feature map through 1×1 convolution.

[0044] Among them, the receptive field adaptation module can be found in Figure 5 In this embodiment, the first feature map is replicated five times, four of which are parallelized with large-kernel dilated convolutions of 3×3, 5×5, 9×9, and 13×13 to obtain feature maps with different receptive field sizes. The resulting image is then concatenated with the original first feature map in the channel dimension and subjected to 3×3 convolution to obtain a multi-receptive field feature map. The multi-receptive field feature map is then subjected to channel attention (SE), assigning different weights to different channels (different receptive fields). Finally, dimensionality reduction is performed through 1×1 convolution, compressing the image to the input channel to obtain an adaptive receptive field feature map.

[0045] The receptive field adaptation module is expressed as follows: ; ; Adaptive receptive field feature map: ; Where x is the feature map input to the receptive field adaptation module, y is the output feature map after multi-scale feature extraction, DWC is dilated convolution, N×N is the size of the convolution kernel, SE is channel attention, and Conv1×1 represents a 1×1 convolution. is the output result after the operation, y2 is the output result of the channel attention module, and y3 is the output result of the receptive field adaptation module.

[0046] In this embodiment, the receptive field adaptive module is used to obtain the receptive field adaptive feature map according to the first feature map, specifically including: The first feature map is copied five times to obtain a first copy feature map, a second copy feature map, a third copy feature map, a fourth copy feature map and a fifth copy feature map; the first copy feature map, the second copy feature map, the third copy feature map and the fourth copy feature map are respectively subjected to parallel 3×3, 5×5, 9×9 and 13×13 large kernel void convolutions to obtain a receptive field feature map; the receptive field feature map and the first feature map are spliced in the channel dimension and then subjected to 3×3 convolution to obtain a multiple receptive field feature map; the channel attention method is used to assign different weights to each receptive field channel of the multiple receptive field feature map to obtain a weighted receptive field feature map; the weighted receptive field feature map is reduced in dimension by 1×1 convolution to obtain a receptive field adaptive feature map.

[0047] S4. Fusing the smallest visible light feature map with the satellite feature map in a Transformer manner to obtain a background fusion feature map.

[0048] In this embodiment, the minimum visible light feature map C1 and the satellite feature map R are passed through a background fusion module to obtain a background fusion feature map B1.

[0049] Among them, the specific steps of the background fusion module include: Assume that the minimum visible light feature map C1 has n channels and the satellite feature map R has m channels. The satellite feature map R and the minimum visible light feature map C1 are stacked together by channel. They are then passed through a 3×3 convolution and a 5×5 average pooling layer to remove redundant information. The features are then fused using the Transformer method and finally reduced to the same number of channels, c, as the minimum visible light feature map C1, through a 1×1 convolution.

[0050] Expressed as a formula: ; ; ; ; Among them, z1 and z2 represent the input visible light feature map and satellite feature map, and z3, z4, and z5 represent the intermediate feature maps. Indicates channel-wise concatenation of feature maps. Maxpooling indicates maximum feature pooling. ATT indicates the attention mechanism. H represents the height of the input feature map, W represents the width of the input feature map, m represents the number of channels in the satellite feature map R, and n represents the number of channels in the visible light feature map C1. R represents the size of the feature map. z6 represents the size of the output feature map.

[0051] Among them, ATT is expressed by the formula: ; ; ; ; in, is the input satellite feature map. 、 、 Represents a learnable matrix, Softmax is an activation function. represents the query matrix, represents the bond matrix, Value matrix, Represents the transpose of the key matrix. It is a learnable parameter used to balance the magnitude of the dot product and prevent the gradient of the Softmax function from disappearing.

[0052] In this embodiment, the smallest visible light feature map and the satellite feature map are fused in a Transformer manner to obtain a background fusion feature map, specifically including: The smallest visible light feature map and the satellite feature map are stacked according to channels to obtain a stacked feature map; the stacked feature map is subjected to 3×3 convolution and 5×5 average pooling to obtain a redundant feature map; the redundant feature map is subjected to feature fusion in a Transformer manner to obtain a fused feature map; the fused feature map is subjected to 1×1 convolution to obtain a background fused feature map.

[0053] S5. After extracting the up-sampled features of the background fusion feature map using an up-sampling method, the up-sampled features are sequentially fused and up-sampled with the visible light feature map corresponding to the same level to obtain a plurality of fusion feature maps.

[0054] In this embodiment, the background fusion feature map B1 is extracted using an upsampling method, and then combined with the visible light feature map C2 through an upsampling module to obtain a fused feature map B2. The fused feature map B2 is extracted using an upsampling method, and then combined with the visible light feature map C3 through an upsampling module to obtain a fused feature map B3. The fused feature map B3 is extracted using an upsampling method, and then combined with the visible light feature map C4 through an upsampling module to obtain a fused feature map B4. Thus, multiple fused feature maps B2, B3, and B4 are obtained.

[0055] See also Figure 6 , the specific steps of the fusion upsampling module include: The input fused feature map is convolved with 3×3 convolution and then upsampled with bilinear interpolation to obtain a feature map of doubled size. The input visible light feature map is convolved with 1×1 convolution and 5×5 dilated convolution to obtain a feature map of unchanged size. The two feature maps are then concatenated and convolved with 3×3 convolution to reduce the channels to their original size, resulting in the fused feature map.

[0056] Expressed as a formula: ; ; ; ; Where w represents the fusion feature map, w0 represents the visible light feature map, H represents the height of the input feature map, and W represents the width of the input feature map. Conv1×1 represents a 1×1 convolution. Conv3×3 represents a 3×3 convolution. DWC5×5 represents a 5×5 hole convolution. ReLU represents the activation function. Upsample represents bilinear interpolation upsampling. Indicates concatenation by channel dimension. R represents the size of the feature map, and C represents the number of channels of the input feature. w1, w2, and w3 represent intermediate feature maps. w4 represents the output feature map.

[0057] S6. Using the plurality of fused feature maps as input and the target position information and target category information contained in the drone image data as labels, a target detection model is trained to obtain a drone small target detection model.

[0058] In this embodiment, the fused feature maps B2, B3, and B4 are combined with the background fused feature map B1 through the background foreground enhancement module to generate background foreground enhanced feature maps D1, D2, and D3. D1, D2, and D3 are then passed through the prediction head of the YOLO network to generate prediction results. The predicted results are compared with the actual results to calculate the loss function, and backpropagation is performed. The network is trained using gradient descent optimization. The network weights are then saved and loaded onto devices such as drones to achieve drone target detection.

[0059] See also Figure 7 The background-foreground enhancement module in this embodiment includes the following steps: 1. The input background fusion feature map B1 undergoes a 3×3 convolution to obtain a feature map u0`. 2. The other feature map B undergoes a 3×3 convolution, followed by matrix multiplication according to the corresponding channel. 3. Finally, the feature map B undergoes a ReLU activation function, multiplies the corresponding position of the input feature map B, and is concatenated with the feature map B. 4. Finally, the feature map undergoes a 1×1 convolution and is output.

[0060] Expressed as a formula: ; ; ; Where u0 represents the background fusion feature map, u1 represents the intermediate output of the 3×3 convolution, u2 represents the intermediate output of the ReLU rectifier function, and u3 represents the output. Conv1×1 represents a 1×1 convolution. Conv3×3 represents a 3×3 convolution. Indicates matrix multiplication of the two-dimensional feature maps of the corresponding channels. · indicates element-wise multiplication. Cancat indicates concatenation. ReLU is the rectifier activation function.

[0061] In this embodiment, the target detection model is trained using several fused feature maps as input and the target location information and target category information contained in the drone image data as labels to obtain a drone small target detection model. The background fusion feature map is subjected to 3×3 convolution to obtain a background convolution feature map; the fusion feature map is subjected to 3×3 convolution to obtain a fusion convolution feature map; the background convolution feature map and the fusion convolution feature map are multiplied according to the matrix in the corresponding channel to obtain a multiplied feature map; the multiplied feature map and the fusion feature map are multiplied at corresponding positions through the ReLU activation function to obtain an activation feature map; the activation feature map and the fusion feature map are spliced to obtain a spliced feature map; the spliced feature map is subjected to 1×1 convolution to obtain a background foreground enhancement feature map; the background foreground enhancement feature map is used as input, and the target position information and target category information contained in the drone image data are used as labels to train a target detection model to obtain a drone small target detection model.

[0062] The loss functions when training the target detection model include: classification loss , confidence loss and positioning loss .

[0063] Classification loss is used to measure the difference between the predicted category and the true category. Classification loss: ; Among them, i is each sample, is the true label value, is the probability of the predicted category.

[0064] Confidence loss is used to measure the degree of overlap between the predicted box and the true box. Confidence loss: ; Among them, j is each sample, is the true label value, is the probability of the predicted category.

[0065] The localization loss is used to measure the position difference between the predicted box and the true box. Localization loss: ; in, , represents the ratio of the intersection and union of the predicted box and the true box. A represents the predicted box and B represents the true box. represents the area of the intersection, C represents the area of the minimum bounding rectangle between the predicted box and the true box. IOU represents the intersection-over-union loss function, and GIOU is a loss function that takes into account overlap and positional relationships.

[0066] Finally, the overall loss function of the network is the classification loss , confidence loss and positioning loss The weighted average of: in, 、 Represents the weight of each loss function.

[0067] This application uses the average pooling method to filter the foreground features, and then uses the Transformer method to fuse the two background features.

[0068] 1. The background-foreground module utilizes the fused background of satellite maps and the RGB camera. It can identify objects in the RGB camera's field of view by comparing the differences between the background and foreground. This is similar to the formula "original image - background = foreground (object)." Furthermore, foreground objects can be enhanced by the background. For example, identifying a parking space can lead the algorithm to associate it with cars, while identifying a soccer field can lead the algorithm to associate it with the goal post, and so on.

[0069] 2. Use multiple different convolution kernels to obtain feature maps of receptive fields of different sizes, and use channel attention to strengthen the channel weights of the most appropriate receptive field. Finally, use 1×1 convolution to restore the tensor to its original size, thereby retaining important information.

[0070] 3. The network adopts the method of multiplication first and then addition: multiplication is equivalent to a weighted strategy, which can enhance the features of important parts; addition can retain the original important features and further improve them based on the original features.

[0071] Based on the same inventive concept, Figure 8 As shown, the embodiment of the present application also provides a method for applying a drone small target detection model, including: A1. Acquire aerial photography data to be detected; the aerial photography data to be detected includes: first drone image data and a corresponding first camera angle and first drone position.

[0072] A2. Obtain a satellite map corresponding to the first drone image data based on the first camera angle and the first drone position, and use Resnet18 to extract features of the satellite map to obtain a first satellite feature map.

[0073] A3. Gradually downsample the first UAV image data to extract target features to obtain a plurality of first visible light feature maps.

[0074] A4. Perform feature fusion on the smallest first visible light feature map and the first satellite feature map using a Transformer method to obtain a first background fusion feature map.

[0075] A5. After extracting the upsampled features of the first background fusion feature map using an upsampling method, the upsampled features are sequentially fused and upsampled with the first visible light feature map corresponding to the same level to obtain a plurality of first fusion feature maps.

[0076] A6. Inputting several of the first fused feature maps into a UAV small target detection model to obtain target position information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained using the UAV small target detection model training method described above.

[0077] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for training a small target detection model for a drone or a method for applying a small target detection model for a drone is implemented.

[0078] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0079] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0080] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0081] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0083] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0084] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0085] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A UAV small target detection model training method, characterized in that: include: Acquire aerial photography data; The aerial photography data includes: drone aerial image data and corresponding camera angle and drone position; the drone aerial image data includes: drone image data and target position information and target category information contained in the drone image data; Obtain a satellite map corresponding to the drone image data based on the camera angle and the drone position, and use Resnet18 to extract features of the satellite map to obtain a satellite feature map; Stepwise downsampling of the drone image data is performed to extract target features to obtain a plurality of visible light feature maps; The smallest visible light feature map and the satellite feature map are fused using a Transformer method to obtain a background fusion feature map; After extracting the upsampled features of the background fusion feature map by using an upsampling method, the upsampled features are sequentially fused and upsampled with the visible light feature map corresponding to the same level to obtain a plurality of fusion feature maps; Taking the plurality of fused feature maps as input and the target position information and target category information contained in the drone image data as labels, a target detection model is trained to obtain a drone small target detection model.

2. The UAV small target detection model training method according to claim 1 is characterized in that: The drone image data is gradually downsampled to extract target features to obtain several visible light feature maps, including: Using several multi-scale feature extraction modules, the UAV image data is gradually downsampled to extract target features to obtain several visible light feature maps; The feature extraction process of each multi-scale feature extraction module specifically includes: Performing 3×3 convolution downsampling on the drone image data to obtain a first feature map; Using a receptive field adaptation module, a receptive field adaptation feature map is obtained according to the first feature map; Using a spatial association module, a spatial association module feature map is obtained according to the first feature map; Multiplying the receptive field adaptive feature map and the spatial association module feature map according to corresponding positions of the elements to obtain a second feature map; After adding the second feature map to the receptive field adaptive feature map, a visible light feature map is obtained through 1×1 convolution.

3. The UAV small target detection model training method according to claim 2 is characterized in that: The utilizing the receptive field adaptive module to obtain the receptive field adaptive feature map according to the first feature map specifically includes: Copying the first characteristic map five times to obtain a first copied characteristic map, a second copied characteristic map, a third copied characteristic map, a fourth copied characteristic map, and a fifth copied characteristic map; The first copy feature map, the second copy feature map, the third copy feature map, and the fourth copy feature map are respectively subjected to parallel 3×3, 5×5, 9×9, and 13×13 large kernel dilated convolution to obtain receptive field feature maps; After concatenating the receptive field feature map and the first feature map in the channel dimension, a 3×3 convolution is performed to obtain a multiple receptive field feature map; Using the channel attention method, different weights are assigned to each receptive field channel of the multiple receptive field feature maps to obtain a weighted receptive field feature map; The weighted receptive field feature map is reduced in dimension by using a 1×1 convolution to obtain a receptive field adaptive feature map.

4. The UAV small target detection model training method according to claim 1 is characterized in that: The smallest visible light feature map and the satellite feature map are fused using a Transformer method to obtain a background fusion feature map, specifically including: Stacking the minimum visible light feature map and the satellite feature map according to channels to obtain a stacked feature map; Perform 3×3 convolution and 5×5 average pooling on the stacked feature map to obtain a redundant feature map; Perform feature fusion on the redundant feature map by using a Transformer to obtain a fused feature map; The fused feature map is subjected to 1×1 convolution to obtain a background fused feature map.

5. The UAV small target detection model training method according to claim 1 is characterized in that: The method uses the plurality of fused feature maps as input and the target position information and target category information contained in the drone image data as labels to train a target detection model to obtain a drone small target detection model, specifically including: For each of the fused feature maps: Performing a 3×3 convolution on the background fusion feature map to obtain a background convolution feature map; Performing a 3×3 convolution on the fused feature map to obtain a fused convolution feature map; Multiplying the background convolution feature map and the fused convolution feature map according to the corresponding channel matrix to obtain a multiplied feature map; The multiplied feature map and the fused feature map are multiplied at corresponding positions by a ReLU activation function to obtain an activated feature map; Splicing the activation feature map and the fusion feature map to obtain a spliced feature map; Perform 1×1 convolution on the spliced feature map to obtain a background foreground enhancement feature map; The background foreground enhancement feature map is used as input, and the target position information and target category information contained in the drone image data are used as labels to train a target detection model to obtain a drone small target detection model.

6. The UAV small target detection model training method according to claim 1 is characterized in that: The loss functions when training the target detection model include: classification loss, confidence loss, and positioning loss.

7. A method for applying a small target detection model for drones, characterized in that: include: Obtain the aerial photography data to be detected; The aerial photography data to be detected includes: first drone image data and corresponding first camera angle and first drone position; Obtaining a satellite map corresponding to the first drone image data based on the first camera angle and the first drone position, and extracting features of the satellite map using Resnet18 to obtain a first satellite feature map; gradually downsampling the first UAV image data to extract target features to obtain a plurality of first visible light feature maps; Perform feature fusion on the smallest first visible light feature map and the first satellite feature map in a Transformer manner to obtain a first background fusion feature map; After extracting upsampled features of the first background fusion feature map using an upsampling method, the upsampled features are sequentially fused and upsampled with the first visible light feature map corresponding to the same level to obtain a plurality of first fusion feature maps; Inputting several of the first fusion feature maps into a UAV small target detection model to obtain target position information and target category information contained in the aerial photography data to be detected; the UAV small target detection model is trained by the UAV small target detection model training method described in any one of claims 1-6.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the UAV small target detection model training method described in any one of claims 1 to 6 or the UAV small target detection model application method described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for training a small target detection model for a drone according to any one of claims 1 to 6 or the method for applying a small target detection model for a drone according to claim 7 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for training a small target detection model for a drone according to any one of claims 1 to 6 or the method for applying a small target detection model for a drone according to claim 7 is implemented.

Images