Unmanned aerial vehicle small target detection method, deep learning network model, system and readable storage medium

Abstract

The invention relates to the technical field of deep learning, in particular to an unmanned aerial vehicle small target detection method, a deep learning network model, a system and a readable storage medium. A more lightweight deep space feature fusion module is used for replacing a deep residual block to reduce channel interaction, so that redundancy and low efficiency of computing resources caused by a large number of stacked residual blocks in deep feature extraction are avoided; in a feature alignment task, a lightweight up-sampling operator DySample is introduced, so that computing resources are further saved under the condition that the detection precision is not sacrificed. The objective of the invention is to reduce the calculation overhead of a small target detection method and improve the semantic information extraction capability.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and particularly relates to a method for detecting small targets of an unmanned aerial vehicle, a deep learning network model, a system, and a readable storage medium. Background Art

[0002] Detecting small targets (also known as lightweight targets) from the perspective of an unmanned aerial vehicle is a technical difficulty that needs to be overcome urgently at present. The current common solution is to detect small targets in the images collected by the unmanned aerial vehicle based on the Feature Pyramid Network (FPN). FPN fuses feature maps of different scales by means of horizontal connection and vertical addition, so as to combine shallow features with deep features to enhance the semantic information of shallow features and improve the performance of small target detection.

[0003] However, the applicant found during the conception and implementation of this application that although the deep semantic information is effective for small target detection, the effect of directly detecting small targets on deep features is not good - because there are a large number of stacked residual blocks in the deep feature extraction of the traditional FPN, which is prone to cause redundancy and inefficiency of computing resources.

[0004] Based on this, this application proposes a method for detecting small targets of an unmanned aerial vehicle using a new FPN architecture. Summary of the Invention

[0005] The main purpose of this application is to provide a method for detecting small targets of an unmanned aerial vehicle, aiming to solve the problem of how to reduce the computational overhead of the small target detection method and improve the semantic information extraction ability.

[0006] To achieve the above purpose, a method for detecting small targets of an unmanned aerial vehicle provided by this application includes:

[0007] Obtain an image data set containing multiple small targets collected by the unmanned aerial vehicle;

[0008] Input the image data set into a trained deep learning network for feature extraction;

[0009] Among them, the deep learning network uses multiple deep space feature fusion modules to fuse semantic features of different scales, and aligns the fused feature map to the same resolution as the second convolutional layer through a lightweight upsampling operator DySample, and inputs the aligned feature map into the feature extraction layer to extract detailed features after fusing with shallow features using residual blocks;

[0010] Obtain the predicted category and predicted bounding box coordinates corresponding to each image output by the deep learning network;

[0011] Filter out the target prediction boxes without overlap according to the coordinates of the prediction boxes, and mark the target positions and target categories corresponding to each small target on the image based on the target prediction boxes.

[0012] Optionally, the depth spatial feature fusion module includes:

[0013]

[0014]

[0015] In the formula, represents the output after the semantic feature is processed by the depthwise dilated convolution, is the output after the previous layer of dilated convolution processing, is the learnable parameter of the depthwise dilated convolution, represents the depthwise dilated convolution operation, is the dilation rate, is the final output feature map, represents the pointwise convolution, represents the channel concatenation.

[0016] Optionally, the depthwise dilated convolution operation is jointly completed by a standard convolution layer, a depthwise separable convolution layer, and a dilated convolution layer, where:

[0017] The convolution kernel of the standard convolution is 3;

[0018] The convolution kernel of the depthwise separable convolution is 3, the number of grouped convolutions is the number of hidden channels, and the number of input channels is 4;

[0019] The convolution kernel of the dilated convolution is 3, and the dilation rates are 1, 2, and 3 respectively.

[0020] Optionally, the fusion strategy of the deep learning network is a dual-head asymptotic fusion strategy, and the dual-head asymptotic fusion strategy specifically includes:

[0021] In the primary stage of fusion, let represent the feature map at the layer and at the stage (starting from BackBone), and fuse the outputs from BackBone in parallel:

[0022]

[0023]

[0024] Among them, correspond to the outputs of BackBone respectively, represents the strided convolution downsampling operation, Indicates an upsampling operation, and Indicates being in the middle stage of the downsampling phase;

[0025] In the later stage of fusion, the semantics of the generated intermediate layers are gradually fused:

[0026]

[0027]

[0028] Among them, The fusion is at the intermediate layer , and further fuses the output from the deepest layer of the BackBone and the output from the previous stage of the P3 layer , where DSF is the Depth Spatial Feature Fusion Module;

[0029] In the dual-head asymptotic fusion strategy, the shallow layer obtains the semantics of the deep layer in an asymptotic manner. For , when there is:

[0030]

[0031] When there is:

[0032]

[0033] In the formula, Indicates concatenation along the channel dimension, Indicates adjusting the resolution of the feature map through upsampling.

[0034] Optionally, the step of aligning the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample specifically includes:

[0035] Using a sampling point generator to generate dynamic sampling offsets for each pixel point, and then adding the original grid position to the dynamic sampling offsets to obtain a sampling set;

[0036] Resampling the input fused feature map through a grid sampling function to obtain the upsampled feature;

[0037] Among them, the generation strategies of the sampling point generator include a static range factor strategy and a dynamic range factor strategy. The static range factor strategy includes: generating dynamic sampling offsets through a linear layer; the dynamic range factor strategy includes: first generating a range factor through content awareness, and generating the dynamic sampling offsets by passing the range factor through a linear layer and pixel transformation.

[0038] Optionally, the hyperparameters in the training process of the deep learning network are set as follows:

[0039] The number of iterations is set to 300, the learning rate is set to 0.01, and the number of images input at one time (batch) is set to 8.

[0040] Optionally, the screening of the target prediction boxes without overlap according to the coordinates of the prediction boxes includes:

[0041] Using the non-maximum suppression method to filter out the overlapping prediction boxes, and taking the remaining prediction boxes as the target prediction boxes.

[0042] In addition, to achieve the above object, the present application also provides a deep learning network model, and the deep learning network model includes:

[0043] YOLOv8 architecture;

[0044] Multiple deep space feature fusion modules for fusing semantic features of different scales in the image dataset containing multiple small targets collected by the drone;

[0045] A dual-head asymptotic fusion module for aligning the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample, and inputting the aligned feature map into the feature extraction layer to extract detailed features after fusing with the shallow features using residual blocks.

[0046] In addition, to achieve the above object, the present application also provides a drone small target detection system, and the drone small target detection system includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, it implements the steps of the drone small target detection method described in any one of the above.

[0047] In addition, to achieve the above object, the present application also provides a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the drone small target detection method described in any one of the above.

[0048] The present application has at least the following beneficial effects:

[0049] 1. Using a more lightweight deep space feature fusion module to replace the deep residual blocks to reduce channel interaction, thereby avoiding the redundancy and inefficiency of computing resources caused by a large number of stacked residual blocks in deep feature extraction;

[0050] 2. In the feature alignment task, introducing the lightweight upsampling operator DySample enables the present application to further save computing resources without sacrificing detection accuracy. Brief Description of the Drawings

[0051] Figure 1 It is a schematic flowchart of the first embodiment of the method for detecting small targets of the UAV in this application;

[0052] Figure 2 It is a schematic diagram of the construction process of the deep space feature fusion module involved in the embodiment of this application;

[0053] Figure 3 It is the step of the feature extraction layer involved in the embodiment of this application using a residual block to fuse it with the shallow feature to extract detailed features;

[0054] Figure 4 It is a schematic diagram of a multi-scale L-shaped pyramid involved in the embodiment of this application;

[0055] Figure 5 It is a deep learning network structure diagram of the embodiment of this application;

[0056] Figure 6 It is a schematic diagram of the architecture of the hardware operating environment of the UAV small target detection system involved in the embodiment of this application.

[0057] The implementation, functional features and advantages of this application will be further described with reference to the embodiments and the accompanying drawings. Detailed Embodiments

[0058] To better understand the above technical solutions, the exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0059] First Embodiment

[0060] Refer to Figure 1 , in this embodiment, the method for detecting small targets of the UAV includes the following steps:

[0061] Step S10, obtaining an image data set containing multiple small targets collected by the UAV;

[0062] In this embodiment, a small target refers to a target with a pixel area less than or equal to 1024 pixels.

[0063] Step S20, inputting the image data set into the trained deep learning network for feature extraction;

[0064] Among them, the deep learning network uses multiple deep spatial feature fusion modules to fuse semantic features of different scales, and aligns the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample, and inputs the aligned feature map into the feature extraction layer to extract detailed features after fusing with shallow features using residual blocks;

[0065] In this embodiment, YOLOv8 is used as the basic model of the deep learning network, and the backbone network of YOLOv8 is used as the backbone network of this embodiment.

[0066] In the deep learning network architecture of this embodiment, on the one hand, the deep residual blocks in the network are replaced with lightweight deep spatial feature fusion modules, while residual blocks are still used in the shallow layer to extract features.

[0067] Optionally, in some specific embodiments, as Figure 2 shown, the construction process of the deep spatial feature fusion module is as follows:

[0068] (1) Build a standard convolutional layer, a depthwise separable convolutional layer, and an atrous convolutional layer, and cascade them serially to form a deep spatial feature fusion module;

[0069] (2) Set input parameters: c1: number of input channels, c2: number of output channels, e: channel scaling factor (set the default value to 4);

[0070] (3) Divide the number of input channels c1 by the scaling factor e to obtain the number of hidden channels c_;

[0071] (4) Create the first standard convolution to reduce the number of input channels from c1 to the number of hidden channels c_, with a kernel size of 1 and a stride of 1;

[0072] (5) Create the second standard convolution to transform the concatenated number of channels c_ * 4 + c1 to the number of output channels c2, with a kernel size of 1 and a stride of 1;

[0073] (6) Create three depthwise separable convolutional layers, with a kernel size of 3 and the number of grouped convolutions being the number of hidden channels;

[0074] (7) Create three atrous convolutional layers, with a kernel size of 3 and dilation rates of 1, 2, and 3 respectively;

[0075] (8) Perform 1×1 convolution on the input x to reduce the number of channels from c1 to c_, and output y0;

[0076] (9) Pad y0 (pad = 1) to ensure that the output size remains unchanged, and use a depthwise separable convolution and an atrous convolution with a dilation rate of 1 on it, and output y1;

[0077] (10) Pad y1 (pad = 2) to ensure the output size remains unchanged, and apply depthwise separable convolution and dilated convolution with a dilation rate of 2 to it, outputting y2;

[0078] (11) Pad y2 (pad = 3) to ensure the output size remains unchanged, and apply depthwise separable convolution and dilated convolution with a dilation rate of 3 to it, outputting y3;

[0079] (12) Concatenate x, y0, y1, y2, and y3 along the channel dimension, and perform feature transformation through the convolutional layer cv2 to output the final result.

[0080] On the other hand, during the process of fusing semantic features of different scales by the depth spatial feature fusion module, the fusion strategy of the deep learning network is a dual-head asymptotic fusion strategy, and the specific dual-head asymptotic fusion strategy includes:

[0081] In the initial stage of fusion, let represent the feature map at the layer and at the stage (starting from BackBone) of this layer, and fuse the outputs from BackBone in parallel:

[0082]

[0083]

[0084] Among them, correspond to the outputs of BackBone respectively, represents the strided convolutional downsampling operation, represents the upsampling operation, and represent the intermediate stage in the downsampling stage;

[0085] In the later stage of fusion, gradually fuse the semantics of the generated intermediate layers:

[0086]

[0087]

[0088] Among them, fuses the at the intermediate layer, while further fuses the output from the deepest layer of BackBone and the output from the previous stage of P3 layer , and DSF is the depth spatial feature fusion module;

[0089] In the dual-headed asymptotic fusion strategy, the shallow layer gradually obtains the semantics of the deep layer. For When there is:

[0090]

[0091] When there is:

[0092]

[0093] In the formula, represents concatenation along the channel dimension, represents adjusting the resolution of the feature map through upsampling.

[0094] Optionally, the depth-spatial feature fusion module includes:

[0095]

[0096]

[0097] In the formula, represents the output after the semantic feature is processed by depthwise dilated convolution, is the output after the previous layer of dilated convolution processing, is the learnable parameter of the depthwise dilated convolution, represents the depthwise dilated convolution operation, is the dilation rate, is the final output feature map, represents pointwise convolution, represents channel concatenation.

[0098] Furthermore, in some specific implementation schemes, the depthwise dilated convolution operation is jointly completed by a standard convolution layer, a depthwise separable convolution layer, and a dilated convolution layer, where: the convolution kernel of the standard convolution is 3; the convolution kernel of the depthwise separable convolution is 3, the number of group convolutions is the number of hidden channels, and the number of input channels is 4; the convolution kernel of the dilated convolution is 3, and the dilation rates are 1, 2, and 3 respectively.

[0099] On the other hand, the fused feature map is aligned to the same resolution as the second convolution layer through the lightweight upsampling operator DySample, and the aligned feature map is input into the feature extraction layer to extract detailed features after fusing with the shallow features using residual blocks.

[0100] Specifically, a sampling point generator is used to generate dynamic sampling offsets for each pixel point, and then the original grid position is added to the dynamic sampling offsets to obtain a sampling set;

[0101] Resample the input fused feature map through a grid sampling function to obtain the upsampled feature;

[0102] Among them, the generation strategies of the sampling point generator include a static range factor strategy and a dynamic range factor strategy. The static range factor strategy includes: generating a dynamic sampling offset through a linear layer.

[0103] The dynamic range factor strategy includes: first generating a range factor through content awareness, and generating the dynamic sampling offset by passing the range factor through a linear layer and pixel transformation.

[0104] Optionally, in a specific embodiment, referring to Figure 3 , the steps of the feature extraction layer using a residual block to fuse it with the shallow feature to extract detailed features can be as follows:

[0105] (1) Use DySample to upsample C5 to the C4 resolution, and use a 2*2 convolution to downsample C2 to the C3 resolution;

[0106] (2) Use a depth spatial feature fusion module to fuse the sampled C5 and C4, C2 and C3 respectively to obtain C4´, C3´;

[0107] (3) Use DySample to upsample C4´ to the C3 resolution, and use a depth spatial feature fusion module to fuse C4´ and C3´ to obtain C3´´;

[0108] (4) Use DySample to upsample C5 to the C3 resolution, and use a depth spatial feature fusion module to fuse C5 and C3´´ to obtain C3´´´;

[0109] (5) Use DySample to upsample C3´, C3´´, C3´´´ to the C2 resolution respectively, and use a residual block to extract features respectively to obtain C2´, C2´´, C2´´´.

[0110] Further, use Concat to fuse C2 and C2´, C2 and C2´´, C2 and C2´´´, C2´ and C2´´, C2´´ and C2´´´ respectively, and use a residual block to extract the features in the fused feature map;

[0111] Use the Concat architecture to splice the above-mentioned fused feature maps;

[0112] Arrange the above-mentioned feature maps of each layer in sequence from C2 to C5 to obtain a multi-scale L-shaped pyramid, as Figure 4 shown.

[0113] Step S30, obtain the predicted category and predicted box coordinates corresponding to each image output by the deep learning network;

[0114] Step S40, screen out the target prediction boxes without overlap according to the coordinates of the prediction boxes, so as to mark the target positions and target categories corresponding to each small target on the image based on the target prediction boxes.

[0115] In this embodiment, the test set images in the dataset are adjusted to 640×64_, and input into the trained deep learning network for forward propagation to obtain the prediction categories and prediction box coordinates of each image.

[0116] Optionally, the non-maximum suppression method can be used to screen out the overlapping prediction boxes, and the final prediction boxes are used to mark the positions and categories of the targets on the image to complete the target detection.

[0117] In the technical solution provided in this embodiment, on the one hand, a lighter deep spatial feature fusion module is used to replace the deep residual blocks to reduce channel interaction, thereby avoiding the redundancy and inefficiency of computing resources caused by a large number of stacked residual blocks in deep feature extraction; on the other hand, in the feature alignment task, a lightweight upsampling operator DySample is introduced, so that the present application further saves computing resources without sacrificing detection accuracy.

[0118] Second Embodiment

[0119] As an optional embodiment, a construction and training method based on the deep learning network involved in the first embodiment is provided in this embodiment.

[0120] Specifically, for the construction link, refer to Figure 5 The shown deep learning network structure diagram. Based on YOLOv8, the prediction module of the fully convolutional one-stage object detection YOLOv8 framework is selected, which includes a classification sub-network and a bounding box regression sub-network. The backbone network of YOLOv8 adopts a modular design, and feature extraction and downsampling are performed through components such as convolutional layers, and the input image is encoded into a multi-level feature map for subsequent detection heads to perform target localization and classification. The convolutional kernel size of the convolutional layer is 3×3, and the stride is 1 or 2. When it is 2, downsampling is performed, and the number of channels gradually increases: 64 → 128 → 256 → 512 → 1024.

[0121] For the training process, the data is normalized to the range [0, 1], and the image size is adjusted to 640×640. Set the hyperparameters for training the deep learning network: the number of iterations (epoch) is set to 300, the learning rate is set to 0.01, and the number of images input at one time (batch) is set to 8. Then, batch images from the training sample set are input into the deep learning network for forward propagation to obtain the predicted class and the coordinates of the predicted bounding box. The CIoU loss function is used to calculate the cross-entropy loss between the target class and the predicted class. The cross-entropy loss is used to measure whether the model correctly predicts the target. Finally, metrics such as mAP50 and mAP50-95 are calculated on the validation set for each epoch to measure the detection accuracy, and the performance changes for each epoch are recorded to determine whether the model is overfitting or underfitting. Training is terminated after reaching the specified number of epochs.

[0122] Third Embodiment

[0123] In this embodiment, the effect of the deep learning network model proposed in the foregoing embodiment is further described:

[0124] (1) Simulation experiment conditions:

[0125] The experiments in this embodiment are carried out on the Windows operating system. The experimental code is written in Python (3.9), implemented based on the deep learning framework PyTorch (2.4.0+cu121), and run on an NVIDIA 4060 Ti GPU workstation.

[0126] (2) Simulation experiment content and result analysis

[0127] On the above-mentioned publicly available dataset VisDrone2019, the deep learning network model proposed in this application and the object detection methods of the prior art are respectively used for simulation and comparative analysis to obtain the detection bounding boxes and classes of the objects. The average precision value (mAP50) and the mean average precision (mAP50-95) calculated within the range of IoU thresholds from 0.50 to 0.95 (increasing by 0.05 each time) are used as evaluation metrics. The detection results of the simulation experiment are shown in Table 1:

[0128] Table 1 Detection results of the present invention and the prior art in the simulation experiment

[0129] As can be seen from Table 1, compared with the prior arts GFL and RetinaNet, under the condition of the same input image size, the present invention has higher accuracy, and the computational cost is far less than the two. Compared with YOLOv5n, although the computational cost of the present invention is larger, the detection accuracy is significantly improved: mAP50 is 11.9% higher than YOLOv5n, and mAP50-95 is 9.4% higher than YOLOv5n. This fully shows that the present invention can significantly improve the inference speed while ensuring high accuracy. Especially in the small target detection task, the present invention not only has advantages in computational efficiency, but also achieves higher accuracy in the small target detection task, can effectively reduce the computational cost without sacrificing the detection performance of the model, meets the strict requirements for computational efficiency in practical applications, has good practical application value, and is particularly suitable for resource-limited embedded or edge computing devices.

[0130] In addition, as an implementation solution, the present application also proposes a deep learning network model, which includes:

[0131] The YOLOv8 architecture;

[0132] A plurality of deep spatial feature fusion modules for fusing semantic features of different scales in the image dataset containing multiple small targets collected by the drone;

[0133] A dual-head asymptotic fusion module for aligning the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample, and inputting the aligned feature map into the feature extraction layer to extract detailed features after fusing with the shallow features using the residual block.

[0134] In addition, as an implementation solution, Figure 6 It is a schematic diagram of the architecture of the hardware operating environment of the small target detection system for drones involved in the embodiment solution of the present application.

[0135] Such as Figure 6As shown in the figure, the small target detection system of the unmanned aerial vehicle may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0136] Those skilled in the art can understand that Figure 6 the architecture of the small target detection system of the unmanned aerial vehicle shown in the figure does not constitute a limitation on the small target detection system of the unmanned aerial vehicle, and may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0137] As Figure 6 shown in the figure, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and a computer program. Among them, the operating system is a program for managing and controlling the hardware and software resources of the small target detection system of the unmanned aerial vehicle, and the operation of the computer program and other software or programs.

[0138] In Figure 6 the small target detection system of the unmanned aerial vehicle shown in the figure, the user interface 1003 is mainly used to connect to the terminal and perform data communication with the terminal; the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server; the processor 1001 may be used to call the computer program stored in the memory 1005.

[0139] In this embodiment, the small target detection system of the unmanned aerial vehicle includes: a memory 1005, a processor 1001, and a computer program stored on the memory and executable on the processor, where:

[0140] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0141] Obtain an image data set containing multiple small targets collected by the unmanned aerial vehicle;

[0142] Input the image data set into the trained deep learning network for feature extraction;

[0143] Among them, the deep learning network uses multiple deep spatial feature fusion modules to fuse semantic features of different scales, and aligns the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample, and inputs the aligned feature map into the feature extraction layer to extract detailed features after fusing with shallow features using residual blocks;

[0144] Obtain the predicted category and predicted bounding box coordinates corresponding to each image output by the deep learning network;

[0145] According to the predicted bounding box coordinates, filter out the target predicted bounding boxes without overlap, so as to mark the target positions and target categories corresponding to each small target on the image based on the target predicted bounding boxes.

[0146] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0147] In the initial stage of fusion, let represent the feature map at the layer and at the stage (starting from BackBone) of this layer, and fuse the outputs from BackBone in parallel:

[0148]

[0149]

[0150] Among them, respectively correspond to the outputs of BackBone , represents the strided convolutional downsampling operation, represents the upsampling operation, and represent the intermediate stage in the downsampling stage;

[0151] In the later stage of fusion, gradually fuse the semantics of the generated intermediate layers:

[0152]

[0153]

[0154] Among them, fuses at the intermediate layer, while further fuses the output of the deepest layer of BackBone and the output of the previous stage of P3 layer , and DSF is the deep spatial feature fusion module;

[0155] In the dual - head asymptotic fusion strategy, the shallow layer obtains the semantics of the deep layer in an asymptotic manner. For When there is:

[0156]

[0157] When there is:

[0158]

[0159] In the formula, represents concatenation by channel dimension, represents adjusting the resolution of the feature map through upsampling.

[0160] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0161] Use a sampling point generator to generate a dynamic sampling offset for each pixel point, and then add the original grid position to the dynamic sampling offset to obtain a sampling set;

[0162] Resample the input fused feature map through a grid sampling function to obtain the upsampled feature;

[0163] Among them, the generation strategy of the sampling point generator includes a static range factor strategy and a dynamic range factor strategy. The static range factor strategy includes: generating a dynamic sampling offset through a linear layer; the dynamic range factor strategy includes: first generating a range factor through content awareness, and generating the dynamic sampling offset by passing the range factor through a linear layer and pixel transformation.

[0164] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0165] Adopt the non - maximum suppression method to screen out the overlapping prediction boxes, and use the remaining prediction boxes as the target prediction boxes.

[0166] In addition, those of ordinary skill in the art can understand that all or part of the processes in the methods of implementing the above - mentioned embodiments can be completed by instructing relevant hardware through a computer program. This computer program includes program instructions, and the computer program can be stored in a storage medium, and this storage medium is a computer - readable storage medium. The program instructions are executed by at least one processor in the small - target detection system of the unmanned aerial vehicle to implement the process steps of the embodiments of the above - mentioned method.

[0167] Therefore, the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, each step of the drone small target detection method described in the above embodiments is implemented.

[0168] Wherein, the computer-readable storage medium may be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disc, etc., various computer-readable storage media that can store program codes.

[0169] It should be noted that since the storage medium provided in the embodiments of the present application is the storage medium adopted for implementing the method of the embodiments of the present application, those skilled in the art can understand the specific structure and variations of the storage medium based on the method introduced in the embodiments of the present application, so it will not be elaborated here. Any storage medium adopted for the method of the embodiments of the present application falls within the scope to be protected by the present application.

[0170] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0171] The present application is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can also be implemented. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the specified functions in one Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0172] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the specified functions in one Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, thereby the instructions executed on the computer or other programmable apparatus provide steps for realizing the functions specified in one process or multiple processes and / or one block or multiple blocks. Figure 1 one process or multiple processes and / or blocks Figure 1 steps for realizing the functions specified in one block or multiple blocks.

[0174] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim listing several means, several of these means can be embodied by the same item of hardware. The use of the words first, second, third, etc. does not denote any order. These words can be interpreted as names.

[0175] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made by those skilled in the art once they learn the basic inventive concept. Therefore, the appended claims are intended to be construed to cover the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0176] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A method for detecting small targets of an unmanned aerial vehicle, characterized in that, The method includes the following steps: Obtain an image dataset containing multiple small targets collected by a drone; Input the image dataset into a trained deep learning network for feature extraction; Among them, the deep learning network uses multiple deep spatial feature fusion modules to fuse semantic features of different scales, and aligns the fused feature map to the same resolution as the second convolutional layer through a lightweight upsampling operator DySample, and inputs the aligned feature map into the feature extraction layer to extract detailed features after fusing with shallow features using residual blocks; Obtain the predicted category and predicted bounding box coordinates corresponding to each image output by the deep learning network; Filter out non-overlapping target prediction bounding boxes according to the predicted bounding box coordinates, so as to mark the target positions and target categories corresponding to each small target on the image based on the target prediction bounding boxes.

2. The method according to claim 1, characterized in that, The deep spatial feature fusion module includes: ； ； In the formula, represents the output after the semantic features are processed by deep atrous convolution, is the output after the previous atrous convolution processing, are the learnable parameters of the deep atrous convolution, represents the deep atrous convolution operation, is the dilation rate, is the final output feature map, represents the pointwise convolution, represents the channel concatenation.

3. The method according to claim 2, wherein The depthwise dilated convolution operation is jointly completed by a standard convolutional layer, a depthwise separable convolutional layer, and a dilated convolutional layer, where: The convolutional kernel of the standard convolution is 3; The convolutional kernel of the depthwise separable convolution is 3, the number of group convolutions is the number of hidden channels, and the number of input channels is 4; The convolutional kernel of the dilated convolution is 3, and the dilation rates are 1, 2, and 3 respectively.

4. The method according to claim 1, characterized in that, The fusion strategy of the deep learning network is a dual-head asymptotic fusion strategy, and the dual-head asymptotic fusion strategy specifically includes: In the initial stage of fusion, assume represents being in layer, and the feature map at the following stage, and fuses the outputs from BackBone in parallel: ； ； Among them, respectively correspond to the output of the BackBone , represents the strided convolutional downsampling operation, represents the upsampling operation, and represents the intermediate stage in the downsampling phase; In the later stage of fusion, gradually fuse the semantics of the generated intermediate layers: ； ； Among them, the fusion is at the middle layer , and further fuses the output from the deepest layer of the BackBone and the output of the previous stage on the P3 layer , and DSF is the deep space feature fusion module; In the dual-head asymptotic fusion strategy, the shallow layer obtains the semantics of the deep layer in an asymptotic manner. For , when , there is: ； When there is ； In the formula, denotes concatenation along the channel dimension, denotes adjusting the resolution of the feature map through upsampling.

5. The method according to claim 1, characterized in that The step of aligning the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample specifically includes: Use a sampling point generator to generate dynamic sampling offsets for each pixel point, and then add the original grid position and the dynamic sampling offsets to obtain a sampling set; Resample the input fused feature map through a grid sampling function to obtain the upsampled features; Among them, the generation strategy of the sampling point generator includes a static range factor strategy and a dynamic range factor strategy. The static range factor strategy includes: generating dynamic sampling offsets through a linear layer; the dynamic range factor strategy includes: first generating a range factor through content awareness, and generating the dynamic sampling offsets through a linear layer and pixel transformation.

6. The method according to claim 1, characterized in that, The hyperparameter settings during the training process of the deep learning network are: The number of iterations is set to 300, the learning rate is set to 0.01, and the number of images input at one time batch is set to 8.

7. The method according to claim 1, wherein The filtering out of non-overlapping target prediction bounding boxes according to the predicted bounding box coordinates includes: Use the non-maximum suppression method to filter out overlapping prediction bounding boxes, and regard the remaining prediction bounding boxes as the target prediction bounding boxes.

8. A deep learning network model, characterized in that, The deep learning network model includes: YOLOv8 architecture; Multiple deep spatial feature fusion modules for fusing semantic features of different scales in an image dataset containing multiple small targets collected by a drone; The dual-head asymptotic fusion module is used to align the fused feature map to the same resolution as the second convolutional layer through the lightweight upsampling operator DySample, and input the aligned feature map into the feature extraction layer to extract detailed features after fusing with the shallow features using residual blocks.

9. A small target detection system for unmanned aerial vehicles, characterized in that, The drone small target detection system includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the drone small target detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, it implements the steps of the drone small target detection method according to any one of claims 1 to 7.

Images