Lightweight network structure and method for real-time target detection from the perspective of local security drones

Abstract

The present invention relates to a lightweight network structure and method for real-time target detection from the perspective of a local security drone, belonging to the field of low-altitude security technology in a local security system, and specifically to a lightweight method for real-time target detection from the perspective of a drone for local security. The method comprises: (1) constructing a lightweight target detection model framework; (2) designing an efficient feature extraction module; and (3) training the model using an improved training strategy. Compared with existing methods, the present invention achieves a better balance between detection accuracy, detection speed, and model parameter quantity by designing a more efficient target detection framework and feature extraction module, in conjunction with an improved training strategy. The proposed method is suitable for deployment on drone platforms with limited computing resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of low-altitude security in a local security system, and relates to a lightweight network structure and method for real-time target detection from the perspective of a local security drone. Background Art

[0002] In modern local security systems, drones are widely used as efficient reconnaissance tools for critical tasks such as cross-domain reconnaissance, intelligent search and rescue, and disaster warning. Object detection, a key technology for enhancing drone perception capabilities, has become a research hotspot and garnered widespread attention. However, due to the computing power limitations of drone platforms, the scale of object detection models is limited. Existing methods often suffer from large model parameters, slow processing speed, or low recognition accuracy. Achieving an ideal balance between detection accuracy and speed, and the number of model parameters, is challenging.

[0003] At present, a large number of scholars have conducted research on the above issues.

[0004] In the paper "Z. Zhang, X. Lu, G. Cao, Y. Yang, L. Jiao, and F. Liu, Vit-Yolo: Transformer-Based Yolo for Object Detection, in Proceedings of the IEEE international conference on computer vision, pages: 2799-2808, 2021," Z. Zhang et al. proposed ViT-YOLO, based on YOLOv4-P7. ViT-YOLO enhances the model's ability to process global context and extract features by integrating multi-head self-attention (MHSA) into the CSP-Darknet backbone. ViT-YOLO also uses a weighted bidirectional feature pyramid network (BiFPN) to effectively fuse cross-scale features, improving the model's recognition accuracy. However, the use of MHSA and BiFPN increases the model's complexity and computing power requirements, making it unsuitable for deployment on drone platforms. In the paper “HVKoay, JHChuah, C.-O.Chow, Y.-L.Chang, and KKYong, Yolo-Rtuav: Towards Real-Time Vehicle Detection through Aerial Images with Low-Cost Edge Devices, Remote Sensing, 13(21): 4196, 2021,” HVKoay et al. used multiple 1×1 convolutions based on YOLOv4-tiny to reduce the complexity of the model and shorten the inference time. However, this resulted in a slight decrease in detection accuracy, and the overall performance of the model was not significantly improved compared to the baseline. Therefore, it is necessary to provide a lightweight method for real-time target detection from a drone’s perspective in order to solve the above problems. Summary of the Invention

[0005] Technical problems to be solved

[0006] In order to avoid the shortcomings of the existing technology, the present invention proposes a lightweight network structure and method for real-time target detection from the perspective of a local security drone. By designing a reasonable target detection framework, an efficient feature extraction module, and an improved training strategy, the problems of the existing method such as low detection accuracy, poor real-time performance, large number of model parameters, and unsuitability for deployment on drone-mounted equipment are alleviated.

[0007] Technical Solution

[0008] A lightweight network structure for real-time target detection from the perspective of a ground-based security drone is characterized by comprising a backbone network, a neck and a detection head; specifically comprising a downsampling module and an upsampling module, an inverted part extended residual module and a fast spatial pyramid pooling module and a detection head module; the backbone network is: the image input end is a series of the first 4-fold downsampling module, the inverted part extended residual module × 3, the first 2-fold downsampling module, the first inverted part extended residual module, the second 2-fold downsampling module and the second inverted part extended residual module; the neck is: the output of the second inverted part extended residual module is connected to the fast spatial pyramid pooling module, and the output of one path is connected to the inverted part after passing through the 4-fold upsampling module. The output of the extended residual module × 3 is spliced ​​according to the channel dimension, and then sent to the Head2 detection head for decoding after passing through the third inverted partial extended residual module, and finally the detected content is output; the other output of the third inverted partial extended residual module is spliced ​​with the other output of the fast spatial pyramid pooling module according to the channel dimension after passing through the second 4x downsampling module, and then sent to the Head4 detection head for decoding after passing through the fourth inverted partial extended residual module, and finally the detected content is output; the Head2 detection head and the Head4 detection head are detection head modules, and the merging of their output contents after the non-maximum suppression operation is a set of high-confidence and non-overlapping final detection results.

[0009] The 4x downsampling module is a series structure of two 2x downsampling modules.

[0010] The 2x downsampling module is a series of 3×3 convolution, batch normalization layer and SiLU activation layer, where the stride of the 3×3 convolution is equal to 2.

[0011] The inverted partial expansion residual module is a series of 1×1 convolution, 3×3 partial expansion convolution and 1×1 convolution with residual connection, wherein the 3×3 partial expansion convolution is a part split by the channel dimension, which is subjected to 1×1 convolution, 3×3 grouped convolution and 1×1 convolution, and then merged with the other part split by the channel dimension according to the channel dimension.

[0012] The inverted partially expanded residual module ×3 is composed of three inverted partially expanded residual modules stacked in series.

[0013] The detection head module is Head2 corresponding to the 4-fold down-sampling feature map and Head4 corresponding to the 16-fold down-sampling feature map; or Head3 corresponding to the 8-fold down-sampling feature map, Head4 corresponding to the 16-fold down-sampling feature map or Head5 corresponding to the 32-fold down-sampling feature map.

[0014] When training the network structure, the classification loss function is binary cross entropy loss, and the regression loss function is complete intersection-over-union loss CIoU loss and distribution focal loss Distribution Focal Loss; the optimizer used in training is stochastic gradient descent SGD, with a momentum of 0.9 and a weight decay of 5e-4; cosine annealing learning rate scheduling is used, and a total of 300 rounds of training are used, with the first 3 rounds being warm-up training, an initial learning rate of 0.02, and a batch size of 32.

[0015] The network structure training process is as follows: the standard multi-round training process is divided into four stages, and the intensity of data augmentation gradually decreases during the training process from the first stage to the fourth stage. Specifically, in the first stage of training, in addition to using the four conventional data augmentation methods of translation, horizontal flipping, scaling, and HSV hue conversion, three strong data augmentation methods of MixUp, Mosaic, and CopyPaste are also used. Starting from the second stage, one strong data augmentation method is removed in each stage in the order of MixUp, Mosaic, and CopyPaste. In the fourth training stage, only the above four conventional data augmentation methods are performed on the training data, and the optimal model is finally trained.

[0016] A lightweight method for real-time target detection from the perspective of a ground-based security drone is characterized by adopting the aforementioned network structure, including a backbone network extracting features, a neck performing feature interaction, and a detection head predicting the bounding box and category of the target in the image based on the neck features. The method steps are as follows:

[0017] Backbone network data processing process: The size of the input image is B×3×H×W, where B represents the batch size, 3 represents the number of channels, and H and W represent the height and width of the image respectively;

[0018] The image first passes through the first 4x downsampling module, the spatial resolution is reduced to 4 times the original, the number of channels becomes C1, and the feature size is B×C1×H / 4×W / 4. Figure 1 b;

[0019] feature Figure 1 b After the inverted part of the expanded residual module × 3, the shallow fine-grained features are extracted to obtain the shallow fine-grained features Figure 1 c. The dimensions of the feature map in this step remain unchanged;

[0020] feature Figure 1 After the first 2x downsampling module, the spatial resolution is reduced by 2 times, and the number of channels becomes C2, resulting in a feature with a shape of B×C2×H / 8×W / 8. Figure 1 d;

[0021] feature Figure 1d is expanded through the residual module after the first inversion part to obtain the feature Figure 1 e. The dimensions of the feature map in this step remain unchanged;

[0022] feature Figure 1 After the second 2x downsampling module, the spatial resolution is reduced by 2 times, and the number of channels becomes C3, resulting in a feature with a shape of B×C3×H / 16×W / 16. Figure 1 f;

[0023] feature Figure 1 fAfter the second inverted part of the extended residual module, the deep high-level semantic features of 16 times downsampling are extracted Figure 1 g; the dimensions of the feature map in this step remain unchanged;

[0024] Shallow fine-grained features Figure 1 c and deep high-level semantic features Figure 1 g is sent to the neck data processing for further feature fusion and interaction;

[0025] Neck data processing process:

[0026] Deep high-level semantic features Figure 1 g is processed through the fast spatial pyramid pooling module to perform maximum pooling operations of different scales on the feature map to obtain the pooled features. Figure 2 a, which captures semantic information under different receptive field sizes;

[0027] Features after pooling Figure 2 a is interpolated through a 4-fold upsampling module, and the spatial resolution H and W are magnified by 4 times to obtain the feature Figure 2 b, Features Figure 2 The shape of b becomes B×C3×H / 4×W / 4;

[0028] feature Figure 2 b and shallow fine-grained features Figure 1 c is concatenated according to the channel dimension to obtain a merged feature with a dimension of B×(C1+C3)×H / 4×W / 4 Figure 2 c;

[0029] Merge Features Figure 2 c After the third inverted part of the extended residual module, the deep and shallow features are further interactively fused, and the channel dimension of the feature map is adjusted to C1, and the feature map with the dimension of B×C1×H / 4×W / 4 is obtained. Figure 2 d;

[0030] feature Figure 2 d After the second 4-fold downsampling module, the spatial resolution H, W is reduced by 4 times to obtain the feature Figure 2 e, Features Figure 2The shape of e is B×C1×H / 16×W / 16;

[0031] feature Figure 2 e and features after pooling module Figure 2 a is concatenated according to the channel dimension to obtain a merged feature with a dimension of B×(C1+C3)×H / 16×W / 16 Figure 2 f;

[0032] Merge Features Figure 2 After the fourth inverted part of the extended residual module, the features are further interactively fused, and the channel dimension of the feature map is adjusted to C3, resulting in a feature with a dimension of B×C3×H / 16×W / 16. Figure 2 g;

[0033] The features Figure 2 g and characteristics Figure 2 d is sent to the detection head to decode the feature map and generate the final detection result;

[0034] Detection head data processing process:

[0035] feature Figure 2 d is used as the input of the detection head 2, and after two parallel convolution branches, the output is the detection result 3a containing the bounding box and category of the detected object;

[0036] feature Figure 2 g is used as the input of the detection head 4, and after two parallel convolution branches, the output is the detection result 3b containing the bounding box and category of the detected target; the convolution branch is a 3×3 convolution module → 1x1 convolution.

[0037] A non-maximum suppression operation is performed on detection results 3a and 3b to remove a large number of redundant and low-quality prediction boxes, and a set of final detection results with high confidence and non-overlapping is obtained.

[0038] A computer program product, characterized in that it includes computer-executable instructions, which, when executed, are used to implement the lightweight method for real-time target detection from the perspective of a nearby security drone.

[0039] Beneficial effects

[0040] The present invention proposes a lightweight network structure and method for real-time target detection from the perspective of a local security drone, which belongs to the field of low-altitude security technology in the local security system, and specifically relates to a lightweight method for real-time target detection from the perspective of a drone for local security. It includes: (1) constructing a lightweight target detection model framework; (2) designing an efficient feature extraction module; (3) training the model using an improved training strategy. Compared with existing methods, the present invention achieves a better balance between detection accuracy, detection speed, and model parameter quantity by designing a more efficient target detection framework and feature extraction module, combined with an improved training strategy. The proposed method is suitable for the deployment of drone platforms with limited computing resources.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The present invention provides a lightweight method for real-time target detection from a drone's perspective. Based on the proposed cross-scale mechanism, a minimalist model architecture is constructed in which a backbone network performs only 16x downsampling, and the neck and detection head only retain the corresponding modules of stage 2 and stage 4. This method effectively takes into account the number of model parameters, inference delay, and detection accuracy. Specifically, the number of parameters is reduced from 3.01M to 0.79M (a reduction of 2.22M), the inference delay is reduced from 2.9ms to 2.6ms (a reduction of 0.3ms), and the detection accuracy AP is increased from 18.4% to 19.9% ​​(an increase of 1.5%).

[0043] 2. This paper provides a lightweight method for real-time target detection from a drone's perspective. By introducing an inverted, partially expanded residual module for efficient feature extraction, this method improves detection accuracy while significantly reducing inference latency. Specifically, detection accuracy (AP) increased from 19.9% ​​to 20.1% (an improvement of 0.2%), and inference latency decreased from 2.6ms to 2.0ms (a reduction of 0.6ms).

[0044] 3. This paper provides a lightweight method for real-time object detection from a drone's perspective. This method incorporates a training strategy using attenuated data augmentation within the standard training rounds. By dividing the 300-round training process into four phases and removing each phase in the order of MixUp, Mosaic, and CopyPaste, the model's generalization performance is improved. Specifically, detection accuracy (AP) increases from 20.1% to 20.3% (a 0.2% improvement). BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of a lightweight method for real-time target detection from an unmanned aerial vehicle perspective according to the present invention.

[0046] Figure 2This is a YOLOv8-N model architecture diagram for a lightweight method for real-time target detection from a drone's perspective.

[0047] Figure 3 This is a diagram of the cross-scale detection head mechanism model architecture of a lightweight method for real-time target detection from a drone perspective in the present invention.

[0048] Figure 4 This is a cross-scale neck mechanism model architecture diagram of a lightweight method for real-time target detection from a drone perspective in the present invention.

[0049] Figure 5 This is a diagram of the enhanced backbone network mechanism model architecture of a lightweight method for real-time target detection from a drone perspective in the present invention.

[0050] Figure 6 This is a structural diagram of the FasterNet module of a lightweight method for real-time target detection from the perspective of a drone.

[0051] Figure 7 This is a structural diagram of the inverted part extended residual module of a lightweight method for real-time target detection from a drone perspective in the present invention.

[0052] Figure 8 This is a diagram of the CS-Det-N model architecture of a lightweight method for real-time target detection from a drone's perspective.

[0053] Figure 9 This is the ablation experiment roadmap for the lightweight method of real-time target detection from a drone's perspective. The corresponding beneficial effects section shows the performance improvements brought about by each innovation.

[0054] Figure 10 This is a line chart comparing the AP-inference latency of a lightweight method for real-time target detection from a drone's perspective and the state-of-the-art algorithms from the past two years. This is a visualization of some of the data in Table 1.

[0055] Figure 11 This is a line chart comparing the AP-parameters of a lightweight method for real-time target detection from a drone perspective and the state-of-the-art algorithms from the past two years. This is a visualization of some of the data in Table 1. DETAILED DESCRIPTION

[0056] The present invention will now be further described with reference to the embodiments and accompanying drawings:

[0057] In an embodiment of the present invention, a lightweight target detection model framework is constructed. A cross-scale mechanism Cross-Scale Mechanisms is proposed for efficient model framework design, which mainly includes the following three sub-mechanisms. Taking the YOLOv8 target detection model as a benchmark, firstly, based on the cross-scale detection head mechanism, the Head2 detection head with 4 times downsampling and the Head4 detection head with 16 times downsampling are selected to form the model detection head, and the model with the optimal balance between speed and accuracy is obtained. On this basis, the cross-scale neck mechanism is used to further remove the model neck stage 5 and stage 3 modules, and the stage 5 of the backbone network is also removed at the same time. Finally, based on the strengthening backbone network mechanism, the stage 2 of the backbone network is enhanced. A minimalist model architecture is obtained in which the backbone network is only downsampled 16 times, and the neck and detection head only retain the corresponding modules of stage 2 and stage 4.

[0058] Design an efficient feature extraction module. First, replace the C2f module with the FasterNet module for efficient feature extraction. Then, relocate the 3×3 partial convolution in the FasterNet module between two 1×1 convolutions to form an inverted bottleneck module. Deep reparameterization techniques and a grouping strategy are used to expand the 3×3 partial convolution to further enhance its representational capabilities, resulting in an inverted partial expanded residual module.

[0059] Among them: Lightweight target detection model framework. Specifically including (1a) to (1c):

[0060] (1a) Cross-scale detection head mechanism. Based on the YOLOv8 target detection algorithm, its structure is referenced Figure 2 . It is agreed that Head1-Head5 represent 2x, 4x, 8x, 16x, and 32x downsampling detection heads respectively. By removing the original Head3 and Head5 detection heads, adding the Head2 detection head, and modifying the neck module accordingly, only the 4x downsampling Head2 detection head and the 16x downsampling Head4 detection head are used to form a cross-scale detection head configuration. For details, refer to Figure 2 For the neck, remove the fifth 2-fold downsampling module and the subsequent channel-wise splicing operation and the sixth C2f_1 module in stage 5. For the detection head, remove the Head3 detection head and the Head5 detection head. Figure 3 , add the corresponding modules of the neck stage 2, that is, add the third 2x upsampling module and the subsequent splicing operation according to the channel dimension, the fifth C2f_1 module, the fourth 2x downsampling module and the subsequent splicing operation according to the channel dimension, and the sixth C2f_1 module.

[0061] This cross-scale detection head design can efficiently utilize the extracted multi-scale features to detect targets at different scales. The Head2 detection head focuses on detecting ultra-small targets in drone-view images, while the Head4 detection head focuses on detecting larger targets. Compared with the default Head3-Head5 three-head setting, the cross-scale design paradigm can significantly reduce the number of model parameters, improve inference speed, and achieve better detection performance.

[0062] (1b) Cross-scale neck mechanism. Based on (1a), remove the modules in the stage where the model neck is not directly connected to the detection head, that is, remove the modules in stage 5 and stage 3 of the neck. For details, refer to Figure 3 , first adjust the position of the fast spatial pyramid pooling module to connect it to the output of the second C2f_2 module. Remove the first 2x upsampling block and the subsequent splicing operation according to the channel dimension, the third C2f_1 module, the second 2x upsampling block and the subsequent splicing operation according to the channel dimension, and the fourth C2f_1 module. Change the third 2x upsampling module to a 4x upsampling module. Remove the fourth 2x downsampling module and the subsequent splicing operation according to the channel dimension, and the sixth C2f_2 module. Change the fifth 2x downsampling module to a 4x downsampling module. Furthermore, when the stage 5 of the neck is removed, the stage 5 of the backbone network is no longer necessary, so the third 2x downsampling module and the second C2f_1 module are also removed at the same time. The model structure finally obtained in this step refers to Figure 4 .

[0063] The cross-scale neck design enables the neck to adapt to the structure of the detection head, thereby more effectively utilizing model parameters and reducing inference latency.

[0064] (1c) Strengthen the backbone network mechanism. First, increase the number of bottleneck blocks n in the C2f_1 module in the early stage 2 of the backbone network. The actual value of n is 3, and the C2f_3 module is obtained. Then, the C2f_1 module is replaced by two serial C2f_3 modules. The final model structure obtained in this step is referenced. Figure 5 .

[0065] By strengthening the representation capabilities of the early stage 2 of the backbone network, the quality of the input features of the neck and detection head at the same stage is improved, thereby enhancing the model's ability to detect small objects. The extracted effective features also facilitate learning in subsequent stages. The result is a cross-scale model architecture that only performs 16x downsampling, with the backbone consisting of only three stages, the neck consisting of only stages 2 and 4, and the detection heads retaining only Head 2 and Head 4.

[0066] The efficient feature extraction module specifically includes (2a) to (2c):

[0067] (2a) Embed FasterNet module. Use FasterNet module to replace the default C2f module used by YOLOv8 to improve feature extraction efficiency. The C2f module structure is referenced Figure 2 , the FasterNet module structure refers to Figure 6 .

[0068] (2b) Repositioning the 3×3 partial convolution. The 3×3 partial convolution is repositioned in the FasterNet module and moved between two 1×1 convolutions to form an inverted bottleneck structure. This allows for efficient feature extraction using slice-based in-place operations during inference.

[0069] (2c) Expand and group the 3×3 partial convolution. Use the depth reparameterization technique to expand the convolution branch of the 3×3 partial convolution in the training phase into three consecutive convolutions: 1×1 convolution, 3×3 convolution, and 1×1 convolution, to increase the depth. And set the expansion rate between each layer to 4 to increase the width. In addition, using the idea of ​​grouping, replace the 3×3 convolution with grouped convolution to effectively improve the representation ability of the model and reduce the training cost. The inverted partial expansion residual module is constructed, and the specific structure is referenced. Figure 7 It is worth noting that in the inference phase, the 3×3 partial expansion convolution in the inverted partial expansion residual module can be reparameterized into a standard 3×3 partial convolution to maintain extremely low inference latency. The final constructed model structure refers to Figure 8 .

[0070] The specific network structure includes a backbone network, a neck and a detection head; specifically, it includes a downsampling module and an upsampling module, an inverted part extended residual module and a fast spatial pyramid pooling module and a detection head module; the backbone network is: the image input end is the first 4-fold downsampling module, the inverted part extended residual module × 3, the first 2-fold downsampling module, the first inverted part extended residual module, the second 2-fold downsampling module and the second inverted part extended residual module in series; the neck is: the output of the second inverted part extended residual module is connected to the fast spatial pyramid pooling module, and the output of one path is connected to the output of the inverted part extended residual module × 3 after the 4-fold upsampling module. The output of the third inverted partially expanded residual module is spliced ​​in the channel dimension, and then sent to the Head2 detection head for decoding after passing through the third inverted partially expanded residual module, and finally the detected content is output; the other output of the third inverted partially expanded residual module is spliced ​​with the other output of the fast spatial pyramid pooling module according to the channel dimension after passing through the second 4x downsampling module, and then sent to the Head4 detection head for decoding after passing through the fourth inverted partially expanded residual module, and finally the detected content is output; the Head2 detection head and the Head4 detection head are detection head modules, and the merging of their output contents after the non-maximum suppression operation is a set of high-confidence and non-overlapping final detection results.

[0071] The 4x downsampling module is a series structure of two 2x downsampling modules.

[0072] The 2x downsampling module is a series of 3×3 convolution, batch normalization layer and SiLU activation layer, where the stride of the 3×3 convolution is equal to 2.

[0073] The inverted partial expansion residual module is a series of 1×1 convolution, 3×3 partial expansion convolution and 1×1 convolution with residual connection, wherein the 3×3 partial expansion convolution is a part split by the channel dimension, which is subjected to 1×1 convolution, 3×3 grouped convolution and 1×1 convolution, and then merged with the other part split by the channel dimension according to the channel dimension.

[0074] The inverted partially expanded residual module ×3 is composed of three inverted partially expanded residual modules stacked in series.

[0075] The detection head module is Head2 corresponding to the 4-fold down-sampling feature map and Head4 corresponding to the 16-fold down-sampling feature map; or Head3 corresponding to the 8-fold down-sampling feature map, Head4 corresponding to the 16-fold down-sampling feature map or Head5 corresponding to the 32-fold down-sampling feature map.

[0076] The training of the network structure of this embodiment:

[0077] The dataset used for experiments and validation of the proposed method is VisDrone2019, a challenging drone aerial photography benchmark dataset consisting of 8,629 drone aerial images containing 10 types of remote sensing objects. The dataset is divided into three subsets: the training set (VisDrone2019-DET-train), the validation set (VisDrone2019-DET-val), and the test set (VisDrone2019-DET-test-dev). The training set: validation set: test set = 6,471 images: 548 images: 1,610 images. During development, the proposed method was trained using the VisDrone2019-DET-train set and validated using the validation set (VisDrone2019-DET-val). Finally, the proposed method was compared on the test set with the most advanced lightweight object detectors from the past two years.

[0078] Training setup: The proposed model was trained from scratch on four NVIDIA RTX 3090 GPUs. The classification loss function was the binary cross-entropy loss, and the regression loss function was the complete intersection-over-union loss (CIoU loss) and the distribution focal loss (DFL). The optimizer used for training was stochastic gradient descent (SGD), with a momentum of 0.9 and a weight decay of 5e-4. A cosine annealing learning rate schedule was used for 300 epochs, with the first three epochs being warm-up training. The initial learning rate was 0.02 and the batch size was 32.

[0079] Training process: The method of the present invention introduces a strong data augmentation method into the training process of the lightweight model. Specifically, the standard 300-round training process is divided into four stages, each stage is 75 rounds, and the intensity of data augmentation gradually weakens during the training process from the first stage to the fourth stage. Specifically, in the first stage of training, in addition to the four conventional data augmentation methods of translation, horizontal flipping, scaling and HSV hue transformation, the three strong data augmentation methods of MixUp, Mosaic and CopyPaste are also used. Starting from the second stage, one strong data augmentation method is removed in each stage in the order of MixUp, Mosaic and CopyPaste. By the fourth training stage, only the above four conventional data augmentation methods are executed on the training data. The optimal model is finally trained. This method of removing strong data augmentation stage by stage not only makes full use of the diversified representations generated by strong data augmentation, but also avoids the adverse effects of data offset caused by strong data augmentation on lightweight model learning.

[0080] Experimental conditions for this example: The training platform for this example is an NVIDIA GeForce RTX 3090, an Ubuntu 18.04 operating system, and a PyTorch 2.0.0 framework. The speed test platform is an NVIDIA GeForce RTX 2080Ti, an Ubuntu 18.04 operating system, and a PyTorch 2.0.1 framework.

[0081] Experimental data: The VisDrone2019 dataset is used, with a training set: validation set: test set ratio of 6471:548:1610.

[0082] The detection method steps of this embodiment are as follows:

[0083] Backbone network data processing process: The size of the input image is B×3×H×W, where B represents the batch size, 3 represents the number of channels, and H and W represent the height and width of the image respectively;

[0084] The image first passes through the first 4x downsampling module, the spatial resolution is reduced to 4 times the original, the number of channels becomes C1, and the feature size is B×C1×H / 4×W / 4. Figure 1 b;

[0085] feature Figure 1 b After the inverted part of the expanded residual module × 3, the shallow fine-grained features are extracted to obtain the shallow fine-grained features Figure 1 c. The dimensions of the feature map in this step remain unchanged;

[0086] feature Figure 1 After the first 2x downsampling module, the spatial resolution is reduced by 2 times, and the number of channels becomes C2, resulting in a feature with a shape of B×C2×H / 8×W / 8. Figure 1 d;

[0087] feature Figure 1 d is expanded through the residual module after the first inversion part to obtain the feature Figure 1 e. The dimensions of the feature map in this step remain unchanged;

[0088] feature Figure 1 After the second 2x downsampling module, the spatial resolution is reduced by 2 times, and the number of channels becomes C3, resulting in a feature with a shape of B×C3×H / 16×W / 16. Figure 1 f;

[0089] feature Figure 1 fAfter the second inverted part of the extended residual module, the deep high-level semantic features of 16 times downsampling are extracted Figure 1 g; the dimensions of the feature map in this step remain unchanged;

[0090] Shallow fine-grained features Figure 1 c and deep high-level semantic features Figure 1 g is sent to the neck data processing for further feature fusion and interaction;

[0091] Neck data processing process:

[0092] Deep high-level semantic features Figure 1 g is processed through the fast spatial pyramid pooling module to perform maximum pooling operations of different scales on the feature map to obtain the pooled features. Figure 2 a, which captures semantic information under different receptive field sizes;

[0093] Features after pooling Figure 2 a is interpolated through a 4-fold upsampling module, and the spatial resolution H and W are magnified by 4 times to obtain the feature Figure 2 b, Features Figure 2 The shape of b becomes B×C3×H / 4×W / 4;

[0094] feature Figure 2 b and shallow fine-grained features Figure 1 c is concatenated according to the channel dimension to obtain a merged feature with a dimension of B×(C1+C3)×H / 4×W / 4 Figure 2 c;

[0095] Merge Features Figure 2 c After the third inverted part of the extended residual module, the deep and shallow features are further interactively fused, and the channel dimension of the feature map is adjusted to C1, and the feature map with the dimension of B×C1×H / 4×W / 4 is obtained. Figure 2 d;

[0096] feature Figure 2 d After the second 4-fold downsampling module, the spatial resolution H, W is reduced by 4 times to obtain the feature Figure 2 e, Features Figure 2 The shape of e is B×C1×H / 16×W / 16;

[0097] feature Figure 2 e and features after pooling module Figure 2 a is concatenated according to the channel dimension to obtain a merged feature with a dimension of B×(C1+C3)×H / 16×W / 16 Figure 2 f;

[0098] Merge Features Figure 2 After the fourth inverted part of the extended residual module, the features are further interactively fused, and the channel dimension of the feature map is adjusted to C3, resulting in a feature with a dimension of B×C3×H / 16×W / 16. Figure 2 g;

[0099] The features Figure 2 g and characteristics Figure 2 d is sent to the detection head to decode the feature map and generate the final detection result;

[0100] Detection head data processing process:

[0101] feature Figure 2 d is used as the input of the detection head 2, and after two parallel convolution branches, the output is the detection result 3a containing the bounding box and category of the detected object;

[0102] feature Figure 2 g is used as the input of the detection head 4, and after two parallel convolution branches, the output is the detection result 3b containing the bounding box and category of the detected target; the convolution branch is a 3×3 convolution module → 1x1 convolution.

[0103] A non-maximum suppression operation is performed on detection results 3a and 3b to remove a large number of redundant and low-quality prediction boxes, and a set of final detection results with high confidence and non-overlapping is obtained.

[0104] In order to prove the effectiveness of the method proposed in this application, we compared it with the most advanced lightweight target detection algorithm in the past two years. The results are shown in Table 1. The corresponding visual comparison line chart is referenced. Figure 10 and Figure 11 .

[0105] Table 1 Comparison between the proposed method and the SOTA algorithms in the past two years

[0106]

[0107]

[0108] In the above table, the five most advanced methods in the past two years are selected: YOLOv8, YOLOv7, YOLOv6-0.3.0, HIC-YOLOv5, and Gold-YOLO.

[0109] Among them, Year indicates the year when the algorithm was proposed, and Size indicates the scaled size of the long side of the image. As can be seen from Table 1, compared with the comparison method, the present invention achieves a better balance between mean average precision AP, real-time FPS, and model parameter amount Params. Compared with YOLOv8-N, the parameters of CS-Det-N are reduced by 70% (only 0.9M), and the inference speed is increased by 45% (up to 500FPS), but there is still an improvement of 1.1% in AP on the VisDrone2019 dataset. Compared with YOLOv6-0.3.0-N / Gold-YOLO-N, CS-Det-N provides competitive detection accuracy, but with lower latency and fewer parameters. Compared with YOLOv7-Tiny, although CS-Det-N has slightly lower accuracy, it is 167FPS faster and has only 15% of its parameters. Furthermore, for larger models, CS-Det-S reduces parameter count and inference latency compared to YOLOv8-S, while still achieving a 2.5% improvement in AP. Compared to YOLOv6-0.3.0-S / Gold-YOLO-S, CS-Det-S leads in AP by 2.2%, while also being faster and more parameter-efficient. This achieves a state-of-the-art balance between detection accuracy, speed, and parameter count.

[0110] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

[0111] The network structure and lightweight method for real-time target detection from the perspective of a local security drone proposed in this paper have broad application value, especially showing obvious advantages in the following three key areas of local security:

[0112] 1. Cross-domain reconnaissance. In cross-domain reconnaissance missions, drones must quickly and accurately identify diverse targets, such as enemy positions, mobile equipment, and personnel activities, in complex geographical environments and weather conditions, providing real-time intelligence support for tactical decision-making. While traditional target detection models demonstrate high recognition accuracy in laboratory settings, their large number of model parameters and high computational complexity mean that directly porting them to drone platforms will significantly reduce inference speed, making it difficult to meet the stringent real-time and low-latency requirements of remote control or autonomous flight modes.

[0113] 2. Intelligent search and rescue. In emergency situations such as disaster relief and field search and rescue, real-time target detection systems carried by drones are crucial for quickly locating missing persons, identifying signs of life, and assessing the conditions in disaster areas. However, such scenarios are often accompanied by challenges such as extreme lighting conditions, complex terrain, and dense obstructions, which place extremely high demands on the robustness and generalization capabilities of target detection models. Although existing large-scale target detection models can accurately identify targets under ideal conditions, when running on resource-constrained drone platforms, insufficient computing resources may lead to decreased frame rates and increased false detections and missed detections, affecting search and rescue efficiency and effectiveness. Therefore, developing lightweight target detection methods that can ensure recognition accuracy is the key to promoting efficient and precise operations of drones in the field of intelligent search and rescue.

[0114] 3. Disaster early warning. Drones act as "aerial sentinels" in disaster early warning, providing timely and accurate data support for early warning, disaster assessment, and emergency response by real-time monitoring of geological changes, vegetation changes, fire smoke, flood water levels, and other phenomena. However, the disaster scene environment changes rapidly, requiring the target detection system onboard the drone to have the ability to respond quickly and track continuously. Currently, some complex target detection models have an excessively heavy computational burden when processing large-scale, high-resolution remote sensing imagery. This not only affects detection speed, but can also cause the drone to be overloaded and the battery to drain too quickly, limiting its ability to continuously observe in disaster early warning. Therefore, the development of lightweight target detection methods suitable for drone platforms can effectively reduce computational pressure, improve processing speed, and ensure that drones can achieve long-term, high-frequency real-time monitoring and target identification in complex disaster environments.

Claims

1. A lightweight network structure for real-time target detection from the perspective of a nearby security drone, characterized by It includes a backbone network, a neck and a detection head; specifically, it includes a downsampling module and an upsampling module, an inverted part extended residual module and a fast spatial pyramid pooling module and a detection head module; the backbone network is: the image input end is the first 4-fold downsampling module, the inverted part extended residual module × 3, the first 2-fold downsampling module, the first inverted part extended residual module, the second 2-fold downsampling module and the second inverted part extended residual module in series; the neck is: the output of the second inverted part extended residual module is connected to the fast spatial pyramid pooling module, and the output of one path is connected to the output of the inverted part extended residual module × 3 after the 4-fold upsampling module according to the channel dimension The two outputs of the third inverted partially expanded residual module are spliced ​​together in the channel dimension, and then sent to the Head2 detection head for decoding after passing through the third inverted partially expanded residual module, and finally the detected content is output; the other output of the third inverted partially expanded residual module is spliced ​​with the other output of the fast spatial pyramid pooling module according to the channel dimension after passing through the second 4x downsampling module, and then sent to the Head4 detection head for decoding after passing through the fourth inverted partially expanded residual module, and finally the detected content is output; the Head2 detection head and the Head4 detection head are detection head modules, and the merging of their output contents after the non-maximum suppression operation is a set of high-confidence and non-overlapping final detection results.

2. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The 4x downsampling module is a series structure of two 2x downsampling modules.

3. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The 2x downsampling module is a series of 3×3 convolution, batch normalization layer and SiLU activation layer, where the stride of the 3×3 convolution is equal to 2.

4. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The inverted partial expansion residual module is a series of 1×1 convolution, 3×3 partial expansion convolution and 1×1 convolution with residual connection, wherein the 3×3 partial expansion convolution is a part split by the channel dimension, which is subjected to 1×1 convolution, 3×3 grouped convolution and 1×1 convolution, and then merged with the other part split by the channel dimension according to the channel dimension.

5. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The inverted partially expanded residual module ×3 is composed of three inverted partially expanded residual modules stacked in series.

6. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The detection head module is Head2 corresponding to the 4-fold down-sampling feature map and Head4 corresponding to the 16-fold down-sampling feature map; or Head3 corresponding to the 8-fold down-sampling feature map, Head4 corresponding to the 16-fold down-sampling feature map or Head5 corresponding to the 32-fold down-sampling feature map.

7. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: When training the network structure, the classification loss function is binary cross entropy loss, and the regression loss function is complete intersection-over-union loss CIoU loss and distribution focal loss Distribution Focal Loss; the optimizer used in training is stochastic gradient descent SGD, with a momentum of 0.9 and a weight decay of 5e-4; cosine annealing learning rate scheduling is used, and a total of 300 rounds of training are used, with the first 3 rounds being warm-up training, an initial learning rate of 0.02, and a batch size of 32.

8. The lightweight network structure for real-time target detection from the perspective of a nearby security drone according to claim 1 is characterized by: The network structure training process is as follows: the standard multi-round training process is divided into four stages, and the intensity of data augmentation gradually decreases during the training process from the first stage to the fourth stage. Specifically, in the first stage of training, in addition to using the four conventional data augmentation methods of translation, horizontal flipping, scaling, and HSV hue conversion, three strong data augmentation methods of MixUp, Mosaic, and CopyPaste are also used. Starting from the second stage, one strong data augmentation method is removed in each stage in the order of MixUp, Mosaic, and CopyPaste. In the fourth training stage, only the above four conventional data augmentation methods are performed on the training data, and the optimal model is finally trained.

9. A lightweight method for real-time target detection from the perspective of a nearby security drone, characterized by The network structure of any one of claims 1 to 8 is adopted, including a backbone network extracting features → performing feature interaction on the neck → a detection head predicting the bounding box and category of the target in the image based on the neck features, and the method steps are as follows: Backbone network data processing process: The size of the input image is B×3×H×W, where B represents the batch size, 3 represents the number of channels, and H and W represent the height and width of the image respectively; The image first passes through the first 4x downsampling module, reducing the spatial resolution to 4 times the original, and the number of channels becomes C1, resulting in a feature map 1b with a size of B×C1×H / 4×W / 4; Feature map 1b is subjected to the inverted part expansion residual module ×3 to extract shallow fine-grained features, obtaining shallow fine-grained feature map 1c. In this step, the dimensions of the feature map remain unchanged. After the first 2x downsampling module, the spatial resolution of feature image 1c is reduced by 2 times, and the number of channels becomes C2, resulting in feature image 1d with a shape of B×C2×H / 8×W / 8; Feature map 1d is expanded through the first inverted residual module to obtain feature map 1e. In this step, the dimensions of the feature map remain unchanged. After the second 2x downsampling module, the spatial resolution of feature image 1e is reduced by 2 times, and the number of channels becomes C3, resulting in feature image 1f with a shape of B×C3×H / 16×W / 16; The feature map 1f is subjected to the second inverted partial expansion residual module to extract a 16-fold downsampled deep high-level semantic feature map 1g; the dimensions of the feature map remain unchanged in this step; The shallow fine-grained feature map 1c and the deep high-level semantic feature map 1g are sent to the neck data processing for further feature fusion and interaction; Neck data processing process: The deep high-level semantic feature map 1g is passed through the fast spatial pyramid pooling module, which performs maximum pooling operations at different scales on the feature map to obtain the pooled feature map 2a, which captures the semantic information under different receptive field sizes; The pooled feature map 2a is interpolated by a 4x upsampling module, and the spatial resolution H and W are magnified by 4 times to obtain the feature map 2b. The shape of the feature map 2b becomes B×C3×H / 4×W / 4; Feature map 2b is then concatenated with the shallow fine-grained feature map 1c according to the channel dimension to obtain a merged feature map 2c with dimensions of B×(C1+C3)×H / 4×W / 4; The merged feature map 2c is further interactively fused with the deep and shallow features through the third inverted partial expansion residual module, and the channel dimension of the feature map is adjusted to C1, resulting in a feature map 2d with a dimension of B×C1×H / 4×W / 4. Feature image 2d undergoes the second 4x downsampling module, reducing the spatial resolution H and W by 4 times to obtain feature image 2e. The shape of feature image 2e is B×C1×H / 16×W / 16; Feature map 2e is concatenated with feature map 2a after the pooling module according to the channel dimension to obtain a merged feature map 2f with dimensions of B×(C1+C3)×H / 16×W / 16; The merged feature map 2f is further interactively fused after the fourth inverted partial expansion residual module, and the channel dimension of the feature map is adjusted to C3, resulting in a feature map 2g with a dimension of B×C3×H / 16×W / 16; Send feature map 2g and feature map 2d to the detection head, decode the feature map, and generate the final detection result; Detection head data processing process: The feature map 2d is used as the input of the detection head 2. After two parallel convolution branches, the output is the detection result 3a containing the bounding box and category of the detected object. The feature map 2g is used as the input of the detection head 4. After passing through two parallel convolution branches, the output is the detection result 3b containing the bounding box and category of the detected object. The convolution branch is a 3×3 convolution module → 1x1 convolution. A non-maximum suppression operation is performed on detection results 3a and 3b to remove a large number of redundant and low-quality prediction boxes, and a set of final detection results with high confidence and non-overlapping is obtained.

10. A computer program product, characterized in that The invention comprises computer executable instructions, which are used to implement the method of claim 9 when the instructions are executed.

Images