A small target detection lightweight algorithm based on RK3588

Abstract

The present application relates to the cross technical field of computer vision and embedded system, and is aimed at optimizing and improving the precision and real-time balance problem of aerial image small target detection of RK3588 embedded platform. A four-detection-head structure with a newly-added 4-fold downsampling small target detection head is adopted to enhance the small target feature capturing capability. Model lightening is realized by reconstructing the network through a Ghost module, replacing the ReLU activation function and channelizing pruning, so as to reduce the parameter quantity and calculation quantity. A hybrid quantization technology and RKNN thread pool asynchronous reasoning scheme are adopted to adapt to the RK3588 hardware characteristics and improve the NPU utilization rate. While ensuring the detection precision, the present application realizes model lightening and reasoning acceleration, and is suitable for embedded scenes such as unmanned aerial vehicle monitoring, intelligent transportation, urban security and other real-time target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision and embedded systems, specifically a lightweight technology solution for small target detection in aerial images adapted to the RK3588 embedded platform. Through a four-detection-head structure, model compression, and hardware perception deployment, it achieves high-precision, low-latency real-time detection and can be widely applied to embedded scenarios requiring high-precision real-time detection of small targets, such as UAV monitoring, intelligent transportation, and urban security. Background Technology

[0002] With the widespread adoption of embedded vision applications such as drone surveillance and intelligent transportation, the detection of small targets (such as pedestrians, vehicles, and obstacles) in aerial images has become a key challenge. While current mainstream target detection algorithms (such as the YOLOv5 series) perform well in general scenarios, achieving high-precision, real-time small target detection on embedded platforms still faces the following technical bottlenecks: 1. Insufficient ability to capture small target features Existing algorithms such as YOLOv5 typically employ multi-scale detection heads with downsampling of 8x, 16x, and 32x. However, as the downsampling factor increases, the detailed features of small targets are severely lost in deep networks, resulting in a high false negative rate and poor localization accuracy for small targets. In particular, small targets have a low pixel ratio and are densely distributed in aerial images, making it difficult for existing three-detection-head structures to effectively extract their shallow high-resolution features.

[0003] 2. It is difficult to balance lightweight model design with maintaining accuracy. To adapt to the limited computing power and memory of embedded platforms, lightweight networks (such as GhostNet), channel pruning, and activation function simplification (such as ReLU replacing SiLU) are often used. However, direct replacement or pruning can easily lead to a decrease in feature representation ability, especially in small object detection tasks, where lightweight operations often exacerbate feature loss and cause a significant decrease in accuracy.

[0004] 3. Quantitative deployment results in significant loss of accuracy. Embedded platforms often use INT8 quantization to accelerate inference, but traditional global unified quantization strategies do not consider the differences in precision sensitivity among different layers. Upsampling and concatenation operations in small target detection heads and feature fusion paths are extremely sensitive to numerical precision; direct low-bit quantization can easily cause feature distortion and a sharp drop in detection performance.

[0005] 4. Low hardware utilization of multi-core NPU Although embedded chips such as RK3588 have multi-core NPUs (such as NPU_0 / 1 / 2), existing deployment schemes often use single-instance serial inference, which fails to effectively schedule multi-core resources in parallel. This results in the hardware computing potential not being fully utilized, making it difficult to meet the real-time processing requirements of high frame rate video streams.

[0006] Therefore, there is an urgent need for a lightweight algorithm that can achieve high-precision and high-efficiency small target detection on embedded platforms to solve the above-mentioned technical challenges.

[0007] This invention conducts in-depth research and optimization design on the application of the YOLOv5 algorithm on embedded platforms. First, addressing the challenge of small target detection in aerial images, a small target detection head is added, effectively improving the model's performance in small target detection scenarios. Second, to meet the high requirements of embedded systems for storage space and computational efficiency, the YOLOv5 model is lightweighted, including introducing a Ghost module to optimize the network structure, replacing the activation function with ReLU to improve inference speed, and reducing model parameters and computational load through channelization pruning. Finally, considering the hardware characteristics of the RK3588 embedded platform, hybrid quantization technology is used to improve computational efficiency, and asynchronous inference technology using thread pools is combined to improve NPU utilization and inference speed. Summary of the Invention

[0008] This invention addresses the shortcomings of YOLO v5 in small target detection in aerial images captured on the RK3588 platform. It proposes a lightweight technical solution from two aspects: algorithm optimization and hardware adaptation, achieving a balance between accuracy, speed, and resource consumption. The specific steps are as follows: Step 1: Optimize the YOLOv5 detection head structure to enhance small target detection capabilities. To address the issue of sparse features for small targets in aerial images, a lightweight four-detector YOLOv5 network model is constructed. Based on the original three-detector YOLOv5 model, a new small target detector based on 4x downsampling is added to form a four-detector structure, and feature fusion is performed through FPN and PAN.

[0009] The original YOLOv5 network employs a multi-scale object detection strategy. Specifically, if the input image size is 640×640×3, after downsampling by 8x, 16x, and 32x, feature maps of 80×80×255, 40×40×255, and 20×20×255 are obtained, respectively. The detection process is performed on these three feature maps at different resolutions. Among these feature maps, the 8x downsampled feature map has the smallest local receptive field. However, for some smaller targets, the receptive field of the 8x downsampled feature map is still relatively large, causing the network to lose details and location information of these small targets, resulting in poor detection performance for small targets.

[0010] Therefore, to address this issue, this invention optimizes the original YOLOv5 algorithm by adding a small target detection head with 4x downsampling, forming a four-detection-head structure. This branch is connected to the detection head after feature fusion through FPN (Feature Pyramid Network) and PAN (Path Aggregation Network), thereby enhancing the model's ability to recognize small targets in aerial images and improving overall detection accuracy. Figure 1 As shown, the area within the dotted line box represents the network structure of the detection head specifically designed for small targets, which is added in this invention.

[0011] Because the designed small target detection head is based on 4x downsampling, it has higher resolution and a smaller receptive field. Therefore, this large-scale feature map can be applied to the detection of small targets. It can capture more details and small-scale features, and through interaction and fusion with lower-level features, it effectively combines low-level positional information with deep semantic information, thereby improving the ability to recognize small targets.

[0012] Step 2: Lightweight Design of YOLOv5 Network The YOLOv5 network model is lightened by replacing the C3_1 module in the backbone network with the C3_Ghost module and replacing the SiLU activation function with the ReLU activation function.

[0013] (1) Lightweight YOLOv5 network structure The backbone network is the feature extraction part of the YOLOv5 model, typically responsible for extracting deep feature information from the input image. The original YOLOv5 used the deep convolutional network CSPDarknet53 as its backbone network, which offers high accuracy but is computationally expensive. Compared to other lightweight networks, the GhostNet algorithm significantly reduces the model size while maintaining computational efficiency and accuracy without significantly impacting overall model performance.

[0014] Therefore, this invention replaces some convolutional layers in the YOLOv5 backbone network with a Ghost structure, reducing computational cost and parameter count: such as Figure 2 As shown, the Ghost module achieves feature extraction through two processes: generating core feature maps through a small number of convolutions and generating virtual features through simple linear transformations. The final features are obtained by concatenating the two maps.

[0015] According to the practical application of the present invention, simply replacing the backbone network to lightweight the model would result in a significant loss of detection accuracy, which does not meet the requirements of practical use. Therefore, based on the lightweight concept of the GhostNet network, the C3_1 module of YOLOv5 was reconstructed by introducing the Ghost Bottleneck module. For example... Figure 3 As shown, when the stride is 1, Ghost Bottleneck consists of two Ghost modules connected in series, which, together with residual connections, maintain the feature representation capability. When the stride is 2, depthwise separable convolutional downsampling is added to balance feature extraction and computational efficiency. Specifically, in the reconstruction process, the core structure of the C3_1 module is first retained, including the feature branches and fusion methods. Then, by introducing the Ghost Bottleneck module, the efficient feature map generation feature map generation feature of the Ghost Bottleneck module is utilized to further reduce redundant computation. Figure 4 The C3_Ghost module has a structure that effectively reduces the computational load in the feature extraction stage and retains the feature representation capability through a more efficient feature generation mechanism. This avoids a significant performance drop caused by simply replacing the backbone network and greatly reduces computational overhead, making YOLOv5 more valuable for practical applications on embedded platforms.

[0016] This invention provides a lightweight improvement to the YOLOv5 backbone network by replacing some convolutional operations and the C3 structure in the backbone network, while leaving the convolutional operations of the neck and head networks unchanged. Its structural diagram is shown below. Figure 5 As shown.

[0017] GhostNet's Ghost module significantly reduces computational complexity by generating inexpensive features, but this design introduces information redundancy and feature sparsity in deep networks, affecting feature representation capabilities. This is particularly detrimental to tasks requiring multi-scale feature extraction. Replacing the C3_1 module with C3_Ghost effectively reduces the computational burden on shallow network layers while avoiding changes to the C3_2 and C3_3 modules, ensuring that feature extraction capabilities in deep networks are not compromised by additional feature degradation issues. Since C3_1 is located at the initial part of the network and is mainly responsible for shallow feature extraction, replacing it with C3_Ghost significantly reduces computational costs while preserving rich detail information while maintaining high efficiency. This optimization is especially crucial for object detection tasks, as shallow features play a vital role in capturing small targets.

[0018] (2) Activation function optimization In deep learning models, activation functions are key components that introduce nonlinear factors, enabling neural networks to fit various complex curves and learn to approximate nonlinear functions. This nonlinear capability gives neural networks powerful expressive and learning abilities and is one of the core mechanisms of deep learning models.

[0019] (1) The activation function used in the YOLOv5 algorithm is SiLU, and the calculation formula for SiLU is shown in Equation 1. The characteristic of the SiLU function is the use of the sigmoid function, which makes the network output range between 0 and 1. This smooth and non-linear characteristic allows for adaptive scaling of input values, resulting in superior performance on certain tasks. However, due to the introduction of the sigmoid function, the calculation process of SiLU is relatively complex and computationally intensive.

[0020] (2) The ReLU activation function has an extremely simple calculation form, as shown in Equation 2. ReLU only requires one maximum value determination, avoiding complex calculations and significantly reducing computational cost, thus decreasing the overhead of forward and backward propagation in neural networks. Simultaneously, ReLU effectively alleviates the vanishing gradient problem, ensuring gradient constancy in deep networks and improving training efficiency. Considering computational efficiency and inference speed, this invention replaces the SiLU activation function in the YOLOv5 model with the ReLU activation function.

[0021] Step 3: Channel pruning and fine-tuning The lightweight modified model is trained, and a channel pruning method based on the scaling factor of the BN layer is used to compress the model. After pruning, the model is fine-tuned to restore detection accuracy. This invention performs channel pruning based on the BN layer. The computational cost of the YOLOv5 model is mainly in the CBS module, which consists of convolutional layers, BN layers, and activation functions. The BN layer normalizes the features of each channel, and the standard calculation formula for the BN layer is shown in Equation 3: (3) In the formula, where and These are the sample mean and variance, respectively. It is a very small constant. It is the scaling factor of the BN layer. X and Y are the mean parameters in the batch normalization layer, respectively, and X and Y are the input and output of the BN layer. Scaling factor. The absolute value reflects the importance of the channel; the closer the scaling factor is to 0, the lower the contribution of the channel to the model performance. The distribution of scaling factors is often quite dispersed. L1 regularization is introduced to constrain the scaling factor. The specific calculation method is shown in Equation 4. Through sparse training, the scaling factors of redundant channels are made to approach 0.

[0022] (4) In the formula, Indicates the training input, This represents the network's predicted output value. This represents the trainable weights. The first summation term is the error loss between the network's predicted output value and the true values ​​of the training sample labels. It is a sparsity penalty on the scaling factor. It is a sparse penalty factor.

[0023] The entire channel pruning process is as follows Figure 6 As shown: The model is pre-trained to obtain a pre-trained model with good target detection performance before pruning; sparsity training is performed using L1 regularization; a threshold α is set to prune channels with an absolute value of scaling factor lower than α, with a pruning ratio of 40%; after pruning, the model is fine-tuned with a small learning rate to recover the accuracy loss.

[0024] Step 4: Hybrid Quantization Conversion The fine-tuned model was converted to RKNN format and mixed precision quantization was performed. The backbone network used INT8 quantization, while the key operators in the newly added small target detection head and FPN / PAN feature fusion path retained FP16 or FP32 floating-point precision.

[0025] Quantization is a key technique in deep learning model compression. By converting floating-point weights and activation values ​​in the model into lower-bit integers, it significantly reduces the model's storage space and improves the computational efficiency of hardware accelerators. Quantization is divided into symmetric quantization and asymmetric quantization based on the definition of zeros. Symmetric quantization has a fixed zero of 0, suitable for simple scenarios or symmetrical data distributions. Asymmetric quantization has arbitrary integer zeros, which can be dynamically adjusted flexibly, suitable for most asymmetric distributions, and can significantly improve the performance of the quantized model.

[0026] On hardware accelerators, low-bit integer operations are generally far more fast and power-efficient than floating-point operations. However, during quantization, blindly reducing the number of bits in all weights or activation values ​​can introduce quantization errors, leading to a significant decrease in model accuracy.

[0027] To address this issue, this invention employs hybrid quantization technology. Unlike traditional single-precision quantization, hybrid quantization uses different quantization bit depths based on the precision and computational performance requirements of different parts of the model. For example, low-bit integers (INT8) are used for computationally intensive parts that have a smaller impact on precision, while higher precision (such as FP16 or FP32) is retained for parts with higher precision requirements. In this way, hybrid quantization not only retains the performance advantages of traditional quantization but also significantly reduces the quantization error of the model, thereby achieving a better balance between inference speed, memory usage, and model accuracy.

[0028] It should be noted that during the mixed precision quantization process, some network layers are highly sensitive to numerical precision. If forcibly converted to INT8 format, the detection accuracy of small targets will decrease significantly. These layers are defined as "precision-sensitive layers", mainly including: (1) the convolutional layer in the newly added 4x downsampled small target detection head, which plays a decisive role in the localization and classification of small targets because it directly processes high-resolution shallow features; (2) the upsample layer in the FPN / PAN feature fusion path, whose interpolation operation will amplify the rounding error introduced by low precision; (3) the feature concatenation layer in FPN / PAN, whose multi-scale feature fusion process requires high numerical consistency of input features, and low precision representation is prone to distribution mismatch. Therefore, in this invention, the above precision-sensitive layers retain FP16 or FP32 floating-point precision, thereby achieving a real-time inference speed of 60 FPS and an AP50 detection accuracy of 37.6% on the RK3588 platform.

[0029] This invention utilizes the RKNN-Toolkit2 tool to optimize the model through quantization, balancing accuracy and memory usage. First, a quantization accuracy analysis is performed on the model. The results of this analysis visually identify network layers that suffer significant accuracy loss after hybrid quantization. For these network layers, hybrid quantization is then re-performed by modifying the quantization configuration file to optimize model performance.

[0030] The hybrid quantization process of RKNN-Toolkit2 is as follows: Figure 7As shown, the hybrid quantization function is implemented in two stages: the first stage calls the interface `hybrid_quantization_step1`, which generates a temporary model file, configuration file, and quantization configuration file based on the loaded original model; the second stage calls the interface `hybrid_quantization_step2`, which generates the hybrid quantized file based on the temporary model file, configuration file, quantization configuration file, and calibration dataset generated in the first stage. It is important to note that since the configuration file generated in the first stage is an initial version, the directly generated hybrid quantized model will have significant quantization errors. Therefore, accuracy analysis of the hybrid quantized model is necessary. By comparing the performance differences between the full-precision model and the quantized model, the errors introduced by the quantization process can be effectively evaluated, thereby optimizing the model's accuracy and performance.

[0031] Step 5: Multi-core asynchronous inference deployment The RKNN model is deployed on the RK3588 platform, and asynchronous inference is performed through a preset RKNN thread pool. The RKNN thread pool creates an RKNN model instance for each of the three computing cores NPU_0, NPU_1 and NPU_2, and uses a round-robin scheduling mechanism to allocate video stream frames to each instance for parallel inference.

[0032] The RK3588 platform is equipped with a multi-core processor, including multiple CPU cores and a powerful NPU core, providing robust computing capabilities. To more effectively utilize these hardware resources, the RK3588 platform offers a triple-core concurrent mode, such as... Figure 8 As shown. In the three-core concurrent mode, the RKNN model can use all three cores for inference, but the NPU utilization remains low, even leading to memory leaks or out-of-bounds errors during RKNN model inference, causing system crashes. Therefore, a simple and effective method for creating an RKNN thread pool is proposed, such as... Figure 9 As shown, this is to support parallel inference of the RKNN model, thereby improving inference efficiency.

[0033] The core of RKNN thread pool technology lies in combining NPU cores with model inference tasks through threads, leveraging the parallel computing capabilities of multi-core NPUs to achieve efficient model inference. The RKNN thread pool manages an equal number of RKNN models as the number of threads. These RKNN models are allocated to different NPU cores (such as NPU_0, NPU_1, and NPU_2) for computation according to a polling mechanism, thus achieving concurrent processing of inference tasks. Compared to simple multi-core utilization, RKNN thread pool technology not only significantly improves the utilization of NPU cores but also dynamically allocates tasks, reducing resource contention and increasing throughput, thereby better meeting the demands of real-time video frame processing in high-load scenarios. This design is highly scalable, adapting to different scenarios by adjusting the number of threads, and maximizing performance by fully utilizing hardware resources.

[0034] The specific implementation process of the RKNN thread pool is as follows: (1) Initialize the RKNN thread pool, and complete the initialization of parameters such as the number of threads, RKNN model, queue, and counter; (2) Creation of RKNN model instances and allocation of NPU cores. First, create RKNN model instances with the same number of threads; then, according to the sequence number of the created RKNN model instances, allocate them to different cores of NPU_0, NPU_1 and NPU_2 for inference tasks in sequence; (3) Obtain the inference results of different NPU cores and obtain the RKNN model inference results from the queue; (4) Release resources. If the queue is empty, release the RKNN model created by the thread pool and close the thread pool; (5) If the queue obtained is not empty, continue to step (3) to perform model reasoning until the reasoning result is empty; (6) Shut down the thread pool.

[0035] The deployment of algorithms based on the RK3588 platform mainly relies on the RKNN-Toolkit2 toolchain provided by Rockchip. Since the RKNN environment does not support direct conversion of PyTorch to RKNN models, it is necessary to first convert the PyTorch model to an ONNX model, and then convert the ONNX model to an RKNN inference model file.

[0036] First, initialize the PyTorch environment to ensure the availability of PyTorch's functions. Second, load the trained PyTorch model, providing the raw model data. Next, define the tensor shapes and data types for the model's input and output; this step is fundamental to defining the model's inference interface. Then, use PyTorch's `torch.onnx.export` interface to convert the PyTorch model to ONNX format. Finally, export the ONNX model, completing the model conversion from PyTorch to ONNX.

[0037] Then, initialize the RKNN environment, ensuring that the interfaces provided by the RKNN-Toolkit2 toolkit are usable; next, load the ONNX model and provide the model data; then, set the parameters for building the RKNN model according to the provided model data, including the names and shapes of the input and output nodes; subsequently, build the corresponding RKNN model according to the set parameters; finally, export the RKNN model, completing the conversion from the ONNX model to the RKNN model.

[0038] The trained PyTorch model is converted to ONNX format, and then converted to an RK3588-compatible RKNN model using RKNN-Toolkit2. After that, the model can be deployed on the RK3588 embedded platform and tested and evaluated on the VisDrone2019 dataset.

[0039] Beneficial effects This invention achieves three core advantages over the YOLOv5s model through a collaborative design of algorithm optimization and hardware adaptation: 1. Improved small target detection accuracy: The addition of a small target detection head enables this invention to achieve an AP50 of 37.6% on the VisDrone2019 dataset, a 4.2% improvement over the original YOLOv5s, and a 2.8% improvement in AP50-95 accuracy; 2. Lightweight and efficient model: Through the synergistic effect of Ghost module replacement, channel pruning and hybrid quantization, the number of model parameters in this invention is reduced from 14.4MB to 12.2MB, the computational load (FLOPS) is reduced from 16.5G to 14.8G, and the inference speed is improved by 11.1FPS; 3. Preservation of quantization accuracy: The hybrid quantization strategy preserves the accuracy of key layers while accelerating the process, avoiding distortion of small target features; 4. Excellent hardware compatibility: Asynchronous inference via thread pool increases the utilization of RK3588 NPU to over 33%, achieving an inference speed of 60 FPS for 720P video streams and 45 FPS for 1080P, far exceeding the inference speed of the baseline network and meeting the needs of embedded real-time detection. 5. Strong system applicability: Suitable for embedded real-time detection scenarios such as drones and intelligent transportation. Attached Figure Description

[0040] Figure 1 Overall network structure diagram with added small target detection head; Figure 2 Ghost module schematic diagram; Figure 3 Schematic diagram of Ghost Bottleneck structure; Figure 4 C3_Ghost module structure diagram; Figure 5 Schematic diagram of the improved backbone network structure; Figure 6 Model pruning process; Figure 7 Schematic diagram of the mixed quantization process; Figure 8 Traditional three-core operating mode; Figure 9 A schematic diagram of RKNN thread pool technology; Figure 10 Result image of dense scene; Figure 11 Resulting image of the occluded scene; Figure 12 Blurred scene result image; Figure 13 Image showing the target detection results in the video stream. Detailed Implementation

[0041] The present invention will now be further described in conjunction with the embodiments and accompanying drawings, and the specific implementation is as follows: Step 1: Data Preparation Download the VisDrone2019 dataset and divide it into training, validation, and test sets in an 8:1:1 ratio; convert the labeled data to YOLO format and unify the target categories (pedestrians, vehicles, non-motorized vehicles, etc.).

[0042] Step 2: Model Optimization and Training Based on the YOLOv5s baseline network structure, a new small target detection head branch is added. The original detection heads corresponding to 8x, 16x, and 32x downsampling are retained, and a new small target detection branch based on 4x downsampling is added to process high-resolution shallow feature maps. In the Neck part, firstly, the 80×80 feature map is upsampled to a 160×160 feature map. This 160×160 feature map is then element-wise added to a 160×160 feature map of the same size in the Backbone structure to obtain a new 160×160 feature map. Secondly, the new 160×160 feature map is downsampled back to an 80×80 feature map. Finally, the 80×80 feature map is fused by the feature pyramid network and the top-down pyramid network before being passed to the detection head via the C3 module. The YOLOv5 network is improved in a lightweight way by replacing some convolutional operations in the backbone network with Ghost modules, while keeping the convolutional structures of the neck and head networks unchanged; the Ghost Bottleneck module is introduced, the C3 module of YOLOv5 is reconstructed into C3_Ghost, and then the C3_1 module in the backbone network is replaced with C3_Ghost. Change the activation function; replace the original Silu activation function in YOLOv5 with the ReLU activation function; The pre-trained model (100 iterations) is then subjected to sparsity training using L1 regularization (50 iterations), and a threshold α=0.01 is set to remove redundant channels. After pruning, fine-tune the model (50 iterations), gradually reducing the learning rate from 0.001 to 0.0001, and save the optimal weight file.

[0043] Step 3: Hardware Adaptation and Deployment The training weights were converted to ONNX format using PyTorch to verify the consistency between input and output. Load the ONNX model using RKNN-Toolkit2, configure the hybrid quantization parameters (set the accuracy loss threshold to 0.5%), and generate the RKNN model; Write asynchronous inference code for a thread pool, setting the number of threads to 6 (to adapt to the 0-3 cores of the NPU and CPU load). The application is compiled using cross-compilation tools, transferred to the RK3588 platform, and deployed.

[0044] To verify the effectiveness of the lightweight small target detection method based on RK3588 provided by this invention, simulation experiments were also conducted.

[0045] Experimental environment setup The specific experimental environment is shown in Table 1. The workstation was used for training the YOLOv5 object detection algorithm, the PC virtual machine was used for software development and cross-compilation, and the algorithm was finally verified on the RK3588 platform.

[0046] Table 1 Experimental Environment Experimental content This experiment verifies the effectiveness of each optimization module through ablation experiments and verifies the superiority of the overall solution through comparative experiments. The test scenarios include various typical aerial photography scenarios on the VisDrone2019 dataset.

[0047] Evaluation indicators This invention uses Average Precision (AP) to evaluate the accuracy of the detection algorithm, model computation and model parameters to measure model complexity, and frames per second (FPS) to evaluate the detection speed of the model.

[0048] (1) Average accuracy Precision, recall, and average precision can be calculated from the parameters in the confusion matrix. Table 2 shows the confusion matrix and the definitions of its parameters. TP refers to the number of positive samples predicted as positive, FN is the number of positive samples predicted as negative, FP is the number of negative samples predicted as positive, and TN is the number of negative samples predicted as negative.

[0049] Table 2. Definitions of the confusion matrix and its parameters. (5) The precision and recall of the detection results were calculated based on the confusion matrix, as shown in Equation 5. Different precision and recall values ​​were obtained using different confidence thresholds. A precision-recall curve was plotted with recall on the x-axis and precision on the y-axis. The area under the precision-recall curve obtained in the previous step is the average precision. The calculation method is shown in Equation 6. In this paper, AP50 and AP50-95 are introduced when analyzing the precision index, representing the average precision of the model when the IoU threshold is 0.5 and the average precision when the IoU threshold is from IoU=0.5 to IoU=0.95 with a step size of 0.05, respectively.

[0050] (6) (2) Model computational load and model parameter count In addition to accuracy metrics, model computational load and model parameter count also need to be considered. Model computational load refers to the total number of floating-point operations performed during the forward propagation process, usually measured in floating-point operations per second (FLOPS). Model parameter count includes factors such as the number of layers in the neural network, the number of nodes per layer, the connection methods between layers, the number of parameters, and the data types used to store these parameters, and is represented by MB.

[0051] (3) Frame rate per second Frames per second (fps) represents the number of images a model can process per second, and is used to measure the model's detection speed.

[0052] Simulation test results This invention verifies the improved algorithm through simulation on the VisDrone2019 dataset. First, it analyzes the impact of optimizing the detection head on model performance. Then, for the lightweight design of the YOLOv5 model, it analyzes the changes in the algorithm's target detection performance before and after network structure optimization, activation function replacement, model pruning, and hybrid quantization.

[0053] (1) Results of the detection head optimization experiment The experimental results after optimizing the detection head are shown in Table 3. Models with the suffix "-4F" are those optimized with small target detection heads. As can be seen from the table, the YOLOv5s model, after adding the small target detection head, saw an increase of 6.1% in AP50 accuracy, 4.4% in AP50-95 accuracy, an increase of 4.3 GFLOPS in computation, and an increase of 2 MB in model size. The YOLOv5n model, after adding the small target detection head, saw an increase of 5.3% in AP50 accuracy, 4.5% in AP50-95 accuracy, an increase of 1.3 GFLOPS in computation, and an increase of 0.4 MB in model size. The results show that adding a small target detection head effectively improves algorithm accuracy, especially on aerial image datasets. However, adding a small target detection head also leads to an increase in computation and model size, which places higher demands on the computing resources of embedded platforms. Therefore, further lightweighting of the model is needed to achieve a reasonable balance between accuracy improvement and computational resource consumption.

[0054] Table 3 Results of the Detector Head Optimization Experiment (2) Experimental results of network structure optimization Table 4. Experimental Results of Network Structure Optimization The experimental results after network structure optimization are shown in Table 4. YOLOv5s and YOLOv5n were used as baseline models. Models with the suffix "-Ghost" are those improved by introducing the Ghost module. As can be seen from the table, the YOLOv5s-Ghost model showed a 1.6% decrease in AP50 accuracy and a 1.2% decrease in AP50-95 accuracy compared to the YOLOv5s model, with a reduction of 2 GFLOPs in computation and a reduction of 2.2 MB in model size. The YOLOv5n-Ghost model showed a 1.3% decrease in AP50 accuracy and a 1.2% decrease in AP50-95 accuracy compared to the YOLOv5n model, with a reduction of 0.3 GFLOPS in computation and a reduction of 0.5 MB in model size. The experiments demonstrate that replacing some convolutional operations in the YOLOv5 network with the Ghost convolution module slightly reduced model accuracy, but significantly reduced model size and computation. Based on the actual data, the optimized model still maintains good detection performance and achieves the expected results.

[0055] (3) Experimental results of changing the activation function The experimental results after changing the activation function are shown in Table 5. Models with the suffix "-ReLU" are those with the changed activation function, while those without the suffix are the original models. As can be seen from the table, the accuracy of the YOLOv5s and YOLOv5n models decreased only slightly after changing the activation function, while the model size and computational cost remained essentially unchanged. However, the FPS metric shows a significant improvement in inference speed. The YOLOv5s model saw an 11 FPS increase in inference speed after the activation function change, and the YOLOv5n model saw an 8 FPS increase. The experimental results demonstrate that replacing SiLU with ReLU activation function effectively improves the inference speed of network models in practical applications with only a small improvement.

[0056] Table 5. Experimental results of changing the activation function (4) Results of the model pruning experiment The results of the model pruning experiment are shown in Table 6. The model size was compressed by pruning the model by 40%. As can be seen from the table, after pruning the YOLOv5s model by 40%, the AP50 accuracy decreased by 0.8%, the AP50-95 accuracy decreased by 0.5%, the computational cost decreased by 6.7 GFLOPs, and the model size decreased by 4.2 MB. After pruning the YOLOv5n model by 40%, the AP50 accuracy decreased by 0.8%, the AP50-95 accuracy decreased by 0.5%, the computational cost decreased by 1.7 GFLOPs, and the model size decreased by 1.4 MB. The experimental results show that model pruning effectively reduces the computational cost and model size while maintaining accuracy.

[0057] Table 6 Results of the Model Pruning Experiment (5) Results of mixed quantification experiment The results of the hybrid quantization experiment are shown in Table 7. Hybrid quantization is a quantization process performed on the model on the RK3588 platform. The evaluation metric replaces the original computational load with memory usage, which helps to better reflect the results of hybrid quantization. As can be seen from the table, after hybrid quantization, the YOLOv5s model size decreased by 3.8MB, memory usage decreased by 12.5MB, and inference speed increased by 10.2 FPS; the YOLOv5n model, after hybrid quantization, had a smaller model size, lower memory usage, and increased inference speed by 4.9MB, while inference speed increased by 9.6 FPS; the model accuracy remained essentially unchanged after hybrid quantization. The experimental results show that hybrid quantization significantly reduces the resource consumption of the embedded platform while maintaining model accuracy as much as possible, effectively alleviating resource pressure.

[0058] Table 7 Results of the mixed quantization experiment (6) Experimental results of the improved target detection model The experimental results of the improved target detection model are shown in Table 8. The suffix "-RK" indicates that the target detection model integrates the above improvements. As can be seen from the table, compared to the YOLOv5s model, the YOLOv5s-RK model reduces computation by 1.7 GFLOPS, model size by 2.2 MB, AP50 accuracy by 4.2%, AP50-95 accuracy by 2.8%, and inference speed by 11.1 FPS. The YOLOv5n-RK model reduces computation by 0.7 GFLOPS, model size by 0.8 MB, AP50 accuracy by 4.0%, AP50-95 accuracy by 2.8%, and inference speed by 12.0 FPS. The experimental results show that the improved target detection model not only significantly improves accuracy but also reduces computation and model size, which is beneficial for application on the RK3588 platform.

[0059] Table 8 Experimental results of the improved target detection model (7) Experimental results of RKNN model conversion Table 9 shows the results of model conversion of the improved object detection model YOLOv5s-RK. After converting the object detection model from PyTorch to RKNN, the model size decreased by 4.7MB, memory usage decreased by 8.3MB, and accuracy remained unchanged. Experimental results show that the model conversion achieves the conversion from PyTorch to RKNN without loss of accuracy, and effectively reduces the number of model parameters and memory usage.

[0060] Table 9. RKNN Model Conversion Results (8) Experimental results of RKNN thread pool technology Table 10 shows the experimental results of the RKNN thread pool technology. The experiments used CPU and tri-core NPU utilization rates and object detection frame rates on the RK3588 platform as evaluation metrics. The results show that compared to simply using all three cores, the proposed RKNN thread pool technology increases the object detection frame rate from 18 FPS to 60 FPS, significantly improving object detection inference speed. However, when using RKNN, it should be noted that increasing the number of threads is not unlimited. As the number of threads increases, the load on both the NPU and CPU increases, and too many threads can lead to increased resource contention and reduced inference efficiency. Experimental results show that the RKNN thread pool technology can improve resource utilization and effectively increase object detection inference speed when the number of threads is reasonably configured. The object detection inference speed reaches its fastest when the number of threads is 6.

[0061] Table 10 Experimental Results of RKNN Thread Pool Technology The above performance metrics (60 FPS, AP50=37.6%) were measured under the following conditions: RK3588 development board, Ubuntu 20.04 system, no other high-load tasks, and input is a 720P video stream from the VisDrone2019 test suite.

[0062] Visualization Analysis This invention provides a visual analysis and comparison of the detection results of YOLOv5n and YOLOv5s models, as well as the improved algorithms YOLOv5n-RK and YOLOv5s-RK in this chapter, under different scenarios in the VisDrone2019 dataset.

[0063] Image of target detection results in dense scenes Figure 10As shown in Figures (a) and (b), it is clear that the YOLOv5n-RK model has a significantly lower rate of missed detections when detecting densely packed vehicle targets at the top of the image compared to the YOLOv5n model. Similarly, the comparison results in Figures (c) and (d) further confirm this. The YOLOv5s-RK model exhibits higher accuracy in detecting densely packed targets at the top of the image. This indicates that the improved model has significantly enhanced its detection capabilities in dense scenes, which is of great significance for improving the overall performance of aerial image target detection tasks.

[0064] The target detection results in occluded scenes are shown in the figure below. Figure 11 As shown in the figure, the scene includes situations such as pedestrians occluding each other and pedestrians being obscured by utility poles. Comparing Figures (a) and (b), it can be seen that the YOLOv5n-RK algorithm detected the occluded target on the right side of the image, while the YOLOv5n model failed to detect it. For the pedestrian occluded on the left side of the image, the YOLOv5n model missed detection, while the YOLOv5s model detected the target. By comparing Figures (c) and (d), it can be observed that the YOLOv5s-RK algorithm detected the target obscured by the utility pole, while the YOLOv5s model failed to detect it. In addition, the YOLOv5s-RK model performs significantly better than the YOLOv5s model in detecting occluded targets in the center of the image. Experiments show that the improved YOLOv5 target detection model has improved ability to detect occluded targets.

[0065] The target detection results in a blurred scene are shown in the image below. Figure 12 As shown in Figure (a) and Figure (b), it can be seen that the YOLOv5n model misdetects the vehicle on the right side of the image and misses the vehicle obscured by trees at the top of the image, while the YOLOv5n-RK model does not exhibit these issues. For the blurred vehicle in the upper left corner of the image, the first three models all miss the detection, while the YOLOv5s-RK model accurately detects the blurred vehicle. Experiments demonstrate that the improved method proposed in this chapter can enhance the detection accuracy of the YOLOv5 model in blurred scenes.

[0066] This invention converts the YOLOv5s model to the improved YOLOv5s-RK model and deploys it on the RK3588 platform for testing and verification. Object detection experiments are conducted on 720P and 1080P resolution video streams to test whether the platform's frame rate for video detection tasks in practical applications meets real-time requirements.

[0067] like Figure 13The image shows the target detection results of actual aerial video streams. As can be seen from the image, the YOLOv5s-RK model achieves an inference speed of over 60 frames per second for 720P video streams and over 45 frames per second for 1080P video streams, far exceeding the inference speed of the baseline network. Comparing Figures (a) and (b), it can be seen that the YOLOv5s-RK model significantly outperforms YOLOv5s in detecting small, distant targets. Comparing Figures (c) and (d), it can be seen that the YOLOv5s-RK model significantly reduces missed detections compared to the YOLOv5s baseline model. The experimental results fully verify the effectiveness of the optimization strategy of the proposed YOLOv5s-RK model. The YOLOv5s-RK model achieves improved inference speed while maintaining high detection accuracy, providing strong support for embedded real-time target detection applications.

Claims

1. A method for detecting small targets, characterized in that, The method is executed on an embedded AI chip with a multi-core NPU architecture and mixed-precision computing capabilities, and includes the following steps: Step 1: Construct a lightweight four-detector YOLOv5 algorithm for small object detection. The model retains the three detectors corresponding to the original 8x, 16x and 32x downsampling on the basis of the YOLOv5 network, and adds a small object detection head based on 4x downsampling. This small object detection head is used to process the 80x80 small object detection branch obtained by downsampling the 160×160 feature map, and is connected to the detection head after feature fusion through FPN and PAN. Step 2: Perform lightweight modification on the YOLOv5 network model, including replacing the C3_1 module in the YOLOv5 network backbone with the C3_Ghost module, and replacing the SiLU activation function in the network with the ReLU activation function; Step 3: Train the lightweight modified four-detector YOLOv5 network model, and compress the trained model using a channel pruning method based on the BN layer scaling factor. After pruning, fine-tune the model to restore accuracy. Step 4: Convert the fine-tuned model to RKNN format and perform mixed precision quantization. The backbone network uses INT8 quantization, while the key operators defined as "precision-sensitive layers" retain FP16 or FP32 floating-point precision. The precision-sensitive layer refers to a network layer that is highly sensitive to numerical errors during quantization and will cause a significant decrease in the accuracy of small target detection if forcibly converted to INT8 format. Step 5: Deploy the RKNN model on the RK3588 platform and perform asynchronous inference through a preset RKNN thread pool; the RKNN thread pool creates an independent RKNN model instance for each of the three NPU computing cores of RK3588, namely NPU_0, NPU_1 and NPU_2, and uses a round-robin scheduling mechanism to distribute the input video stream frames to each instance for parallel processing in order to improve NPU utilization.

2. The lightweight method for small target detection based on RK3588 as described in claim 1, characterized in that, The channel pruning in step 3 specifically includes: pre-training the model; sparse training by constraining the scaling factor of the BN layer with L1 regularization; setting a threshold α=0.01 to prune channels with a scaling factor absolute value lower than α; and fine-tuning the pruned model with a smaller learning rate.

3. The lightweight method for small target detection based on RK3588 as described in claim 1, characterized in that, The hybrid precision quantization in step 4 specifically includes: using INT8 quantization for the convolutional layers in the backbone network, while retaining FP16 or FP32 floating-point precision for the newly added 4x downsampled small target detection head and the key operators in the FPN / PAN feature fusion path.

4. The method as described in claim 3, characterized in that, The precision-sensitive layer includes convolutional layers in the newly added small target detection head and layers in FPN / PAN used for upsampling and feature stitching.

5. The method as described in claim 1, characterized in that, The lightweight four-detector YOLOv5 network model achieves an inference speed of 60 FPS for 720P resolution video streams on the RK3588 platform, and the AP50 index reaches 37.6%.

Images