A lightweight target detection method for embedded platforms

Abstract

The present application is a kind of lightweight target detection method for embedded platform, which overcomes the problems of large network parameter quantity, slow detection speed and poor precision index in the prior art. The present application not only solves the problems of slow speed and low accuracy of the current detection method in the deployment of embedded devices, but also optimizes the computation graph for specific operators from the hardware level, enabling fast and accurate target detection on limited resource devices. The present application comprises the following steps: step 1: obtaining basic data and making data set; step 2: data enhancement preprocessing; step 3: constructing a benchmark network model and pre-training; step 4: building a lightweight detection network model; step 5: reparameterization operation and pruning on the overall network structure; step 6: knowledge distillation to restore accuracy; step 7: deployment and acceleration of embedded platform.

Description

Technical fields:

[0001] This invention belongs to the field of computer vision and embedded development technology, and relates to a lightweight target detection method for embedded platforms. Background technology:

[0002] Object detection is a cutting-edge technology in recent years, belonging to a research branch of computer vision, and possesses excellent generalization and accuracy. For example, it has significant application value in pollution and foreign object detection in the water conservancy field, as well as in anomaly detection in dense crowds. It can extract object features from large amounts of training data, making hardware acceleration for object detection and embedded platforms possible.

[0003] Currently, most object detection algorithms rely on servers as the primary platform for model training and forward inference. Embedded devices, limited by their computing power, size, and power consumption, often face challenges in deployment, or suffer from poor accuracy and speed after deployment. Therefore, developing and deploying a fast object detection algorithm on an embedded platform can effectively improve the automation of inspections in water conservancy scenarios and anomaly detection in densely populated areas. Summary of the Invention:

[0004] The purpose of this invention is to provide a lightweight target detection method for embedded platforms, which overcomes the problems of large network parameters, slow detection speed, and poor accuracy in existing technologies. This invention not only solves the problems of slow speed and low accuracy in current detection methods, but also optimizes the computational graph at the hardware level for specific operators, enabling fast and accurate target detection on resource-constrained devices.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] 1. A lightweight target detection method for embedded platforms, characterized by the following steps:

[0007] Step 1: Obtain basic data and create a dataset;

[0008] Step 2: Data augmentation preprocessing;

[0009] Step 3: Build a baseline network model and pre-train it;

[0010] Step 4: Build a lightweight detection network model;

[0011] Step 5: Perform reparameterization and pruning on the overall network structure;

[0012] Step 6: Knowledge distillation restores accuracy;

[0013] Step 7: Deployment and acceleration of the embedded platform.

[0014] In step 4

[0015] The lightweight detection network model consists of three parts: a backbone network, a neck network, and a detection head.

[0016] The backbone network adopts an improved ResNet50 structure. The process is as follows: an image with a resolution of 416×416 is input, and after preprocessing such as image enhancement, it is input into the network to generate feature maps of sizes 52×52, 26×26, and 13×13 in sequence. A convolution operation with a stride of 2 is used to downsample the feature maps. The feature extraction of each layer adopts the form of multiple basic residual blocks as basic units stacked together. The residual units in the ResNet50 backbone are replaced by 1*1 convolutions instead of 3*3 convolutional networks.

[0017] The backbone network is constructed by halving the number of channels in the feature layers fed into the neck network and matching the resolution.

[0018] The neck network adopts a dual-feature pyramid structure to perform multi-scale feature fusion. It enhances semantic and localization features using top-down and bottom-up methods to extract more effective features and obtain three different scale outputs.

[0019] The detection head part predicts the category and location of the target by performing convolution operations on the outputs of the neck network at three different scales.

[0020] The lightweight network model is trained to obtain a lightweight weight file.

[0021] In step 5

[0022] Before pruning, the lightweight network structure is reparameterized, and then pruning and knowledge distillation are performed on the reparameterized network structure to obtain the final network structure. Reparameterization involves optimizing the parameters of skip connections in the lightweight network structure to obtain a reparameterized network structure, including the following steps:

[0023] First, the input feature map is preserved in the convolutional layer, batch normalization layer, and activation function layer. Specifically, the convolutional layer is subjected to identity mapping and the same number of channels are inserted. The batch normalization layer uses mathematical approximation to ensure that the output obtained after this layer is the same as the input. The activation function layer preserves the values ​​of the input for the jump connection. Second, the feature map preserved in the previous step is merged into the output feature map.

[0024] Pruning involves pruning the weights of each layer in the reparameterized network structure by region. Structural pruning is performed within regions with similar structures, while unstructured pruning is performed between regions. The scaling factor in the batch normalization layer is used as a criterion; channels with smaller scaling factors are considered less important and are pruned accordingly. Finally, the pruned weights are fine-tuned to obtain the pruned weight file.

[0025] Step 7 includes the following steps:

[0026] 7.1 The network model was deployed on the RK3399PRO embedded development platform using Tengine, the AI ​​inference framework launched by OPEN AI LAB;

[0027] 7.2 First, burn the Ubuntu system image onto the main control board;

[0028] 7.3 Use a download tool on the system to download the Tengine source code, install the dependency packages, configure the environment, and compile the source code;

[0029] 7.4 Initialize Tengine to complete the loading and conversion of the final detection model;

[0030] 7.5 The computational graph is optimized using this framework. Specifically, the operators of the final detection model are further optimized by using mathematical approximations to iterate formulas containing linear relationships, thereby minimizing the intermediate processes of data loading and storage. Before operator optimization, the inference processes for convolution, batch normalization, and activation functions are as follows:

[0031] y Conv =w*x+b

[0032]

[0033] y0=SiLU(y bn )

[0034] Where y Conv For the convolution output, y bn For batch normalized output, y0 is the output after activation function, w is the original weight, x is the feature variable, b is the bias, SiLU is the activation function, γ is the scaling factor, β is the translation factor, ε is a constant, μ is the mean, and σ is the standard deviation.

[0035] After operator optimization, the convolutional block inference process is as follows:

[0036]

[0037]

[0038] y bn=w′*x+b′

[0039] y o =SiLU(w′*x+b′)

[0040] Where w′ is the approximate weight, x is the feature variable, and b′ is the approximate bias;

[0041] 7.6 Obtain the input tensor of the computation graph and set the size of the tensor;

[0042] 7.7 The Tengine framework automatically performs graph slicing based on the computation graph and tensors;

[0043] 7.8 Running the computation graph: Tengine's heterogeneous acceleration computing library performs heterogeneous parallel computing on multiple CPUs and GPUs inside the RK3399PRO to maximize the performance of each chip.

[0044] 7.9 Complete model inference, obtain detection results, and release resources.

[0045] Compared with the prior art, the advantages and effects of the present invention are as follows:

[0046] 1. This invention improves target detection speed while maintaining detection accuracy by simultaneously optimizing both the neural network algorithm and hardware device levels, and can be deployed on embedded platforms. It solves the problems of low detection accuracy and slow recognition speed.

[0047] 2. This invention improves the target detection speed under traditional methods: By constructing the original network model and then performing network pruning and compression, as well as knowledge distillation to restore accuracy, the number of parameters and computational load of the neural network model are reduced, thereby improving the inference speed of the network while ensuring minimal decrease in accuracy.

[0048] 3. This invention optimizes some operators on an embedded platform: The resulting lightweight neural network is deployed on an embedded device, and in combination with the resource-constrained characteristics of embedded hardware devices, targeted hardware acceleration optimizations are performed to improve the inference speed on embedded devices. Attached image description:

[0049] Figure 1 This is an overall schematic diagram of the implementation of the present invention;

[0050] Figure 2 This is a flowchart illustrating the inference process implemented by the present invention on an embedded platform;

[0051] Figure 3 This is a diagram showing the target detection effect in an embodiment of the present invention. Detailed implementation method:

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0053] This invention is a lightweight object detection method for embedded platforms. First, a basic dataset is constructed using methods such as web crawling, acquiring video data, and camera shooting, and then preprocessed by image enhancement and data augmentation. Second, a deep learning-based object detection network model is constructed and pruned. Third, the lightweight object network model is hardware-accelerated and deployed. Finally, the object detection model calculation results are output.

[0054] See Figure 1 and Figure 2 The present invention specifically includes the following steps:

[0055] Step 1: Obtain basic data and create a dataset;

[0056] 1. The dataset specifically refers to image information obtained through methods such as web crawling, film and television data acquisition, and camera shooting, and manually labeled, and divided into training set and test set.

[0057] Step 2: Data augmentation preprocessing;

[0058] 2.1 To address the issues of insufficient sample size and small target size in the dataset, it is necessary to expand it.

[0059] 2.2 Specifically, data augmentation can be achieved by adding noise, filtering, and using the Mixup method on the dataset to obtain the final dataset.

[0060] Step 3: Build a baseline network model and pre-train it;

[0061] 3.1 The original YOLOv5 network was pre-trained on the final dataset to obtain the baseline weight file.

[0062] Step 4: Build a lightweight detection network model;

[0063] The lightweight detection network model described in 4.1 mainly consists of three parts: a backbone network, a neck network, and a detection head.

[0064] 4.2 The backbone network adopts an improved ResNet50 structure. The process is as follows: input a 416×416 resolution image, and input it into the network after preprocessing such as image enhancement. Feature maps of size {52×52, 26×26, 13×13} are generated sequentially. A convolution operation with a stride of 2 is used to downsample the feature maps. The feature extraction of each layer adopts the form of multiple basic residual blocks as the basic unit superposition.

[0065] 4.3 Specifically, the residual units in the ResNet50 backbone are replaced with 1x1 convolutions instead of 3x3 convolutional networks, reducing the number of network parameters and increasing network depth to achieve higher network capacity and complexity. The multi-layer activation functions increase the nonlinearity in the convolution process, enhancing the network's expressive power.

[0066] 4.4 To balance network accuracy and computational complexity, and to make the ResNet50 more suitable for current detection networks, the number of channels in the last three stages of the ResNet50 backbone network is halved, and the resolution is matched to construct the backbone network.

[0067] The neck network described in 4.4 adopts a dual-feature pyramid structure to perform multi-scale feature fusion. It enhances semantic and localization features using top-down and bottom-up approaches, extracts more effective features, and obtains three different scale outputs to improve the model's detection accuracy for targets at different scales.

[0068] The detection head section described in 4.5 mainly predicts the category and location of the target by performing convolution operations on the outputs of the neck network at three different scales.

[0069] 4.6 The lightweight network model is trained to obtain a lightweight weight file.

[0070] Step 5: Perform reparameterization and pruning on the lightweight network structure;

[0071] 5.1 Existing pruning operations for neural networks often involve direct structural or unstructured pruning, resulting in significant accuracy degradation and poor pruning performance. This invention addresses this by first reparameterizing the lightweight network structure before pruning, and then performing pruning and knowledge distillation on the reparameterized network structure to obtain the final network structure. This approach leads to faster model inference speed and higher accuracy.

[0072] 5.2 Specifically, reparameterization refers to optimizing the parameters of skip connections in the lightweight network structure to obtain a reparameterized network structure.

[0073] First, the input feature maps are preserved in the convolutional layer, batch normalization layer, and activation function layer. Specifically, the convolutional layer is subjected to identity mapping and the same number of channels are inserted. The batch normalization layer uses mathematical approximation to ensure that the output obtained after this layer is the same as the input. The activation function layer preserves the values ​​of the jump connection inputs.

[0074] Secondly, the feature maps retained in the previous step are merged into the output feature map. This removal of skip connections reduces the model's storage footprint during inference. Verification shows that the number of parameters after reparameterization in this invention is only 30.5% of that of the lightweight network, thus improving detection speed.

[0075] 5.3 The pruning strategy provided by this invention performs regional pruning on the weights of each layer in the reparameterized network structure. Structural pruning is performed within structurally similar regions, while unstructured pruning is performed between regions. The scaling factor in the batch normalization layer is used as a criterion; channels with smaller scaling factors are considered less important and are pruned accordingly. Finally, the pruned weights are fine-tuned to obtain the pruned weight file.

[0076] Step 6: Knowledge distillation restores accuracy;

[0077] 6.1 The lightweight weight file is used as the teacher network, and the pruned weight file is used as the student network.

[0078] 6.2 The overall loss function of knowledge distillation is L total ′=L cls ′+L reg ′, where L cls ' represents the classification loss L of the final network. cls ′=L cls +αL cls-teacher L reg ′ is the predicted bounding box loss, L reg ′=L reg +βL reg-teacher L cls-teacher and L reg-teacher This refers to knowledge of the teacher network. The original classification total loss function is L. total =L cls +L reg .

[0079] 6.2 By fine-tuning the pruned weight file using the loss function, the knowledge distillation of the lightweight weight file can be completed, thus obtaining the final detection model WP-YOLO.

[0080] Step 7: Deployment and acceleration of the embedded platform;

[0081] 7.1 The network model was deployed on the RK3399PRO embedded development platform using Tengine, the AI ​​inference framework launched by OPEN AI LAB;

[0082] 7.2 First, burn the Ubuntu system image onto the main control board;

[0083] 7.3 Use a download tool on the system to download the Tengine source code, install the dependency packages, configure the environment, and compile the source code;

[0084] 7.4 Initialize Tengine to complete the loading and conversion of the final detection model;

[0085] 7.5 This framework is used to optimize the computational graph. Specifically, the operators of the final detection model are further optimized by iterating the formulas containing linear relationships using mathematical approximations. This minimizes the intermediate processes of data loading and storage, thereby accelerating inference. Before operator optimization, the inference process for convolution, batch normalization, and activation functions is as follows, where y Conv For the convolution output, y bn For batch normalized output, y0 is the output after activation function, w is the original weight, x is the feature variable, b is the bias, SiLU is the activation function, γ is the scaling factor, β is the translation factor, ε is a constant, μ is the mean, and σ is the standard deviation.

[0086] y Conv =w*x+b

[0087]

[0088] y0=SiLU(y bn )

[0089] After operator optimization, the inference process of the convolutional block is as follows, where w′ is the approximate weight, x is the feature variable, and b′ is the approximate bias.

[0090]

[0091]

[0092] y bn =w′*x+b′

[0093] y o =SiLU(w′*x+b′)

[0094] 7.6 Obtain the input tensor of the computation graph and set the size of the tensor.

[0095] 7.7 The Tengine framework performs automatic graph slicing based on the computation graph and tensors.

[0096] 7.8 Running the computation graph: Tengine's heterogeneous acceleration computing library performs heterogeneous parallel computing on multiple CPUs and GPUs inside the RK3399PRO, maximizing the performance of each chip as much as possible.

[0097] 7.9 Complete model inference, obtain detection results, and release resources.

[0098] Example:

[0099] The server-side software environment used in this embodiment is: Ubuntu 18.04 operating system, GPU acceleration software CUDA 11.1 and CUDNN 8.2, PyTorch deep learning framework, and Python programming language; the hardware environment is: AMD-R7-5800 processor and NVIDIA M6000 graphics card. The RK3399PRO development board is used as the hardware deployment and acceleration platform.

[0100] In this embodiment, the dataset described in step 2.1 is divided into a training dataset and a test dataset in a 9:1 ratio. During training, the initial learning rate is set to 0.01, the momentum is set to 0.9, the image input pixel size is 416×416, and WP-YOLO is iterated for 300 rounds. After each round of iteration, a model weight file is generated and saved. The model weight file is used for model testing.

[0101] This embodiment compares the test results of the final detection model WP-YOLO with the performance of currently popular deep learning object detection models (YOLOv5, RetinaNet, and FCOS). The performance comparison of different models is shown in Table 1. As shown in Table 1, the WP-YOLO model achieves a detection accuracy of 81.3%, higher than YOLOv5, RetinaNet, and FCOS. The WP-YOLO model also achieves a detection speed of 127.2 FPS, higher than YOLOv5, RetinaNet, and FCOS. The WP-YOLO model has a weight of 67MB. The results in Table 1 demonstrate that the lightweight object detection network proposed in this invention has high detection accuracy. The Tengine deep learning framework is used. Comparison results with other networks are shown in Table 2. Table 2 shows that this network has the highest detection speed on embedded computing platforms, proving the practicality of this network structure on embedded computing platforms. See also... Figure 3 , Figure 3 This is a diagram showing the target detection effect in an embodiment of the present invention.

[0102] Table 1. Experimental results of model speed test based on PC.

[0103]

[0104] Table 2. Experimental results of model speed test based on embedded platform

[0105]

[0106] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. All equivalent structural changes made based on the description and drawings of the present invention should be included within the scope of patent protection of the invention.

Claims

1. A lightweight target detection method for embedded platforms, characterized in that: Includes the following steps: Step 1: Obtain basic data and create a dataset; Step 2: Data augmentation preprocessing; Step 3: Build a baseline network model and pre-train it; 3.1 The original YOLOv5 network was pre-trained on the final dataset to obtain the baseline weight file; Step 4: Build a lightweight detection network model; Step 5: Perform reparameterization and pruning on the overall network structure; Step 6: Knowledge distillation restores accuracy; Step 7: Deployment and acceleration of the embedded platform; In step 4, the lightweight detection network model consists of three parts: a backbone network, a neck network, and a detection head. The backbone network adopts an improved ResNet50 structure. The process is as follows: an image with a resolution of 416×416 is input, and after image enhancement preprocessing, it is input into the network to generate feature maps of sizes 52×52, 26×26, and 13×13 in sequence. A convolution operation with a stride of 2 is used to downsample the feature maps. The feature extraction of each layer adopts the form of multiple basic residual blocks as basic units stacked together. The residual units in the ResNet50 backbone are replaced by 1*1 convolutions instead of 3*3 convolutional networks. The backbone network is constructed by halving the number of channels in the feature layer fed into the neck network and matching the resolution. The neck network adopts a dual-feature pyramid structure to perform multi-scale feature fusion. It enhances semantic and localization features using top-down and bottom-up methods to extract more effective features and obtain three different scale outputs. The detection head part predicts the category and location of the target by performing convolution operations on the outputs of the neck network at three different scales. The lightweight detection network model is trained to obtain a lightweight weight file. In step 5, the lightweight detection network model is reparameterized before pruning, and then pruning and knowledge distillation are performed on the reparameterized network structure to obtain the final network structure. Reparameterization is to optimize the parameters of the skip connections in the lightweight detection network model to obtain the reparameterized network structure. Step 7 includes the following steps: 7.1 The network model was deployed on the RK3399PRO embedded development platform using Tengine, the AI ​​inference framework launched by OPEN AI LAB; 7.2 First, burn the Ubuntu system image onto the main control board; 7.3 Use a download tool on the system to download the Tengine source code, install the dependency packages, configure the environment, and compile the source code; 7.4 Initialize Tengine to complete the loading and conversion of the final detection model; 7.5 This framework is used to optimize the computational graph. Specifically, the operators of the final detection model are further optimized by using mathematical approximations to iterate formulas containing linear relationships, thereby minimizing the intermediate processes of data loading and storage. Before operator optimization, the inference processes for convolution, batch normalization, and activation functions are as follows: in For the convolution output, To batch normalize the output results, The output result after the activation function is applied. For the original weights, As characteristic variables, For bias, For activation function, This is the scaling factor. The translation coefficient is... It is a constant. The mean, Standard deviation; After operator optimization, the convolutional block inference process is as follows: in For approximate weights, As characteristic variables, This is an approximate post-bias; 7.6 Obtain the input tensor of the computation graph and set the size of the tensor; 7.7 The Tengine framework automatically performs graph slicing based on the computation graph and tensors; 7.8 Running the computation graph: Tengine's heterogeneous acceleration computing library performs heterogeneous parallel computing on multiple CPUs and GPUs inside the RK3399PRO to maximize the performance of each chip. 7.9 Complete model inference, obtain detection results, and release resources.

2. The lightweight target detection method for embedded platforms according to claim 1, characterized in that: Step 5 includes the following steps: First, the input feature map is preserved in the convolutional layer, batch normalization layer, and activation function layer. Specifically, the convolutional layer is subjected to identity mapping and the same number of channels are inserted. The batch normalization layer uses mathematical approximation to ensure that the output obtained after this layer is the same as the input. The activation function layer preserves the values ​​of the input for the jump connection. Second, the feature map preserved in the previous step is merged into the output feature map. Pruning involves pruning the weights of each layer in the reparameterized network structure by region. Structural pruning is performed within regions with similar structures, while unstructured pruning is performed between regions. The scaling factor in the batch normalization layer is used as a criterion; channels with smaller scaling factors are considered less important and are pruned accordingly. Finally, the pruned weights are fine-tuned to obtain the pruned weight file.

Images