A lightweight target detection method for intelligent terminals

By designing a lightweight object detection network on edge devices, combining CBL, CSP and Dense modules, the problems of large amount of deep learning object detection and large storage occupancy on edge devices are solved, and efficient object detection and small object recognition are achieved to ensure that Detection accuracy and inference speed.

CN113989518BActive Publication Date: 2024-11-15HEFEI INSTITUTE OF PHYSICAL SCIENCE CHINESE ACADEMY OF SCIENCES
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202111315378.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-08
Publication Date
2024-11-15
Estimated Expiration
2041-11-08

AI Technical Summary

Technical Problem

Existing deep learning object detection methods are difficult to deploy on edge devices, with large computing volume, large storage space, slow inference speed, and difficult to ensure detection accuracy and efficiency at the same time.

Method used

A lightweight object detection method was designed, using backbone network module, detection head module and dense connection module, and a lightweight object detection network was built through CBL, CSP and Dense modules, combined with a dual-stream detection classifier to reduce the computational complexity and Model size and train and deploy on the server.

Benefits of technology

It realizes efficient object detection on resource-constrained smart terminals, solves the problem of small target loss, and achieves fast inference speed and high detection accuracy under small model sizes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113989518B_ABST
    Figure CN113989518B_ABST
Patent Text Reader

Abstract

The present invention discloses a lightweight target detection method applied to a smart terminal, comprising the following steps: 1. constructing a lightweight target detection network model; 2. training the lightweight target detection network model on a server; 3. deploying the trained lightweight target detection network model on a smart terminal to implement target detection. The algorithm model proposed in the present invention improves the inference speed under a smaller model size, and can ensure the detection accuracy, while solving the problem of small target loss, and can be deployed and applied in resource-constrained smart terminals.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of deep learning target detection, and in particular to a lightweight target detection method that can be deployed on an edge device. Background Art

[0002] After years of development, the field of object detection has completely shifted from manual feature-based to deep learning-based technical solutions. With the R-CNN object detection method realizing an industrial-grade deep learning object detection framework, the object detection technology using deep learning has been rapidly iterated. Excellent algorithms such as two-stage object detectors such as SPPNet, Fast R-CNN, Faster R-CNN, and single-stage object detectors such as YOLO, SSD, and RetinaNet have been continuously proposed, and performance indicators such as the network's abstract ability and the positioning accuracy of object candidate box detection have also been further improved.

[0003] Although deep learning technology has an absolute advantage in the field of target detection, the training and reasoning of deep learning networks have placed high demands on the computing power and storage capacity of devices. For example, in the target detection algorithm, if VGG16 is used as an object recognition model, the network has more than 130 million parameters, occupies more than 500 MB of space, and requires more than 30 billion floating-point operations to complete a recognition task. In addition, the two-stage target detector will first select thousands of candidate boxes, each of which requires convolutional network calculations. The amount of calculation required for target detection is undoubtedly huge and time-consuming. Although the advent of Faster R-CNN and single-stage target detection methods has further improved the amount of calculation and speed of target detection, it is still difficult to deploy applications directly on edge devices.

[0004] In actual application scenarios, deep learning servers equipped with GPUs have high requirements for deployment sites and operating environments. Using embedded devices on site to quickly complete target detection has become an increasingly common demand. In particular, with the popularization of the Internet of Things in various industries and the application of edge devices with certain computing capabilities in the Internet of Things, computing has gradually migrated to edge devices, but most edge devices are not designed for computing-intensive tasks, and power consumption and latency will become problems. For example, in the fields of new energy vehicles and intelligent connected vehicles, driving behavior analysis based on visual data, analysis of scenes inside and outside the car, and understanding of indoor and outdoor scenes by intelligent robots using vision all need to ensure the accuracy and effectiveness of target detection at the same time, and have certain restrictions on conditions such as power supply and size of the detection equipment.

[0005] The existing target detection methods and models have the following main problems: target detection methods and models with high detection accuracy and high computing resource consumption have slow reasoning speed and occupy large storage space; target detection methods and models with fast reasoning speed and low computing consumption have low detection accuracy. For application scenarios that need to ensure both detection accuracy and detection efficiency, the existing mainstream target detection models are difficult to directly deploy and apply to edge devices with limited computing and storage resources. Summary of the invention

[0006] In view of the problems of large computational complexity, large storage space occupation, slow inference speed, etc. of deploying deep learning target detection methods on edge devices, the present invention proposes a lightweight target detection method applied to smart terminals, in order to ensure detection accuracy, solve the problem of small target loss, and realize deployment and application in resource-constrained smart terminals.

[0007] The present invention adopts the following technical solutions to solve the technical problems:

[0008] The invention provides a lightweight target detection method for a smart terminal, which comprises the following steps:

[0009] Step 1: Obtain several 2D image data marked with target detection boxes and their target detection categories and perform normalization processing to construct a training data set;

[0010] Step 2: construct a lightweight object detection network, and input the training data set into the lightweight object detection network;

[0011] The lightweight target detection network includes: a backbone network module and a detection head module;

[0012] The backbone network module includes: a primary feature extraction module, an intermediate feature extraction module, and a dense connection module;

[0013] The detection head module includes: two branch structures, the structure of the first branch is: a first pyramid pooling SPP layer, a first path aggregation network PANet, a target pre-selection box calculation module, and a first YOLO HEAD; the structure of the second branch is: an upsampling layer, a tensor splicing layer, a second pyramid pooling SPP layer, a second path aggregation network PANet, a target pre-selection box calculation module, and a second YOLO HEAD;

[0014] Step 2.1 The primary feature extraction module is composed of M CBL blocks, which are denoted as CBL1, ... CBL m ,…,CBL M , among which, CBL m represents the m-th level CBL block, m=1,2,…M; the m-th level CBL block CBLm It consists of a two-dimensional convolution layer, a batch normalization layer, and an activation layer, wherein the m-th level CBL block CBL m The number of convolution kernels in the two-dimensional convolution layer is cbl_c m The step length is cbl_s m , the convolution kernel size is cbl_k m ×cbl_k m ; The activation function used in the activation layer is the Leaky ReLU function;

[0015] The image data of the training data set is input into the first level CBL block CBL1 in the primary feature extraction module and then outputs the feature map FeatureMap CBL_1 ; When m = 2, 3, ..., M, the feature map FeatureMap CBL_m-1 As the mth level CBL block CBL m Input, and get the corresponding output feature map FeatureMap CBL_m ; Thus, the M-th level CBL block CBL M Get the feature map FeatureMap finally output by the primary feature extraction module CBL_M ;

[0016] Step 2.2 The intermediate feature extraction module includes N CSP blocks, which are denoted as CSP1, ... CSP n ,…,CSP N Among them, CSP n represents the n-th level CSP block, n = 1, 2, ... N; the n-th level CSP block consists of a left branch module, a right branch module, and a Concat connection layer; the left branch module consists of a Resblock module and a convolution layer; the right branch module consists of a convolution layer and an average pooling layer; the number of convolution kernels in the convolution layer of the left branch module of the n-th level CSP block is csp_c n The step length is csp_s n , the convolution kernel size is csp_k n ×csp_k n ; The number of convolution kernels in the convolution layer of the right branch module of the nth level CSP block is csp_c′ n , the step length is csp_s′ n , the convolution kernel size is csp_k x ′×csp_k x ′;

[0017] The feature map FeatureMap CBL_MInput the first level CSP block CSP1 in the intermediate feature extraction module, and divide it into two feature maps in CSP1. One part of the feature map is sent to the Resblock module and the convolution layer of the left branch module in turn for calculation to obtain the feature map FeatureMap CSP1_1 The other part is sent to the convolution layer and average pooling layer of the right branch module for calculation to obtain the feature map FeatureMap CSP1_2 ; FeatureMap CSP1_1 and feature map FeatureMap CSP1_2 The Concat connection layer is used to concatenate tensors to obtain the feature map FeatureMap of the final output of the first-level CSP block CSP1 CSP_1 ;

[0018] When m = 2, 3, ..., M, the feature map FeatureMap CSP_m-1 As the mth level CSP block CSP m Input, and get the corresponding output feature map FeatureMap CSP_m ; Thus, the M-th level CSP block CSP M Get the feature map FeatureMap finally output by the intermediate feature extraction module CSP_M ;

[0019] The dense connection module in step 2.3 includes Q Dense blocks, which are respectively denoted as Dense1, ...Dense q ,…,Dense Q Among them, Dense q represents the qth level Dense block, q = 1, 2, ... Q; the qth level Dense block Dense q By E q CBR modules, F q Transition modules; wherein the CBR module consists of a convolution layer, a batch normalization layer and an activation layer; the number of convolution kernels in the convolution layer of the e-th CBR module is cbr_c e The step length is cbr_s e , the convolution kernel size is cbr_k e ×cbr_k e ,e=1,2,…,E q ; The activation function used in the activation layer of the CBR module is the ReLU function; the Transition module consists of a convolution layer and an average pooling layer; wherein the number of convolution kernels in the convolution layer of the fth Transition module is trans_c f The step length is trans_s f, the convolution kernel size is trans_k f ×trans_k f ,,f=1,2,…,F q ;

[0020] The feature map FeatureMap CSP_M Input the first level Dense block, and pass through the convolution layer, batch normalization layer, and activation layer of the first CBR module in turn to obtain the output feature map FeatureMap of the first CBR module CBR_1 ;

[0021] FeatureMap output by the first e-1 CBR modules CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_e-1 After tensor concatenation calculation, it is used as the input of the e-th CBR module, and the e-th CBR module outputs the feature map FeatureMap CBR_e ; where e=2,3,…,E q ; Thus, by E q The CBR module outputs a feature map FeatureMap CBR_Eq ;

[0022] Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_Eq After tensor concatenation calculation, it is used as the input of the first Transition module, and passes through the convolution layer and average pooling layer in the first Transition module in turn to obtain the output feature map FeatureMap of the first Transition module trans_1 ;

[0023] Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_Eq And the feature map FeatureMap output by the first f-1 Transition modules trans_1 , FeatureMap trans_2 , …, FeatureMap trans_f-1 After tensor concatenation calculation, it is used as the input of the fth Transition module, and the fth Transition module outputs the feature map FeatureMap trans_f; where f=2,3,…,F q ; Thus, by F q The Transition module outputs a feature map FeatureMap trans_Fq And it is used as the feature map FeatureMap of the final output of the first-level Dense module Dense_1 ;

[0024] When q=2,3,…,Q, the feature map FeatureMap Dense_q-1 As the qth level Dense q The input of the module and obtain the corresponding output feature map FeatureMap Dense_q ; Thus, the Qth level Dense Q The module obtains the feature map FeatureMap finally output by the dense connection module Dense_Q ;

[0025] Step 2.4 Feature Map FeatureMap Dense_Q respectively sent to the first branch and the second branch of the detection head module;

[0026] The feature map FeatureMap is processed by the pyramid pooling SPP layer of the first branch and the path aggregation network PANet Dense_Q After processing, a feature map FeatureMap1 is output; the target pre-selection box calculation module performs k-means clustering calculation on all target detection box annotation data of the training data set, outputs several anchor values ​​of the target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap1, and finally outputs several first target detection results;

[0027] The upsampling layer in the second branch performs the feature map FeatureMap Dense_Q After upsampling, we get the feature map FeatureMap Dense_Q_upsample , and then the feature map FeatureMap Dense_Q_upsample And the feature map FeatureMap output by the first-level CSP1 block in the intermediate feature extraction module CSP_1 The tensor splicing layer is used to perform tensor splicing to obtain a tensor splicing result, and the tensor splicing result is then sequentially input into the second pyramid pooling SPP layer and the second path aggregation network PANet to output a feature map FeatureMap2; the target pre-selection box calculation module performs k-means clustering calculation on all target detection box annotation data of the training data set, outputs several anchor values ​​of the target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap2, and finally outputs several second target detection results;

[0028] Step 3: training the lightweight object detection network using a stochastic gradient descent method on the server and updating the network weights to generate an optimal lightweight object detection network;

[0029] Step 4: Deploy the optimal lightweight object detection network structure and trained network weights to the smart terminal;

[0030] Step 5: In the smart terminal, the image captured by the real-time camera is used as input data, or the static image stored locally is used as input data, and the trained optimal lightweight target detection network is called to perform forward reasoning calculation on the input data to obtain the target detection result; and each target detection result is<BoundingBox,Class,ConfidenceScore> The triplet represents that BoundingBox is the detection box position, Class is the detection category, and ConfidenceScore is the confidence score.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] 1. The target detection method proposed in the present invention can enable the target detection algorithm based on deep neural network to run not only in servers equipped with GPU cards, but also be deployed on edge devices with low power consumption, thereby providing target detection reasoning calculation under the condition of limited capacity and computing resources.

[0033] 2. The backbone network design of the algorithm model proposed in the present invention integrates the Dense module, CBL module, and CSP module, which effectively reduces the computational complexity of the model. The backbone network parameter volume is 14.9M, while the backbone network parameter volume used in mainstream models such as YOLOv4 is about 116.8M, which achieves a significant reduction in model size and helps to implement deployment on edge devices with limited storage resources.

[0034] 3. The present invention realizes multi-size target detection frame recognition by constructing a dual-stream detection classifier. At the same time, a Dense module is designed and added to the backbone network to realize the recognition and detection of small targets in the target detection scene, solving the problem of small target loss in the mainstream target detection model.

[0035] 4. The present invention can complete end-to-end detection at a frame rate of 22.6FPS with a smaller model size, has a very fast inference speed, and can ensure the detection accuracy of the target. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the model structure of the method of the present invention;

[0037] Figure 2It is a schematic diagram of the CSP module structure of the method of the present invention;

[0038] Figure 3 It is a schematic diagram of the Dense module structure of the method of the present invention;

[0039] Figure 4 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] In this embodiment, a lightweight target detection algorithm applicable to smart terminals is to construct a lightweight target detection network model by designing and integrating a Dense module, a CBL module, a CSP module, and a dual-stream detection classifier, building a backbone network and a detection head structure, and training the lightweight target detection network model on a server to obtain a weight file of the model, and then deploying the model on a smart terminal to provide target detection calculations, thereby effectively reducing the computational complexity and model size of the model, improving the reasoning speed of the model, and solving the problem of small target loss. Specifically, the method is performed in the following steps:

[0041] Step 1: Obtain several 2D image data marked with target detection boxes and their target detection categories and perform normalization processing to 1024×1024×3 to construct a training data set;

[0042] Step 2: Build a lightweight object detection network and input the training data set into the lightweight object detection network;

[0043] The structure of the lightweight object detection network is: a backbone network module for extracting features from images; a detection head module for detecting the location and category of the object based on the features extracted by the backbone network;

[0044] The backbone network module includes: primary feature extraction module, intermediate feature extraction module, and dense connection module; the primary feature extraction module is composed of M CBL blocks, which are respectively denoted as CBL1, ... CBL m ,…,CBL M , among which, CBL m represents the m-th level CBL block, m = 1, 2, ... M; the m-th level CBL block CBL m It consists of a two-dimensional convolution layer, a batch normalization layer, and an activation layer, where the m-th level CBL block CBL m The number of convolution kernels in the two-dimensional convolution layer is cbl_c m The step length is cbl_s m , the convolution kernel size is cbl_k m ×cbl_k m ; The activation function used in the activation layer is the Leaky ReLU function;

[0045] like Figure 1 As shown, the primary feature extraction module of this embodiment includes: a first CBL module CBL1, a second CBL module CBL2, the intermediate feature extraction module includes: a first CSP module CSP1, a second CSP module CSP2, and the dense connection module includes a Dense module; the detection head is a two-stream classifier; wherein, the convolution kernel of the convolution layer of CBL1 is 32, the step length is 2, and the convolution kernel size is 3×3, and the convolution kernel of the convolution layer of CBL2 is 64, the step length is 2, and the convolution kernel size is 3×3;

[0046] In this lightweight target detection network model, the CBL module uses different convolutional layers to extract features to obtain different response feature maps; the CSP module enhances the learning ability of the network through modules such as residuals, while maintaining network accuracy in the lightweight process; the Dense module alleviates the problem of gradient vanishing through layer-by-layer connections, strengthens feature propagation, and greatly reduces the number of parameters by reusing features. At the same time, the Dense module has a regularization effect, which can reduce overfitting even on a smaller training set.

[0047] The image data of the training data set is first input into the first CBL module CBL1 in the primary feature extraction module to obtain a feature map with an output dimension of 208×208×32; then it is input into the second CBL module CBL2 to obtain the output feature map FeatureMap of the primary feature extraction module CBL , the dimension of the feature map is 104×104×64;

[0048] like Figure 2 As shown, the intermediate feature extraction module includes N CSP blocks, which are denoted as CSP1, ... CSP n ,…,CSP N Among them, CSP n represents the n-th level CSP block, n = 1, 2, ... N; each CSP module consists of a left branch module, a right branch module, and a Concat connection layer; the left branch module consists of a Resblock module and a convolution layer; the right branch module consists of a convolution layer and an average pooling layer; the number of convolution kernels in the convolution layer of the left branch module of the n-th level CSP block is csp_c n The step length is csp_s n , the convolution kernel size is csp_k n ×csp_k n ; The number of convolution kernels in the convolution layer of the right branch module of the n-th level CSP block is csp_c′ n The step length is csp_s′ n , the convolution kernel size is csp_kx ′×csp_k x '; In this embodiment, the number of convolution kernels of the convolution layer in the CSP module is 64, the step size is 1, and the convolution kernel size is 3×3;

[0049] FeatureMap CBL Input the first CSP block CSP1 in the intermediate feature extraction module, and divide it into two feature maps in CSP1. One part of the feature map is sent to the Resblock module and convolution layer of the left branch module for calculation to obtain the feature map FeatureMap CSP1_1 The other part is sent to the convolution layer and average pooling layer of the right branch module for calculation to obtain the feature map FeatureMap CSP1_2 ; FeatureMap CSP1_1 and feature map FeatureMap CSP1_2 The tensors are concatenated through the Concat connection layer to obtain the feature map FeatureMap of the final output of the first-level CSP block CSP1 CSP_1 ; The dimension of the feature map is 52×52×128; similarly, the feature map FeatureMap CSP_1 Input the second CSP block CSP2 to finally get the feature map FeatureMap output by the intermediate feature extraction module CSP ; The dimension of the feature map is 26×26×128;

[0050] The dense connection module consists of Q Dense blocks, denoted as Dense1, ...Dense q ,…,Dense Q Among them, Dense q represents the qth level Dense block, q = 1, 2, ... Q; Figure 3 As shown, in this embodiment, each Dense module includes E q =4 CBR modules, F q = 1 Transition module; the CBR module includes a convolution layer, a batch normalization layer, and an activation layer; the number of convolution kernels in the convolution layer of the e-th CBR module is cbr_c e The step length is cbr_s e , the convolution kernel size is cbr_k e ×cbr_k e ,e=1,2,…,E q; In this embodiment, the convolutional layers of the CBR module all have 26 convolution kernels, a step size of 1, and a convolution kernel size of 3×3; the activation function used in the activation layer is the ReLU function; each Transition module includes a convolutional layer and an average pooling layer; among them, the number of convolution kernels in the convolutional layer of the fth Transition module is trans_c f The step length is trans_s f , the convolution kernel size is trans_k f ×trans_k f ,,f=1,2,…,F q ; In this embodiment, the number of convolution kernels of the convolution layer is 26, the step size is 1, and the convolution kernel size is 1×1;

[0051] The feature map FeatureMap output by the intermediate feature extraction module CSP Input the Dense block, and pass through the convolution layer, batch normalization layer, and activation layer of the first CBR module in turn to obtain the output feature map FeatureMap of the first CBR module CBR_1 ;

[0052] FeatureMap output by the first n-1 CBR modules CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_n-1 After tensor concatenation calculation, it is used as the input of the nth CBR module, and the nth CBR module outputs the feature map FeatureMap CBR_n ; Where n = 2, 3, 4; Thus, the fourth CBR module outputs the feature map FeatureMap CBR_4 ;

[0053] Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_Eq After tensor concatenation calculation, it is used as the input of the first Transition module, and passes through the convolution layer and average pooling layer in the first Transition module in turn to obtain the output feature map FeatureMap of the first Transition module trans_1 ;

[0054] Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_EqAnd the feature map FeatureMap output by the first f-1 Transition modules trans_1 , FeatureMap trans_2 , …, FeatureMap trans_f-1 After tensor concatenation calculation, it is used as the input of the fth Transition module, and the fth Transition module outputs the feature map FeatureMap trans_f ; where f=2,3,…,F q ; Thus, by F q The Transition module outputs a feature map FeatureMap trans_Fq And it is used as the feature map FeatureMap of the final output of the first-level Dense module Dense_1 ;

[0055] When q=2,3,…,Q, the feature map FeatureMap Dense_q-1 As the qth level Dense q The input of the module and obtain the corresponding output feature map FeatureMap Dense_q ; Thus, the Qth level Dense Q The module obtains the feature map FeatureMap of the dense connection module's final output Dense_Q ;

[0056] In this embodiment, the feature map FeatureMap output by the first four CBR modules CBR_1 , FeatureMap CBR_2 , FeatureMap CBR_3 , FeatureMap CBR_4 After tensor splicing calculation, it is used as the input of the Transition module, and passes through the convolution layer and average pooling layer in the Transition module in turn to obtain the output feature map FeatureMap of the Transition module trans , and used as the feature map FeatureMap of the final output of the Dense module Dense ;

[0057] The detection head module includes: two branch structures, the structure of the first branch is: the first pyramid pooling SPP layer, the first path aggregation network PANet, the target pre-selection box calculation module, the first YOLO HEAD; the structure of the second branch is: an upsampling layer, a tensor splicing layer, the second pyramid pooling SPP layer, the second path aggregation network PANet, the target pre-selection box calculation module, and the second YOLO HEAD;

[0058] FeatureMap Dense_Mrespectively sent to the first branch and the second branch of the detection head module;

[0059] The first branch of the pyramid pooling SPP layer and the path aggregation network PANet feature map FeatureMap Dense_M After processing, the feature map FeatureMap1 is output; the target pre-selection box calculation module performs k-means clustering calculation based on the target detection box data of the training data set, outputs the first anchor values ​​of several target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap1, and finally outputs several first target detection results;

[0060] The upsampling layer in the second branch performs feature mapping on the feature map FeatureMap Dense_M After upsampling, we get the feature map FeatureMap Dense_M_upsample , and then the feature map FeatureMap Dense_M_upsample And the feature map FeatureMap output by the first-level CSP1 block in the intermediate feature extraction module CSP_1 The tensors are input into the tensor splicing layer together for tensor splicing to obtain the tensor splicing result, and then the tensor splicing result is input into the second pyramid pooling SPP layer and the second path aggregation network PANet in sequence to output the feature map FeatureMap2; the target pre-selection box calculation module performs k-means clustering calculation based on the target detection box data of the training data set, outputs the second anchor value of several target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap2, and finally outputs several second target detection results;

[0061] like Figure 4 As shown in the figure, when using the lightweight target detection model, first build a training data set, then build a lightweight target detection network model, and then use the stochastic gradient descent method on the server to train the lightweight target detection network model and update the network weights. Finally, deploy the trained lightweight target detection network model on the smart terminal to realize target detection.

[0062] Specifically, this embodiment uses the KITTI dataset and the BDD100k dataset as training datasets. Among them, the training set of the KITTI dataset contains 7481 images, of which 7000 samples are used for training and 481 samples are used for testing. The targets contained therein are all common types in traffic scenes, with a total of 8 categories, namely cars, trucks, vans, pedestrians, pedestrians sitting down, cyclists, trams, and other miscellaneous items. The BDD100k dataset consists of 100,000 images, of which 90,000 are training images and 10,000 are test images. The targets contained therein are all common types in traffic scenes, with a total of 10 categories, namely cars, buses, trucks, trains, pedestrians, bicycles, motorcycles, cyclists, traffic lights, and traffic signs.

[0063] Then, the algorithm model is trained on the server using the image dataset. During training, the stochastic gradient descent SGD algorithm can be used in the Tensorflow framework to train the model. The relevant parameters can be set as follows: learning rate is 0.0009, weight decay is 0.0005, momentum is 0.9, maximum batch is 100000, and the number of steps is 70000 and 90000. During training, the server's GPU card can be used for computational acceleration.

[0064] Load the trained lightweight target detection network model structure and weights into smart terminal edge devices, such as NVIDIA Jeston Tx2 and other smart terminal devices. Input the image data to be identified into the trained model, call the model in the smart terminal device to perform forward reasoning calculation on the input data, and obtain the target detection result. Each target detection result will be<BoundingBox,Class,ConfidenceScore> The triplet represents that BoundingBox is the detection box position, Class is the detection category, and ConfidenceScore is the confidence score.

[0065] The performance of the lightweight object detection network model can be evaluated using the following three indicators, including: model parameter quantity, mean average precision (mAP) of object recognition, and frames per second (FPS). For example, when running the model on NVIDIA Jeston Tx2 and using the test set of the KITTI dataset, the parameter quantity, mAP, and FPS can be tested to be 20.3M, 84.6, and 22.6.

Claims

1. A lightweight target detection method for smart terminals. Its characteristics include the following steps: Step 1: Obtain several 2D image data marked with target detection boxes and their target detection categories and perform normalization processing to construct a training data set; Step 2: construct a lightweight object detection network, and input the training data set into the lightweight object detection network; The lightweight target detection network includes: a backbone network module and a detection head module; The backbone network module includes: a primary feature extraction module, an intermediate feature extraction module, and a dense connection module; The detection head module includes: two branch structures, the structure of the first branch is: a first pyramid pooling SPP layer, a first path aggregation network PANet, a target pre-selection box calculation module, and a first YOLO HEAD; the structure of the second branch is: an upsampling layer, a tensor splicing layer, a second pyramid pooling SPP layer, a second path aggregation network PANet, a target pre-selection box calculation module, and a second YOLO HEAD; Step 2.1 The primary feature extraction module is composed of M CBL blocks, which are denoted as CBL1, ... CBL m ,…,CBL M , among which, CBL m represents the m-th level CBL block, m=1,2,…M; the m-th level CBL block CBL m It consists of a two-dimensional convolution layer, a batch normalization layer, and an activation layer, wherein the m-th level CBL block CBL m The number of convolution kernels in the two-dimensional convolution layer is cbl_c m The step length is cbl_s m , the convolution kernel size is cbl_k m ×cbl_k m ; The activation function used in the activation layer is the Leaky ReLU function; The image data of the training data set is input into the first level CBL block CBL1 in the primary feature extraction module and then outputs a feature map FeatureMap CBL_1 ; When m = 2, 3, ..., M, the feature map FeatureMap CBL_m-1 As the mth level CBL block CBL m Input, and get the corresponding output feature map FeatureMap CBL_m ; Thus, the M-th level CBL block CBL M Get the feature map FeatureMap finally output by the primary feature extraction module CBL_M ; Step 2.2 The intermediate feature extraction module includes N CSP blocks, which are denoted as CSP1, ... CSP n ,…,CSP N Among them, CSP n represents the n-th level CSP block, n = 1, 2, ... N; the n-th level CSP block consists of a left branch module, a right branch module, and a Concat connection layer; the left branch module consists of a Resblock module and a convolution layer; the right branch module consists of a convolution layer and an average pooling layer; the number of convolution kernels in the convolution layer of the left branch module of the n-th level CSP block is csp_c n The step length is csp_s n , the convolution kernel size is csp_k n ×csp_k n ; The number of convolution kernels in the convolution layer of the right branch module of the n-th level CSP block is csp_c′ n , the step length is csp_s′ n , the convolution kernel size is csp_k′ x ×csp_k′ x ; The feature map FeatureMap CBL_M Input the first level CSP block CSP1 in the intermediate feature extraction module, and divide it into two feature maps in CSP1. One part of the feature map is sent to the Resblock module and the convolution layer of the left branch module in turn for calculation to obtain the feature map FeatureMap CSP1_1 The other part is sent to the convolution layer and average pooling layer of the right branch module for calculation to obtain the feature map FeatureMap CSP1_2 ; FeatureMap CSP1_1 and feature map FeatureMap CSP1_2 The Concat connection layer is used to concatenate tensors to obtain the feature map FeatureMap of the final output of the first-level CSP block CSP1 CSP_1 ; When m = 2, 3, ..., M, the feature map FeatureMap CSP_m-1 As the mth level CSP block CSP m Input, and get the corresponding output feature map FeatureMap CSP_m ; Thus, the M-th level CSP block CSP M Get the feature map FeatureMap finally output by the intermediate feature extraction module CSP_M ; The dense connection module in step 2.3 includes Q Dense blocks, which are respectively denoted as Dense1, ...Dense q ,…,Dense Q Among them, Dense q represents the qth level Dense block, q = 1, 2, ... Q; the qth level Dense block Dense q By E q CBR modules, F q Transition modules; wherein the CBR module consists of a convolution layer, a batch normalization layer and an activation layer; the number of convolution kernels in the convolution layer of the e-th CBR module is cbr_c e The step length is cbr_s e , the convolution kernel size is cbr_k e ×cbr_k e ,e=1,2,…,E q ; The activation function used in the activation layer of the CBR module is the ReLU function; the Transition module consists of a convolution layer and an average pooling layer; wherein the number of convolution kernels in the convolution layer of the fth Transition module is trans_c f The step length is trans_s f , the convolution kernel size is trans_k f ×trans_k f ,f=1,2,…,F q ; The feature map FeatureMap CSP_M Input the first level Dense block, and pass through the convolution layer, batch normalization layer, and activation layer of the first CBR module in turn to obtain the output feature map FeatureMap of the first CBR module CBR_1 ; FeatureMap output by the first e-1 CBR modules CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_e-1 After tensor concatenation calculation, it is used as the input of the e-th CBR module, and the e-th CBR module outputs the feature map FeatureMap CBR_e ; where e=2,3,…,E q ; Thus, by E q The CBR module outputs a feature map FeatureMap CBR_Eq ; Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_Eq After tensor concatenation calculation, it is used as the input of the first Transition module, and passes through the convolution layer and average pooling layer in the first Transition module in turn to obtain the output feature map FeatureMap of the first Transition module trans_1 ; Front E q FeatureMap output by the CBR module CBR_1 , FeatureMap CBR_2 , …, FeatureMap CBR_Eq And the feature map FeatureMap output by the first f-1 Transition modules trans_1 , FeatureMap trans_2 , …, FeatureMap trans_f-1 After tensor concatenation calculation, it is used as the input of the fth Transition module, and the fth Transition module outputs the feature map FeatureMap trans_f ; where f=2,3,…,F q ; Thus, by F q The Transition module outputs a feature map FeatureMap trans_Fq And it is used as the feature map FeatureMap of the final output of the first-level Dense module Dense_1 ; When q=2,3,…,Q, the feature map FeatureMap Dense_q-1 As the qth level Dense q The input of the module and obtain the corresponding output feature map FeatureMap Dense_q ; Thus, the Qth level Dense Q The module obtains the feature map FeatureMap finally output by the dense connection module Dense_Q ; Step 2.4 Feature Map FeatureMap Dense_Q respectively sent to the first branch and the second branch of the detection head module; The feature map FeatureMap is processed by the pyramid pooling SPP layer of the first branch and the path aggregation network PANet Dense_Q After processing, a feature map FeatureMap1 is output; the target pre-selection box calculation module performs k-means clustering calculation on all target detection box annotation data of the training data set, outputs several anchor values ​​of the target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap1, and finally outputs several first target detection results; The upsampling layer in the second branch performs the feature map FeatureMap Dense_Q After upsampling, we get the feature map FeatureMap Dense_Q_upsample , and then the feature map FeatureMap Dense_Q_upsample And the feature map FeatureMap output by the first-level CSP1 block in the intermediate feature extraction module CSP_1 The tensor splicing layer is used to perform tensor splicing to obtain a tensor splicing result, and the tensor splicing result is then sequentially input into the second pyramid pooling SPP layer and the second path aggregation network PANet to output a feature map FeatureMap2; the target pre-selection box calculation module performs k-means clustering calculation on all target detection box annotation data of the training data set, outputs several anchor values ​​of the target pre-selection boxes, and then inputs them into YOLO HEAD together with the feature map FeatureMap2, and finally outputs several second target detection results; Step 3: training the lightweight object detection network using a stochastic gradient descent method on the server and updating the network weights to generate an optimal lightweight object detection network; Step 4: Deploy the optimal lightweight object detection network structure and trained network weights to the smart terminal; Step 5: In the smart terminal, the image captured by the real-time camera is used as input data, or the static image stored locally is used as input data, and the trained optimal lightweight target detection network is called to perform forward reasoning calculation on the input data to obtain the target detection result; and each target detection result is<BoundingBox,Class,ConfidenceScore> The triplet represents that BoundingBox is the detection box position, Class is the detection category, and ConfidenceScore is the confidence score.

Citation Information

Cited By

  • A lightweight object detection method based on multiple receptive fields and attention feature pyramids

    CN114937151A