A lightweight target detection network applied to an embedded device and a training method thereof

Abstract

The application provides a lightweight target detection network applied to an embedded device and a corresponding training method. The detection network comprises a backbone network, a prediction network, and a neck network connected with the backbone network and an output end. The backbone network and the neck network reserve strong feature expressiveness while compressing the parameter quantity, and the detection accuracy and efficiency are considered. The detection network maps an input image into feature maps of different scales through the backbone network for detecting targets of different sizes, and fuses small-size feature maps with adjacent large-size feature maps through up-sampling to supplement the semantic information lacking in the large-size feature maps. The fused feature maps are further improved in feature expression through a lightweight residual block in the neck network, and the probability size of the output target center located at each pixel point of the feature map and the corresponding center and width-height offset are predicted.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target detection in computer vision, and particularly relates to a lightweight target detection network suitable for embedded device end and a corresponding training method. BACKGROUND

[0002] With the emergence of deep learning technology, target detection as a popular research direction in the field of computer vision has made great progress in recent years. Compared with traditional target detection technology, target detection based on deep neural network has the characteristics of fast speed and high precision. From the initial two-stage detection network such as Fast / Faster RCNN to today's single-stage detection network such as ssd, yolo, the detection network has significantly improved the detection efficiency while ensuring the detection accuracy.

[0003] The main components of deep neural network are convolution, pooling and activation, which are combined to construct a series of nonlinear transformations. The more the parameter quantity of the network is, the stronger the nonlinear expression ability and generalization ability are, and the increase of the parameter will further improve the complexity of the calculation, so the good performance of deep learning often needs the powerful computing ability of the device as support. Limited computing power and storage space make the mainstream deep neural network unable to run on the embedded device.

[0004] Due to the high computational complexity of neural network, the computing ability of the device has strict requirements, so the target detection network usually needs to be run on a device (such as a cloud server) equipped with a high-performance GPU. However, under the premise of limited network bandwidth and real-time response, applications with high real-time requirements cannot rely on cloud-based processing. In comparison, embedded devices have the characteristics of not relying on network computing and no computing delay.

[0005] However, the computing power of embedded devices is limited, and running real-time target detection on embedded devices relies on efficient neural networks, which means that the embedded neural network must be simplified in structure and compressed in parameters. SUMMARY

[0006] The purpose of the present application is to provide a lightweight target detection network applied to embedded devices and a corresponding training method, so as to realize real-time detection and positioning of targets by embedded devices (such as cameras).

[0007] In the first aspect, the present application provides a lightweight target detection network applied to embedded devices, which is composed of a backbone network, a neck network and a prediction network.

[0008] The input object of the backbone network in the network is a preprocessed image to be detected, the backbone network maps the input image to a plurality of feature maps of different sizes, and is used for detecting targets of different sizes. The neck network is composed of a plurality of lightweight residual blocks, the input end of the neck network is connected to the output end of the backbone network, the neck network sequentially fuses feature maps of sizes close to each other, each residual block in the neck network further enhances the feature expressiveness of the corresponding size feature map, and finally outputs the fused feature map to a subsequent prediction network for result prediction.

[0009] The backbone network is composed of a plurality of double-path dense modules, for the input feature map, features of the input feature map under different scale receptive fields are obtained, and the feature map size output by each output layer of the network is 1 / 2 of the size of the previous output feature map. The neck network fuses the feature map of the current size and the feature map output by the previous layer through splicing after performing feature up-sampling on the feature map of the current size through a near-neighbor interpolation method. The prediction network is similar to the prediction network of a traditional single-stage target detection, and is a double-path network, and two branches are used to predict the category of a target and the offset of the center and the width and height of the target relative to a current pixel position.

[0010] Finally, all prediction results are processed by using non-maximum suppression to filter coordinate frames without objects and multiple invalid coordinate frames surrounding the same target.

[0011] As an optional implementation manner, according to the distribution of the sizes of the detected targets and the detection efficiency, the backbone network can output feature maps of 2-3 sizes.

[0012] As an optional implementation manner, based on the aspect ratio distribution of the coordinate frames of the detected targets, the size of the input layer of the backbone network can be adjusted.

[0013] As an optional implementation manner, according to the size of the detected target relative to the size of the original image, the size of the first output feature map of the backbone network can be 1 / 8 or 1 / 4 of the input size.

[0014] As an optional implementation manner, a spatial attention mechanism can be added to the prediction network to enhance the prediction ability of the prediction network for the center position of the target.

[0015] As an optional implementation manner, based on the occlusion degree between targets of the same kind, an improved non-maximum suppression method such as Soft-NMS and DIoU can be used.

[0016] In a second aspect, the application provides a training method of the lightweight network, and the method comprises the following steps:

[0017] 1) Randomly extract training sample images with a quantity of batch_size*2, and splice the images two by two, and the splicing manner is randomly selected from horizontal and vertical splicing.

[0018] 2) Scale the spliced image to a fixed size and normalize the pixel value to the interval of 0-1;

[0019] 3) Input the preprocessed training sample image into the backbone network to obtain a plurality of size feature maps;

[0020] 4) Obtain the clustering center of the target frame size in the training sample by using the Kmeans clustering algorithm, calculate the average overlap degree of the coordinate frame width and height and the clustering center width and height according to different numbers of clustering centers, find the maximum number of clustering centers with significant gradient changes based on the curve graph of the average overlap degree and the number of clustering centers, and uniformly distribute the corresponding clustering center sizes from small to large to the feature maps of the corresponding sizes;

[0021] 5) Take each pixel in the feature map as the center, and construct a corresponding number of prior boxes according to the different sizes of the assigned clustering center sizes;

[0022] 6) Calculate the overlap degree of the target frame of the sample under different sizes and the corresponding prior boxes, and take the prior boxes with an overlap degree greater than or equal to 0.5 as positive samples containing the target, and vice versa as background samples;

[0023] 7) Calculate the classification error of the predicted class of each prior box and the true class in 6), and the offset error of the prior box containing the target compared with the real target frame in the center point and the width and height;

[0024] 8) Optimize the network parameters by the Adam algorithm to minimize the two errors in 7).

[0025] As an optional implementation, data preprocessing can increase flipping and color jitter.

[0026] As an optional implementation, focal loss or ce loss can be used to calculate the classification error.

[0027] As an optional implementation, Euclidean distance, Smooth L1, GIoU, etc. can be used to calculate the offset error.

[0028] As an optional implementation, all prior boxes are used to calculate the classification error.

[0029] As an optional implementation, a certain number of prior boxes are randomly selected to calculate the classification error.

[0030] As an optional implementation, OHEM is used to find prior boxes with difficult classification for calculating the classification error.

[0031] As an optional implementation, the OHEM is used to find the prior box with difficult classification and randomly select part of the remaining easy-to-classify prior boxes to calculate the classification error. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a structure diagram of a backbone network of a lightweight detection network.

[0033] Figure 2 is a structure diagram of a neck network of a lightweight detection network.

[0034] Figure 3 is a structure diagram of a lightweight residual block (LRM).

[0035] Figure 4 is a structure diagram of a prediction network of a lightweight detection network.

[0036] Figure 5 is a flowchart of a network training method in the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0038] The present application provides a lightweight object detection network applied to an embedded device and a training method.

[0039] The network input size is 448x448, and for the image to be detected, the pixel value is converted from an unsigned 8-bit integer to a 32-bit floating point type, divided by 255, normalized to 0-1, and then input into the backbone network of the detection network.

[0040] Figure 1 is a structure diagram of the backbone network, and as shown in the figure, the backbone network includes multiple two-way dense modules. After obtaining the receptive field information of different scales through multiple layers of two-way dense modules, the 2x2 mean pooling output is used to reduce the sampling feature map. For the input image, the backbone network outputs two sizes of feature maps, 56x56 and 28x28.

[0041] As shown in Figure 2 , the feature map output by the backbone network is input into the neck network, and the feature Figure 1 is obtained by fusing the feature of the previous level after upsampling. Figure 2 The two feature maps are input into the corresponding lightweight residual block to enhance the feature expression. Figure 3It is a structure diagram of a lightweight residual block. The first branch is the input feature map, the second branch is a 1×1 convolution and a 3×3 convolution in sequence, and the third branch is a 1×1 convolution, a 3×3 convolution and a 3×3 depthwise separable convolution in sequence. This structure ensures strong feature representation without increasing the amount of computation too much.

[0042] Figure 4 The prediction network's two branches output the probability of each prior box belonging to its class and the offsets of the prior box's center and dimensions, respectively. Combining the prior box's center and dimensions, the coordinates of the target box's top-left and bottom-right corners can be obtained using the following formula:

[0043] Center(x,y)=pred(Δx,Δy)*prior(x,y)+prior(x,y)

[0044] (w,h)=prior(w,h)*e pred(Δw,Δh)

[0045] top_left(x,y)=Center(x,y)-0.5*(w,h)

[0046] bottom_right(x,y)=Center(x,y)+0.5*(w,h)

[0047] In the above formula, pred(Δx,Δy) and pred(Δw,Δh) represent the offsets of the center and width / height of the prior bounding box predicted by the network; prior(x,y) and prior(w,h) represent the initial center coordinates and width / height of the prior bounding box; Center(x,y) and (w,h) represent the center and width / height of the target bounding box obtained after adjustment.

[0048] The network model, which outputs two types of feature maps from the backbone network, is approximately 3.5M in size. When the input image is 448x448, the detection speed is approximately 200FPS on a GTX1080ti and approximately 35FPS on a HiSilicon AI chip Hi3516CV500, which can meet the real-time detection requirements of embedded devices.

[0049] During the training phase, based on the input image size defined by the detection network, the k-means clustering method is used to calculate the cluster centers (width-height pairs) of the target boxes in the training samples, as well as the average overlap between the width and height of the bounding boxes and the width and height of the cluster centers. The formula for calculating the average overlap is as follows:

[0050] ;

[0051] In the above formula, N represents the total number of bounding boxes in the training samples, and n represents the number of cluster centers. js represents the number of bounding boxes contained in the j-th class. i,j c represents the j-th bounding box located in the i-th class. i Let represent the i-th cluster center. As the number of cluster centers increases, the gradient change of the average overlap gradually approaches 0. Under this premise, the cluster center with the highest average overlap is selected as the prior box size.

[0052] In the current implementation, each batch of training images is paired and resized to 448x448, followed by color dithering or horizontal flipping. Since the backbone network outputs feature maps of two sizes, the cluster centers obtained from k-means clustering are divided into two groups based on their area from smallest to largest. The smaller group is used to create prior boxes for the larger feature maps, while the larger group is used to create prior boxes for the smaller feature maps. Then, for each element on all feature maps of all sizes, its row and column index is increased by 0.5 to determine the center position of the prior box. Next, a corresponding number of prior boxes are constructed around this center according to the previously assigned cluster centers. The center and dimensions of the ground truth boxes in the sample data are mapped to the 56x56 and 48x48 feature maps, and the overlap with all corresponding prior boxes on these feature maps is calculated. Using 0.5 as a threshold, prior boxes with an overlap greater than 0.5 with any ground truth box are considered positive samples (containing the target), while those with less overlap are considered negative samples (background). For positive samples, it is necessary to further calculate the offset of the positive sample relative to the true target box in terms of center coordinates and width and height.

[0053] Because the ratio of positive to negative samples in the self-made dataset is unbalanced, the classification error and bias error are calculated using focal loss and Smooth L1 loss, respectively, as shown in the following formulas:

[0054] ;

[0055] In the formula, α is used to balance the ratio of positive samples to background samples; γ is used to distinguish between easily classified and difficult-to-classify prior boxes, so that the error optimization focuses more on the classification results of difficult-to-classify prior boxes; p and p' represent the true category of the prior box and the category of the prior box predicted by the network, respectively.

[0056] ;

[0057] In the formula, x' and x∈{Δx,Δy,Δw,Δh} represent the predicted prior box offset and the actual prior box offset of the detection network, respectively.

[0058] For the calculation of classification error, both positive and negative samples are used; however, the calculation of offset error only uses the offset of positive samples. To further mitigate the negative impact of a significantly larger number of negative samples compared to positive samples on training, only a subset of negative samples are used in the calculation of the total classification error. Furthermore, to prevent the optimization direction from being dominated by easily classified negative samples, for each calculated prior box classification error, all negative sample classification errors are sorted from largest to smallest. For each batch of input images, the negative samples with the largest classification errors are selected from this sequence, with a number three times the number of positive samples. Simultaneously, a portion of the remaining negative samples is randomly selected and used together with the aforementioned difficult-to-classify negative samples in the calculation of the total classification error.

Claims

1. A training method for a lightweight object detection network applied to embedded devices, characterized in that, The network consists of a backbone network, a neck network, and a prediction network; In this network, the backbone network takes the preprocessed image as input and maps it to feature maps of multiple sizes to detect targets of different sizes. The neck network consists of multiple lightweight residual blocks, which input the feature maps into three branches: the first branch is the input feature map, the second branch is a 1×1 convolution followed by a 3×3 convolution, and the third branch is a 1×1 convolution followed by a 3×3 convolution followed by a 3×3 depthwise separable convolution. The outputs of the three branches are concatenated. The input of the neck network is connected to the output of the backbone network, which merges the feature maps of the nearest size in sequence. Each residual block further enhances the feature representation of the corresponding size feature map in subsequent steps. Finally, the merged feature map is output to the subsequent prediction network for result prediction. The backbone network consists of several dual-path dense modules. For the input feature map, it acquires the features of the target at different receptive fields. The size of the feature map output by each output layer of the network is half the size of the previous output feature map. The neck network upsamples the current size feature map through nearest neighbor interpolation and then merges it with the corresponding feature map output by the previous layer through concatenation. The prediction network is a dual-path network, with two branches predicting the target's category and the offset of the target's center and width / height relative to the current pixel position. All prediction results are processed by using non-maximum suppression to filter out bounding boxes that do not contain objects and multiple invalid bounding boxes that surround the same target; The method includes the following steps: S1. Randomly select training sample images of batch_size*2, and stitch them together in pairs. The stitching method is randomly selected between horizontal and vertical stitching. S2. Scale the stitched image to a fixed size and normalize the pixel values ​​to the 0-1 range; S3. Input the preprocessed training sample images into the backbone network to obtain feature maps of multiple sizes; S4. Use the K-means clustering algorithm to obtain the cluster centers of the target box size in the training samples. Calculate the average overlap between the coordinate box width and height and the cluster center width and height based on different numbers of cluster centers. Find the largest number of cluster centers with significant gradient changes based on the curve of the average overlap and the number of cluster centers, and evenly distribute the corresponding cluster center sizes to the feature maps of the corresponding sizes from small to large. S5. Using each pixel in the feature map as the center, construct a corresponding number of prior boxes based on the different sizes of the cluster centers assigned. S6. Calculate the overlap between the target bounding box of the sample and the corresponding prior bounding box at different sizes. Prior bounding boxes with an overlap of greater than or equal to 0.5 with the target bounding box are regarded as positive samples containing the target, and otherwise are background samples. S7. Calculate the classification error between the predicted class of each prior box and the true class in S6, as well as the offset error of the prior box containing the target relative to the true target box in terms of center point and width and height. S8. Optimize the network parameters using the Adam algorithm to minimize the two types of errors in S7.

2. The method according to claim 1, characterized in that, Based on the distribution of the target size and the detection efficiency, the backbone network outputs feature maps of 2-3 sizes.

3. The method according to claim 1, characterized in that, Based on the aspect ratio distribution of the detected target's coordinate frame, the size of the backbone network's input layer is automatically adjusted.

4. The method according to claim 3, characterized in that, Based on the size of the detected target relative to the size of the original image, the size of the first output feature map of the backbone network is set to 1 / 8 or 1 / 4 of the input size.

Images