A real-time target detection method applied to an edge device

Abstract

The application relates to the technical field of computer vision, in particular to a real-time target detection method applied to an edge device; the method comprises the following steps: constructing a target detection model; performing data enhancement processing on a data set; inputting an image after data enhancement into the target detection model; comparing by using a loss function, calculating errors of a prediction result and label data, and optimizing target detection model parameters by using an optimizer and gradient information of the target detection model; converting the trained target detection model into a quantization model, deploying the quantization model on the edge device, and performing target detection on a video stream in a use scene. The Rep-ELAN structure which is more lightweight and can be re-parameterized and the decoupled detection head can greatly increase the accuracy of the detector while almost not losing the calculation speed; the data enhancement processing method can greatly improve the effectiveness of data labels, improve the training efficiency and training effect when the data set labels are insufficient.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a real-time target detection method applied to an edge device. BACKGROUND

[0002] Target detection refers to finding the specific position and size of all targets of interest in an image and other positioning information, and the problem is one of the basic problems in the fields of computer vision and pattern recognition. The method for solving the problem is widely used in vehicle identification analysis, construction monitoring, video monitoring and analysis, face recognition, industrial fields and the like.

[0003] Existing target detection strategies are mainly divided into two categories: one is a two-stage strategy that sacrifices speed for high accuracy, and the other is a one-stage strategy that is slightly less accurate but much faster than the two-stage strategy. With the continuous development of target detection algorithms, target detection algorithms based on the two-stage strategy design still have advantages in application scenarios that pursue extreme accuracy, and in many other applications, the one-stage strategy has gradually replaced the two-stage strategy.

[0004] At present, the one-stage strategy applied to the fields of image segmentation and target detection mainly has two problems:

[0005] (1) A target detector based on the one-stage strategy design with high accuracy can easily achieve real-time effect on a high-performance server, but high accuracy often means a large number of parameters of the model, and more computing resources are consumed during use. Therefore, the current model is difficult to meet the high accuracy and real-time performance at the same time on the edge computing device with limited computing power.

[0006] (2) In the training process of the target detection model, the current mainstream data enhancement method uses a simple mixture of Mosaic enhancement method and Mixup enhancement method. As shown in the accompanying Figure 1 , although the drawbacks are not obvious in the data set with rich data labels, in the case of less data labels, invalid labels are easily generated, which greatly reduces the training effect.

[0007] (3) In most application scenarios, the camera has a certain distance from the target, and the detection object is usually small in the entire image, and the detection accuracy of small targets is usually low.

[0008] (4) The performance indicators pursued by the current mainstream academic research are different from the performance indicators concerned in actual engineering applications, and cannot well serve the industrial field. SUMMARY

[0009] The application aims at the technical defects of the existing one-stage strategy, and provides a real-time target detection method applied to an edge device.

[0010] In order to achieve the above-mentioned purpose, the technical scheme of the application is as follows:

[0011] A real-time target detection method applied to an edge device comprises the following steps:

[0012] A target detection model is constructed, and the target detection model is an end-to-end integrated network sequentially connected by a backbone network and a detection head network, wherein the target detection model is not trained at this time; wherein the backbone network is used for feature extraction processing of an image, and the corresponding feature information of the image is output; the detection head network is used for decoding the extracted feature information, and a prediction result set containing the position information, category and confidence information of all targets is output.

[0013] A data set is selected, and data augmentation processing is performed on the data set to obtain an image after data augmentation and a corresponding label data set;

[0014] The image after data augmentation is input into the constructed target detection model; the prediction result set output by the target detection model and the label data set are compared by using a loss function, the error of the prediction result and the label data is calculated, the optimizer is used to optimize and adjust the parameters of the target detection model through the error and the gradient information of the target detection model, and the step is repeated until the accuracy of the model prediction result is no longer improved;

[0015] The trained target detection model is converted into a quantization model, and the quantization model is deployed on the edge device to detect targets in a video stream in a use scenario.

[0016] Specifically, the backbone network comprises a plurality of Rep-ELAN network structures which can be re-parameterized; along a feature extraction path input by a multi-scale source image, each Rep-ELAN network structure on the feature extraction path comprises Conv convolution, Rep-Conv convolution and DWConv convolution operation, and the input image is subjected to feature extraction to obtain feature information.

[0017] Specifically, the detection head network comprises a plurality of Lite-Efficient-Decoupled-Head detection head structures, which are arranged along a feature decoding path connected to the input of the backbone network, each of the Lite-Efficient-Decoupled-Head detection head structures on the feature decoding path comprises a 1x1 convolution, a 3x3 convolution, a weight and bias operation, and decodes the feature information output by the feature extraction path, and outputs a prediction result set comprising position information, category and confidence information of all targets.

[0018] Specifically, the data set is subjected to data enhancement processing to obtain an image subjected to data enhancement and a corresponding label data set, comprising:

[0019] Setting a label number threshold N, calculating the average number of labels X per picture in the data set, setting a mirror probability P flip , a maximum rotation angle and an HSV enhancement coefficient G hsv ;

[0020] Setting the number of Mosaic data enhancement branches n, so that n satisfies The input data set is subjected to data enhancement processing;

[0021] 4n+1 pictures are extracted from the data set without repetition, 1 picture is reserved, and the remaining pictures are divided into groups of 4 pictures each, and the pictures in each group are spliced based on Mosaic data enhancement to obtain n pictures, and then the n pictures are subjected to mirror, rotation, HSV enhancement and scale transformation operations according to the parameters set in S2A;

[0022] The reserved picture is subjected to small-amplitude data enhancement;

[0023] The n pictures generated and the 1 picture subjected to small-amplitude data enhancement are subjected to Mixup data enhancement mixing operation to obtain a final image, and the image and the corresponding label data subjected to transformation are input into the target detection model.

[0024] Specifically, the output prediction result and the label data are compared using a loss function, and the error between the prediction result and the label data is calculated, comprising:

[0025] Input and preset parameters, wherein the input parameters include a prediction result set P0 and a label result set T0, and the preset parameters include a target frame loss weight W box , a foreground loss weight W obj and a category loss weight W cls ;

[0026] For each target frame ti in the label result set T0, the result pi with the largest coincidence degree in the prediction result set P0 is traversed, and after matching, the two are put into the set matched from the prediction result set P0 and the label result set T0 respectively into the set matched all the prediction result set P of the whole picture and the all the label result set T of the whole picture;

[0027] When all the results in the label result set T0 have been matched, the remaining unmatched results in the prediction result set P0 are discarded;

[0028] The matched all the prediction result set P of the whole picture and the all the label result set T of the whole picture are used to calculate the prediction frame position loss, the foreground loss and the category loss by using the loss function;

[0029] The calculated prediction frame position loss, foreground loss and category loss are weighted and summed to obtain the final loss.

[0030] Specifically, in the comparison calculation process using the loss function, for the loss function of a single preset result, the formula (1) is as follows:

[0031] hrl(p,t)=[4(1-p) 2 r+(1-r)]tlog(p)+[12p 2 r+(1-r)](1-t)log(1-p) (1)

[0032] In the formula, p represents the prediction result, t represents the target result, and r is a random number between 0 and 1;

[0033] According to the above formula (1), the loss function of each result is summed and averaged to obtain the loss function of the whole picture, and the formula (2) is as follows:

[0034]

[0035] In the formula, P represents the all the prediction result set of the whole picture, T represents the all the label result set of the whole picture, and K represents the number of matched prediction results and label results of the whole picture;

[0036] The prediction frame position loss is calculated based on the bounding box regression loss function sIOU, and the calculation formula (3) of the prediction frame position loss is as follows:

[0037] Loss box =sIOU(P box ,T box ) (3)

[0038] In the formula, Loss box represents the prediction frame position loss, P boxT represents a target frame position set in all matched prediction results P of the whole picture box T represents a target frame position set in all matched label results T of the whole picture

[0039] The foreground loss and the category loss are calculated based on formula (2), and the calculation formula (4) of the foreground loss is as follows:

[0040] Loss obj =HRL(P obj ,T obj ) (4)

[0041] In the formula, Loss obj represents the foreground loss, P obj represents a foreground probability set in all matched prediction results P of the whole picture, and T obj represents a foreground probability set in all matched label results T of the whole picture.

[0042] The calculation formula (5) of the category loss is as follows:

[0043] Loss cls =HRL(P cls ,T cls ) (5)

[0044] In the formula, Loss cls represents the category loss, P cls represents a category probability set in all matched prediction results P of the whole picture, and T cls represents a foreground probability set in all matched label results T of the whole picture.

[0045] Specifically, the optimizer optimizes and adjusts the parameters of the target detection model through the error and the gradient information of the target detection model, and the step is repeated until the accuracy of the model prediction result no longer improves, including:

[0046] Each layer calculation module of the pre-trained target detection model is traversed, and it is judged whether the module is a reparameterizable module, and if so, reparameterization is performed to reduce the model parameters.

[0047] Specifically, the trained target detection model is converted into a quantization model, and the quantization model is deployed on an edge device to perform target detection on a video stream in a use scenario, including:

[0048] The model quantization tool is converted into a quantization model Model for deployment;

[0049] A video stream is obtained, and a preset image input size, a confidence threshold and an overlap threshold are obtained;

[0050] A current frame image is extracted from a video stream, and the current frame image is input into a quantization model Model after pre-processing, and a plurality of target information is obtained after inference by the quantization model Model;

[0051] All targets with a confidence less than a confidence threshold are filtered out, and the target frame overlap between the remaining targets is determined, and if the overlap is higher than an overlap threshold, it is determined that the two target frames predict the same target, and the target with lower confidence is filtered out;

[0052] The remaining target results are used as the final results and displayed on the image, and the above steps are repeated until the video stream ends.

[0053] The beneficial effects of the present application are:

[0054] The real-time target detection method applied to the edge device can greatly increase the accuracy of the detector while almost not losing the calculation speed through the more lightweight and re-parameterizable Rep-ELAN structure and decoupled detection head; meanwhile, a more powerful data enhancement method is provided, which can greatly improve the effectiveness of the data label, improve the training efficiency and training effect when the data set label is insufficient; in addition, the loss function used in the final deployment stage of the model training can effectively improve the detection accuracy of small targets by the detector. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 It is a schematic diagram of the simple mixing of Mosaic and Mixup in the prior art in the background art of the present application;

[0056] Figure 2 It is a main structure diagram of the target detection model in the embodiment of the present application;

[0057] Figure 3 It is a comparison diagram of the original ELAN structure and the re-parameterizable Rep-ELAN structure in the embodiment of the present application, wherein, Figure 3 (a) is the original ELAN structure, Figure 3 (b) is the re-parameterizable Rep-ELAN structure;

[0058] Figure 4 It is a comparison diagram of the conventional 3x3 convolution layer structure and the RepConv convolution layer structure in the embodiment of the present application, wherein, Figure 4 (a) is the conventional 3x3 convolution layer structure, Figure 4 (b) is the RepConv convolution layer structure;

[0059] Figure 5 It is a comparison diagram of three decoupled head structures in the embodiment of the present application, wherein, Figure 5(a) is a primary decoupled head Decoupled-Head, Figure 5 (b) is a YOLOv6 decoupled head Efficient-Decoupled-Head, Figure 5 (c) is a Lite-Efficient-Decoupled-Head;

[0060] Figure 6 is a schematic diagram of the data enhancement structure in the embodiment of the application.

[0061] Figure 7 is a small target detection effect diagram in the embodiment of the application.

[0062] Figure 8 is a flowchart of the real-time target detection method applied to the edge device in the embodiment of the application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0064] EMBODIMENT

[0065] A real-time target detection method applied to an edge device, which is realized based on a real-time target detector applied to an edge computing device of Nvidia Jetson AGX Xavier, the CPU of which is Nvidia Carmel ARMv8.2 (8-core) @ 2.26 GHz, the GPU of which is 512-Core Volta @ 1377 MHz + 64 TensorCores, the memory of which is 32 GB 256-bit LPDDR4x @ 2133 MHz | 137 GB / s, the storage space of which is 32 GB eMMC 5.1 + M.2 SSD 512 GB, the software environment of which is Ubuntu 18.04, Python 3.6.9, TensorRT 7.1.3.0, and the power mode of which is MAXN.

[0066] In the embodiment, the real-time target detector is divided into a model design stage, a training stage and a deployment stage; wherein, referring to the drawings Figure 2 The model design stage mainly includes a backbone network and a detection head network, the training stage includes data enhancement processing and a loss function, and the deployment stage includes deployment and inference.

[0067] In the embodiment, referring to the drawingsFigure 8 A real-time target detection method applied to an edge device, comprising:

[0068] Step 1: constructing a target detection model, the target detection model being an end-to-end integrated network sequentially connected by a backbone network and a detection head network, the target detection model not being trained at this time.

[0069] Wherein, the backbone network is used for feature extraction processing of the image, and outputs corresponding feature information of the image; the detection head network is used for decoding the extracted feature information, and outputs a prediction result set containing position information, category and confidence information of all targets.

[0070] In this embodiment, the backbone network includes a plurality of Rep-ELAN network structures which can be re-parameterized; along the feature extraction path of the multi-scale source image input, each Rep-ELAN network structure on the feature extraction path includes Conv convolution, Rep-Conv convolution and DWConv convolution operation, and the input image is subjected to feature extraction to obtain feature information.

[0071] In this embodiment, in the backbone network, a large number of Rep-ELAN network structures which can be re-parameterized are adopted, as shown in FIG. 3(a), the original ELAN structure has advantages of small parameter quantity, more data interaction between layers, fast running speed and the like compared with the traditional CSPN structure, and therefore, the present application further changes part of the neural network layer and designs the Rep-ELAN structure which can be re-parameterized, as shown in FIG. 3(b). In this structure, the data is divided into four branches for three times, each branch passes through different number of convolution layers, and therefore, after final splicing and merging (Concat), the data contains feature information from multiple different levels of convolution layers, and the feature information elements are enriched.

[0072] It should be understood that when a layer has data from different levels as input data of the layer, the layer is called a data intersection layer; and when the output data of a layer is used as input of multiple different levels of network layers, the layer is called a data shunting layer. Since the two kinds of network layers have multi-scale feature information, the linear parallel convolution layer RepConv which can be re-parameterized is used instead of the ordinary convolution layer Conv.

[0073] In this embodiment, as shown in FIG. 3(a), a conventional 3x3 convolution layer structure is used, as shown in FIG. 3(b), the Rep-ELAN structure which can be re-parameterized is used. Figure 4 (a) is a conventional 3x3 convolution layer structure, and FIG. 3(b) is the Rep-ELAN structure which can be re-parameterized. Figure 4(b) shows the convolution layer structure of RepConv. It can be seen that RepConv adds a 1×1 convolution layer and a branch BN with only a batch normalization layer. This allows for more parameters during training, making model training easier and thus training a more accurate model. At the same time, in the deployment process after training, since there is no nonlinear calculation in the three branches, the attached model can be obtained through the reparameterization method. Figure 4 The RepConv convolution layer structure of (b) can be transformed into the attached Figure 4 (a) Same structure, same number of parameters, and same computational speed. Furthermore, in the lightweight model, we also replaced two conventional 3×3 convolutional layers with depthwise separable convolution (DepthWise Conv), commonly used in lightweight networks (such as MobileNet and ShuffleNet), to reduce computational cost.

[0074] In this embodiment, the detection head network decodes the extracted features and generates information such as target location, category, and confidence. It includes B Lite-Efficient-Decoupled-Head detection head structures, which are deployed along a feature decoding path that integrates the backbone network connection input. Each Lite-Efficient-Decoupled-Head detection head structure on the feature decoding path includes 1×1 convolution, 3×3 convolution, weight and bias operations, decodes the feature information output by the feature extraction path, and generates label data.

[0075] In this embodiment, in the detection head network, the present invention designs a more lightweight structure compared to other decoupled detection heads; among them, compared with the first generation decoupled-head (such as the FCOS detection head structure designed in YOLOX) Figure 5 (a)), the decoupling head designed in YOLOv6 (e.g. Figure 5 (b) Efficient-Decoupled-Head reduces one 3×3 convolution layer and increases the number of feature channels to ensure accuracy, but this also increases the amount of calculation; Figure 5 (c) The Lite-Efficient-Decoupled-Head we designed reduces the number of convolutional layers while maintaining a small number of feature channels. To ensure accuracy, we add additional weight layers and bias layers. These two layers provide additional training parameters during training. After training, the parameters of these two additional layers are merged into the middle convolutional layer through the reparameterization method. At the same time, the foreground prediction branch is coupled with the category prediction branch to improve the model parallelism, thereby achieving faster running speed.

[0076] In this embodiment, the prediction result set includes position information (target center point coordinates (cx, cy), target length and width (w, h)), target confidence (between 0% and 100%, representing the possibility of the existence of the target at the position), and the confidence of each target type class_id_confidence. For example, there are 80 classes in the COCO dataset, and the class_id ranges from 0 to 79. Therefore, each target information can be represented by a 5+n-dimensional vector, where n represents the total number of categories. For example: [cx, cy, w, h, confidence, class_1_confidence, …, class_n_confidence]; the highest category in the category confidence is taken as the category of the target, and the category ID (class_id) is recorded. The confidence is multiplied by the target confidence to obtain the final confidence conf, which can be converted into a 6-dimensional vector [cx, cy, w, h, conf, class_id]. Similarly, the labels loaded from the dataset are also represented in this way.

[0077] Step 2: Select the dataset, and perform data augmentation on the dataset to obtain images after data augmentation and the corresponding label data set.

[0078] Wherein, the reference Figure 6 , the data set is subjected to data augmentation processing to obtain images after data augmentation and corresponding label data, comprising:

[0079] S1: Set the label number threshold N, count the average number of labels X per image in the data set used for this training, and set the mirror probability P flip , the maximum rotation angle and the HSV enhancement coefficient G hsv parameters;

[0080] S2: Set the number of Mosaic data augmentation branches n in this training, so that n satisfies Load the dataset information and wait for the start of this training;

[0081] S3: Extract 4n+1 images from the dataset without repetition, keep 1 image, and perform Mosaic data augmentation on the remaining 4 images in each group, a total of n images, and then perform mirror, rotation, HSV enhancement and scale transformation operations on the n images according to the parameters set in S2A;

[0082] S4: Perform small-scale data augmentation on the image kept in S3 to ensure that the image does not change too much after data augmentation, which is used to ensure the stability of the training effect.

[0083] Among them, generally speaking, according to the different data sets, the parameters for controlling small amplitude data enhancement (such as the mirror, rotation, HSV enhancement and scale transformation operation mentioned above, etc.) are between one tenth and one third of the normal amplitude data enhancement.

[0084] S5: The n pictures generated in S3 and the 1 picture after the enhancement processing in S4 are subjected to a Mixup data enhancement mixing operation to obtain a final image, the image and the corresponding label data after transformation are input into the target detection model, and after waiting for the next training iteration to start, steps S3-S5 are repeated.

[0085] Step 3: input the data-enhanced image into the constructed target detection model; compare the prediction result set output by the target detection model and the label data set using a loss function, calculate the error of the prediction result and the label data, and use an optimizer to optimize and adjust the parameters of the target detection model through the error and the gradient information of the target detection model. Repeat this step until the accuracy of the model prediction result no longer improves.

[0086] Among them, the output prediction result and the label data are compared using a loss function, and the error of the prediction result and the label data is calculated, including:

[0087] Input and preset parameters, wherein the input parameters include a prediction result set P0 and a label result set T0, and the preset parameters include a target frame loss weight W box , a foreground loss weight W obj , and a category loss weight W cls .

[0088] For each target frame ti in the label result set T0, traverse the result pi with the highest coincidence degree in the prediction result set P0, and after matching, put them into the set-matched whole-image prediction result set P and the set-matched whole-image label result set T from the prediction result set P0 and the label result set T0, respectively.

[0089] Because the prediction result set P0 is large in quantity and widely distributed in all positions in the image, the results in the label result set T0 can be matched. When all the results in the label result set T0 have been matched, discard the remaining unmatched results in the prediction result set P0.

[0090] Calculate the prediction frame position loss, the foreground loss, and the category loss using the loss function for the matched whole-image prediction result set P and the whole-image label result set T.

[0091] Weighted sum the calculated prediction frame position loss, the foreground loss, and the category loss to obtain the final loss.

[0092] In the embodiment, the loss function of the loss function calculation module mainly includes three parts, foreground probability loss, category probability loss and predicted frame position loss.

[0093] As mentioned above, each target is represented by a 5+n-dimensional vector [cx, cy, w, h, confidence, class_1_confidence, …, class_n_confidence]; for the predicted frame loss, the predicted [cx, cy, w, h] is compared with the predicted frame in the actual label, the corresponding coincidence is calculated, the higher the coincidence, the smaller the loss, and the more accurate the prediction. In this way, the commonly used sIOU loss function is adopted.

[0094] The predicted frame position loss is calculated based on the bounding box regression loss function sIOU, and the calculation formula (3) of the predicted frame position loss is as follows:

[0095] Loss box = sIOU (P box , T box ) (3)

[0096] In the formula, Loss box represents the predicted frame position loss, P box represents the target frame position set in all matched predicted results P in the whole picture, and T box represents the target frame position set in all matched label results T in the whole picture.

[0097] For the foreground probability loss and the category probability loss, confidence or class_i_confidence (0≤i<n) is taken as the prediction p, and whether the target frame is a target (target is 1, and background is 0) or whether the target frame is this category (1 for yes, and 0 for no) is taken as the result t. The difference between p and t is calculated by the loss function, and the calculation result is the foreground probability loss and the category probability loss.

[0098] The two losses are usually calculated by using the cross-entropy loss function, and the present application designs the following random hybrid loss function (Hybrid-Random Loss).

[0099] For the loss function of a single preset result, the formula (1) is as follows:

[0100] hrl (p, t) = [4 (1-p) 2 r+(1-r)]tlog (p) + [12p 2 r+(1-r)] (1-t) log (1-p) (1)

[0101] Where p represents the prediction result, t represents the target result, and r is a random number between 0 and 1.

[0102] According to the above formula (1), the loss function of each result is summed and averaged to obtain the loss function of the entire image, and the formula (2) is as follows:

[0103]

[0104] Where P represents the set of all prediction results of the entire image, T represents the set of all label results of the entire image, and K represents the number of matched prediction results and label results of the entire image.

[0105] Based on formula (2), the foreground loss and the class loss are calculated, and the calculation formula (4) of the foreground loss is as follows:

[0106] Loss obj = HRL (P obj , T obj ) (4)

[0107] Where Loss obj represents the foreground loss, P obj represents the set of foreground probabilities in all matched prediction results P of the entire image, and T obj represents the set of foreground probabilities in all matched label results T of the entire image.

[0108] The calculation formula (5) of the class loss is as follows:

[0109] Loss cls = HRL (P cls , T cls ) (5)

[0110] Where Loss cls represents the class loss, P cls represents the set of class probabilities in all matched prediction results P of the entire image, and T cls represents the set of foreground probabilities in all matched label results T of the entire image.

[0111] The calculated prediction frame position loss, foreground loss and class loss are weighted and summed to obtain the final loss, and the calculation formula (5) is as follows:

[0112] Loss total = W box Loss box +W obj Loss obj +W cls Loss cls .

[0113] In this embodiment, the optimizer is used to optimize and adjust the target detection model parameters through the error and gradient information of the target detection model, and this step is repeated until the accuracy of the model prediction result no longer improves, including:

[0114] Each layer calculation module of the pre-trained target detection model is traversed to determine whether the module is a reparameterizable module, and if so, reparameterization is performed to reduce the model parameters, and this step is repeated until the accuracy of the model prediction result no longer improves.

[0115] Step 4: converting the trained target detection model into a quantization model and deploying the quantization model on an edge device to perform target detection on a video stream in a use scenario.

[0116] Due to the use of a large number of reparameterizable structures, the application reparameterizes the model when deployed, reduces the parameters of the model without losing the accuracy of the model, and greatly improves the parallelism of the model structure, thereby improving the running speed of the model on the edge computing device without any cost.

[0117] The trained target detection model is converted into a quantization model, and the quantization model is deployed on an edge device to perform target detection on a video stream in a use scenario, including:

[0118] S1: converting into a quantization model Model for deployment through a model quantization tool.

[0119] Further, the model quantization tool, for example: TensorRT of Nvidia Company, OpenVINO of Intel, TFLite of Google, etc.

[0120] S2: obtaining a video stream, presetting an image input size, a confidence threshold and an overlap threshold.

[0121] Further, the video stream Video Stream is obtained, the image input size size is determined, the confidence threshold confidence_threshold is set, and the overlap threshold iou_threshold is set.

[0122] S3: extracting a current frame image from the video stream, inputting the current frame image into the quantization model Model after preprocessing, and obtaining multiple target information after inference of the quantization model Model.

[0123] Further, the current frame image is extracted from the video stream, the image is input into the model after preprocessing (scaled to the same size as size), and multiple target information is obtained after inference of the model, and each target information is represented by a 6-dimensional vector, such as: [cx, cy, w, h, confidence, class_id].

[0124] S4: filter out all targets with confidence < confidence threshold, and calculate the target frame overlap between the remaining targets, if the overlap is higher than the overlap threshold, it is considered that the two target frames predict the same target, and the result with lower confidence is filtered out.

[0125] Further, filter out all targets with confidence < confidence threshold, and calculate the target frame overlap between the remaining targets, if the overlap is higher than the overlap threshold iou_threshold, it is considered that the two target frames predict the same target, and the result with lower confidence is filtered out.

[0126] S5: take the remaining target results as the final results and display them on the image, repeat the above steps until the video stream ends.

[0127] Further, take the remaining target results as the final results and display them on the image, repeat steps S3-S5 until the video stream ends.

[0128] Test results verification: the most advanced model in the current lightweight model is YOLOv6, and the present application compares it on the target detection general dataset MS COCO2017. AP 50 The average accuracy (Average Precision) of the model when the target frame overlap threshold is 50%, which is one of the commonly used indicators in the industry. At the same time, the present application also compares the parameter quantity of the model and the floating point operation number (FLOPs, which can be used to measure the amount of calculation required by the model) when the input image size is 640x640. The specific test results are as follows in Table 1.

[0129] Table 1 test results

[0130]

[0131] As can be seen from Table 1 above, the target detection method disclosed in the present application can achieve higher detection accuracy with fewer model parameters and smaller model calculation amount.

[0132] In addition, through experiments, it is found that the improvement of the accuracy of the model disclosed in the present application is mainly due to the significant improvement in small target detection effect. Using the unmanned aerial vehicle shooting dataset VisDrone2019-DET, retraining is carried out, and the method steps described in the present application are used to detect vehicles and personnel on the road (i.e. small targets), and the small target detection effect is as shown in the following figures. Figure 7 As can be seen from the figures, the vehicles and personnel on the road can be accurately identified.

[0133] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other presenters can view. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be appreciated that the language and combinations of words are intended to be interpreted broadly and liberally to include other related structures and methodologies. It is intended that the present application encompass all technologically equivalent structures and methodologies that might fall within the scope of the claims. It is to be understood that any reference to prior art contained herein is not, and should not be taken as, an acknowledgment or any form of suggestion that such prior art forms part of the common general knowledge in the art in any country in the world, or that the patentees should be taken to admit that such prior art can be part of the prior art, or that the patentees should be taken to admit that such prior art would be obvious to any person skilled in the art in the absence of an enabling disclosure.

[0134] The above description is only some embodiments of the present application, and the common knowledge of the specific structure and characteristics in the scheme is not described too much. The ordinary skilled person in the art knows all the ordinary technical knowledge in the field of the present application before the filing date or the priority date, can know all the prior art in the field, and has the ability to apply the conventional experimental means before that date. The ordinary skilled person in the art can perfect and implement the present scheme under the guidance of the present application, combined with their own ability. Some typical known structures or known methods should not be an obstacle for the ordinary skilled person in the art to implement the present application. It should be noted that for those skilled in the art, without departing from the structure of the present application, a number of modifications and improvements can be made, which should be considered as the protection scope of the present application. These will not affect the effect and practicality of the present application. The protection scope of the present application should be subject to the content of its claims, and the specific embodiments in the specification can be used to explain the content of the claims.

Claims

1. A real-time target detection method applied to edge devices, characterized by: include: Construct an object detection model, which is an end-to-end integrated network consisting of a backbone network and a detection head network connected sequentially. At this point, the object detection model is untrained. The backbone network is used to extract features from the image and output the corresponding feature information of the image. The detection head network decodes the extracted feature information and outputs a set of prediction results containing the location information, category, and confidence information of all objects. Select a data set, perform data enhancement on the data set, and obtain the data-enhanced images and the corresponding label data set; The data-augmented image is input into the constructed target detection model. The prediction result set output by the target detection model and the label data set are compared using the loss function. The error between the prediction result and the label data is calculated. The optimizer is used to optimize and adjust the target detection model parameters based on the error and the gradient information of the target detection model. This step is repeated until the accuracy of the model prediction result no longer increases. Convert the trained object detection model into a quantized model and deploy the quantized model on edge devices to perform object detection on video streams in the usage scenario. The backbone network includes multiple reparameterizable Rep-ELAN network structures, which are deployed along the feature extraction path of the multi-scale source image input. Each Rep-ELAN network structure on the feature extraction path includes Conv convolution, Rep-Conv convolution, and DWConv convolution operations to extract features from the input image and obtain feature information. The detection head network includes multiple Lite-Efficient-Decoupled-Head detection head structures, which are deployed along a feature decoding path that integrates the backbone network connection input. Each Lite-Efficient-Decoupled-Head detection head structure on the feature decoding path contains 1×1 convolution, 3×3 convolution, weight and bias operations, decodes the feature information output by the feature extraction path, and outputs a set of prediction results containing the location information, category and confidence information of all targets.

2. The real-time target detection method applied to edge devices according to claim 1, characterized in that: The data set is subjected to data enhancement processing to obtain a data-enhanced image and a corresponding label data set, including: Set the label number threshold N, calculate the average number of labels per picture in the dataset X, and set the mirror probability P flip , maximum rotation angle and HSV enhancement factor G hsv parameter; Set the number of Mosaic data enhancement branches n so that n satisfies , input data set for data enhancement processing; Extract 4 × n + 1 images from the dataset without duplication, retain 1 image, and group the remaining 4 images into a group and perform mosaicking based on Mosaic data augmentation to obtain n images. Then, perform mirroring, rotation, HSV enhancement, and scale transformation operations on these n images according to the parameters set in S2A. Perform minor data augmentation on the retained images; The n generated images and one image that has undergone a small amount of data enhancement are mixed to obtain the final image, which is then input into the target detection model along with the corresponding label data after the transformation.

3. The real-time target detection method applied to edge devices according to claim 1, characterized in that: The output prediction result and label data are compared using a loss function to calculate the error between the prediction result and label data, including: Input and preset parameters, where the input parameters include the prediction result set P0 and the label result set T0, and the preset parameters include the target box loss weight W box , foreground loss weight W obj and class loss weight W cls ; For each target box ti in the label result set T0, traverse the result pi with the largest overlap in the prediction result set P0. After the two are matched, they are put from the prediction result set P0 and the label result set T0 into the set matching all prediction result sets P of the entire image and all label result sets T of the entire image; When all results in the label result set T0 have been matched, the remaining unmatched results in the prediction result set P0 are discarded; The loss function is used to calculate the prediction box position loss, foreground loss and category loss for all prediction result sets P of the matched images and all label result sets T of the entire image; The calculated prediction box position loss, foreground loss and category loss are weighted summed to obtain the final loss.

4. The real-time target detection method applied to edge devices according to claim 3, characterized in that: In the process of comparative calculation using the loss function, the loss function for a single preset result is as follows: hrl(p,t)=[4(1-p) 2 r+(1-r)]tlog(p)+[12p 2 r+(1-r)](1-t)log(1-p) (1) In the formula, p represents the predicted result, t represents the target result, and r is a random number between 0 and 1; According to the above formula (1), the loss function of each result is summed and averaged to obtain the loss function of the entire image, and its formula (2) is as follows: ， Where P represents the set of all prediction results for the entire image, T represents the set of all label results for the entire image, and K represents the number of matches between the prediction results and label results for the entire image. The prediction box position loss is calculated based on the bounding box regression loss function sIOU. The calculation formula (3) of the prediction box position loss is as follows: Loss box =sIOU(P box ,T box )(3), Where, Loss box represents the prediction box position loss, P box Represents the set of target box positions in all matched prediction results P of the entire image, T box Represents the set of target box positions in all matched label results T of the entire image, Based on formula (2), the foreground loss and category loss are calculated. The foreground loss is calculated using formula (4) as follows: Loss obj =HRL(P obj ,T obj )(4), Where, Loss obj represents the foreground loss, P obj Represents the foreground probability set of all matched prediction results P for the entire image, T obj Represents the foreground probability set in the label result T that has been matched for the entire image, The calculation formula (5) of the category loss is as follows: Loss cls =HRL(P cls ,T cls )(5), Where, Loss cls represents the category loss, P cls Represents the set of category probabilities in all matched prediction results P for the entire image, T cls Represents the foreground probability set of all matched label results T for the entire image.

5. The real-time target detection method applied to edge devices according to claim 1, characterized in that: The optimizer is used to optimize and adjust the parameters of the target detection model using the error and the gradient information of the target detection model, and this step is repeated until the accuracy of the model prediction result no longer increases, including: Traverse each layer of the computational module of the pre-trained target detection model to determine whether the module is a reparameterizable module. If so, reparameterize it to reduce the model parameters.

6. The real-time target detection method applied to edge devices according to claim 1, characterized in that: The trained object detection model is converted into a quantized model and deployed on an edge device to perform object detection on a video stream in a usage scenario, including: Convert to a quantized model for deployment through a model quantization tool; Get the video stream, preset the image input size, confidence threshold, and overlap threshold; Extract the current frame image from the video stream, input the current frame image into the quantization model after preprocessing, and obtain multiple target information after inference by the quantization model; Filter out all targets with confidence less than the confidence threshold, and determine the overlap between the target frames of the remaining targets. If the overlap is higher than the overlap threshold, it is determined that the two target frames predict the same target, and the result with lower confidence is filtered out. The remaining target results are taken as the final results and displayed on the image, and the above steps are repeated until the video stream ends.

Images