A lightweight multi-task video stream real-time inference method and system

Abstract

The present invention proposes a lightweight multi-task video stream real-time inference method and system, which adopts a target detection model with higher inference efficiency as the detection benchmark model, and adds a drivable area segmentation head network and a multi-object tracking head network on its basis to complete the drivable area segmentation task and the multi-object tracking task respectively. By sharing the backbone network and the feature pyramid network, the redundancy of the parameters and the amount of calculation is reduced. The end-to-end structured iterative pruning algorithm of cascaded knowledge distillation is used to further reduce the computational complexity of the model, effectively improving the acceleration ratio of the model on edge chip hardware. Finally, the model can achieve real-time inference at the edge while ensuring accuracy.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and automatic assisted driving, and in particular to a lightweight multi-task video stream real-time reasoning method. Background Art

[0002] With the advancement of AI technology and the proliferation of smart devices, more and more devices are requiring AI without access to cloud platforms, making edge AI increasingly important. In particular, in applications such as autonomous robots or smart cars equipped with computer vision algorithms, data transmission delays can be devastating. Self-driving cars cannot tolerate delays when detecting people or obstacles on the road. Because fast response times are so crucial, edge AI systems must be employed, enabling real-time image analysis and classification without relying on cloud computing connectivity. Autonomous driving relies primarily on various sensors to perceive the vehicle's surroundings, using computer vision processing technologies such as object detection, segmentation, recognition, and tracking to assess safety.

[0003] Currently, the most common methods for handling road object detection, drivable area segmentation, and multi-object tracking tasks are to use three different neural network models. For example, the single-stage target detection model and the two-stage target detection model are specifically designed to handle target detection tasks, while UNet and PSPNet are used to handle semantic segmentation tasks. This method of decoupling multi-task models and using multiple neural network models for separate processing is relatively easy to implement, but multiple models inevitably lead to repeated extraction of image data features and difficulties in model training. In addition, the total number of parameters and the total amount of computation required for multiple models are huge. Edge chips are limited by computing power resources and run slowly, making it difficult to strike a balance between accuracy and latency.

[0004] Therefore, how to use a single model to solve multiple perception tasks while balancing accuracy and latency is an urgent problem that needs to be solved. Currently, there are several representative approaches.

[0005] This algorithm uses an encoder-decoder architecture with a shared encoder and three independent decoders for classification, object detection, and semantic segmentation. The algorithm achieves good results on these tasks and also achieves good results on the KITTI drivable area segmentation task. However, classification is not as important as multi-object tracking when controlling a vehicle.

[0006] This approach combines traffic object detection, drivable area segmentation, and lane detection, and proposes using context tensors to fuse feature maps between decoders for information sharing. While this approach demonstrates strong accuracy, it suffers from high latency and cannot achieve real-time inference.

[0007] By summarizing existing research, we found that the above-mentioned multi-task learning technologies have the following shortcomings: (1) They cannot well meet the requirements of vehicle control in autonomous driving scenarios. (2) They cannot achieve a good balance between accuracy and latency, that is, they cannot achieve real-time reasoning while ensuring accuracy. Summary of the Invention

[0008] In response to the above problems, the target detection model YOLOv5 with higher reasoning efficiency is used as the detection benchmark model. On its basis, the drivable area segmentation head network and the multi-object tracking head network are added to complete the drivable area segmentation task and the multi-object tracking task respectively. The redundancy of the parameters and the amount of calculation is reduced by sharing the backbone network and the feature pyramid network. The end-to-end structured iterative pruning algorithm of cascaded knowledge distillation is used to further reduce the amount of calculation of the model, effectively improving the acceleration ratio of the model on the edge chip hardware. Finally, the model can ensure accuracy while achieving real-time reasoning at the edge. The application scenario of the present invention is automatic assisted driving. The present invention can complete the perception tasks of three assisted driving scenarios: road object detection, drivable area segmentation and road multi-object tracking. Predict or avoid tasks related to planning and control belonging to assisted driving scenarios. It is necessary to design a planning and control algorithm based on our perception results.

[0009] Specifically, the present invention proposes a lightweight multi-task real-time inference method for video streams, which includes:

[0010] Step 1: Obtain a video dataset containing driving scenes. The video dataset includes multiple video streams. The video frames in the video streams have label files. The label files include target anchor box category labels, driving area labels, and object tracking labels.

[0011] Step 2: Obtain the original target detection model with a backbone network and a feature pyramid network. Add two branch networks to the target detection head network of the feature pyramid network, which serve as the driving area segmentation head network and the object tracking head network respectively, to obtain a multi-task detection model.

[0012] Step 3: Input the video dataset as training data into the multi-task detection model, and construct a loss function based on the target anchor box category, driving area, and object tracking results output by the multi-task detection model and the label file to train the multi-task detection model to obtain a teacher model. The teacher model is subjected to end-to-end structured iterative pruning processing of cascaded knowledge distillation to obtain a student model.

[0013] Step 4: Input the real-time driving scene video stream to be identified, which is collected while the vehicle is driving, into the student model to obtain an inference result including target detection results, drivable area, and target tracking results; based on the inference result, control the vehicle to perform control tasks related to assisted driving.

[0014] The lightweight multi-task video stream real-time inference method, wherein the target anchor frame category label includes category, anchor frame center point coordinates, anchor frame width and anchor frame height; the driving area label includes drivable area and non-drivable area; the object tracking label includes the frame position of the video stream, anchor frame number, anchor frame center point coordinates, anchor frame width, anchor frame height, and confidence that the anchor frame contains an object.

[0015] The lightweight multi-task video stream real-time inference method, wherein the training of the multi-task detection model in step 3 specifically includes:

[0016] The multi-task detection model extracts features from video frames through the backbone network to obtain feature maps of different scales. The feature maps of different scales are fused through the feature pyramid network. The output is decoded by the head network of different tasks and the loss function binary-cross-entropy and CIOU loss is calculated with the label. The binary-cross-entropy loss is specifically:

[0017] L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )]

[0018] The formula of the CIOU Loss is specifically:

[0019]

[0020]

[0021]

[0022]

[0023] Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the prediction box A and the label box B, b gt is the center point of the label box, b is the center point of the prediction box, ρ 2 (b,b gt) is the Euclidean distance between the center points of the prediction box A and the annotation box B, c is the length of the diagonal of the minimum circumscribed rectangle between the prediction box A and the annotation box B, α is the weight, and is the distance between the aspect ratio of the prediction box A and the annotation box B; A is the prediction box, B is the annotation box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box;

[0024] When the total loss value reaches a preset standard, a target model is obtained. The target model is used to obtain a video frame to be identified and output a detection target label and a drivable area segmentation label identified by the target model, wherein the detection target label is used as the input of the object tracking head network; the object tracking head network uses Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target; wherein the Mahalanobis distance correlation is:

[0025]

[0026] The cosine distance association measure:

[0027]

[0028] The loss function combining Mahalanobis distance and cosine association metric is as follows:

[0029] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0030] Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates d j with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric;

[0031] When the loss c i,j When the preset standards are reached, the teacher network is obtained.

[0032] In the lightweight multi-task video stream real-time inference method, the pruning process in step 3 includes:

[0033] For the training process of the multi-task detection model, the loss function L BCE (y' i ,y i) adds a penalty factor to constrain the weight of the convolution layer and the scaling factor of the BN layer. The constraint term λR(w) uses the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling factor are as follows:

[0034]

[0035] R(w)=|α|

[0036] After training the end-to-end multi-task detection model until the loss function converges, the absolute values ​​of the filters are summed. Then, the importance function is calculated by combining the scaling coefficient and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows:

[0037]

[0038] m i =α i *E i

[0039] θ=sort p (M)

[0040] The network before pruning is used as the teacher network, and the network after pruning is used as the student network.

[0041] The present invention also proposes a lightweight multi-task video stream real-time reasoning system, which includes:

[0042] Module 1: Obtain a video dataset containing driving scenes, which includes multiple video streams. The video frames in the video streams have label files, which include target anchor box category labels, driving area labels, and object tracking labels;

[0043] Module 2: Obtain the original target detection model with a backbone network and a feature pyramid network. Add two branch networks to the target detection head network of the feature pyramid network, which serve as the driving area segmentation head network and the object tracking head network respectively, to obtain a multi-task detection model.

[0044] Module 3: Input the video dataset as training data into the multi-task detection model, and construct a loss function based on the target anchor box category, driving area, and object tracking results output by the multi-task detection model and the label file to train the multi-task detection model to obtain a teacher model. The teacher model is then subjected to end-to-end structured iterative pruning processing of cascaded knowledge distillation to obtain a student model.

[0045] Module 4: Input the real-time driving scene video stream to be identified, collected while the vehicle is driving, into the student model to obtain an inference result including target detection results, drivable area, and target tracking results; based on the inference result, control the vehicle to perform control tasks related to assisted driving.

[0046] The lightweight multi-task video stream real-time inference system, wherein the target anchor frame category label includes category, anchor frame center point coordinates, anchor frame width and anchor frame height; the driving area label includes drivable area and non-drivable area; the object tracking label includes the frame position of the video stream, anchor frame number, anchor frame center point coordinates, anchor frame width, anchor frame height, and confidence that the anchor frame contains an object.

[0047] The lightweight multi-task video stream real-time inference system, wherein the training of the multi-task detection model in module 3 specifically includes:

[0048] The multi-task detection model extracts features from video frames through the backbone network to obtain feature maps of different scales. The feature maps of different scales are fused through the feature pyramid network. The output is decoded by the head network of different tasks and the loss function binary-cross-entropy and CIOU loss is calculated with the label. The binary-cross-entropy loss is specifically:

[0049] L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )]

[0050] The formula of the CIOU Loss is specifically:

[0051]

[0052]

[0053]

[0054]

[0055] Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the prediction box A and the label box B, bgt is the center point of the label box, b is the center point of the prediction box, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the prediction box A and the annotation box B, c is the length of the diagonal of the minimum circumscribed rectangle between the prediction box A and the annotation box B, α is the weight, and is the distance between the aspect ratio of the prediction box A and the annotation box B; A is the prediction box, B is the annotation box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box;

[0056] When the total loss value reaches a preset standard, a target model is obtained. The target model is used to obtain a video frame to be identified and output a detection target label and a drivable area segmentation label identified by the target model, wherein the detection target label is used as the input of the object tracking head network; the object tracking head network uses Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target; wherein the Mahalanobis distance correlation is:

[0057]

[0058] The cosine distance association measure:

[0059]

[0060] The loss function combining Mahalanobis distance and cosine association metric is as follows:

[0061] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0062] Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates d j with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric;

[0063] When the loss c i,j When the preset standards are reached, the teacher network is obtained.

[0064] In the lightweight multi-task video stream real-time inference system, the pruning process in module 3 includes:

[0065] For the training process of the multi-task detection model, the loss function L BCE (y'i ,y i ) adds a penalty factor to constrain the weight of the convolution layer and the scaling factor of the BN layer. The constraint term λR(w) uses the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling factor are as follows:

[0066]

[0067] R(w)=|α|

[0068] After training the end-to-end multi-task detection model until the loss function converges, the absolute values ​​of the filters are summed. Then, the importance function is calculated by combining the scaling coefficient and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows:

[0069]

[0070] m i =α i *E i

[0071] θ=sort p (M)

[0072] The network before pruning is used as the teacher network, and the network after pruning is used as the student network.

[0073] The present invention also proposes a storage medium for storing a program for executing any one of the lightweight multi-task video stream real-time inference methods.

[0074] The present invention also proposes a client for use in any of the aforementioned lightweight multi-task video stream real-time inference systems.

[0075] From the above scheme, it can be seen that the advantages of the present invention are:

[0076] The present invention provides a lightweight multi-task video stream real-time inference method and an end-to-end neural network design method for sharing multiple perception tasks based on YOLOv5. These methods can realize real-time processing of video stream data on edge chips and complete three perception tasks: road object detection, drivable area segmentation, and multi-object tracking, achieving a balance between accuracy and latency. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A flowchart of a lightweight multi-task video stream real-time inference method provided by an embodiment of the present invention;

[0078] Figure 2A schematic diagram of the structure of an end-to-end neural network model for multi-perception task sharing based on YOLOv5 is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0079] The present invention provides a lightweight multi-task video stream real-time inference method and an end-to-end neural network design method for sharing multiple perception tasks based on YOLOv5, which can enable edge chips to process video stream data in real time and complete three perception tasks: road object detection, drivable area segmentation, and multi-object tracking.

[0080] The present invention can provide a lightweight multi-task video stream real-time reasoning method, comprising the following steps:

[0081] 1) Obtain public edge scenario datasets;

[0082] 2) Preprocess the dataset;

[0083] 3) Design of an end-to-end neural network model for multi-perception task sharing based on YOLOv5;

[0084] 4) End-to-end structured iterative pruning technology based on cascaded knowledge distillation;

[0085] Furthermore, in step 1), public edge scenario datasets are collected and downloaded. Taking the autonomous driving scenario dataset as an example, multiple datasets are integrated into one dataset. The dataset labels include: anchor box category labels for target detection tasks, labels for drivable area segmentation tasks, and object re-identification (ReID) labels for multi-object tracking tasks. In this context, "objects" also include pedestrians, i.e., tracking people.

[0086] Furthermore, in step 2), the dataset is preprocessed to convert the label files of different formats of different datasets into a standard format txt file. The converted txt file includes:

[0087] Target detection task labels: category, anchor frame center coordinate x, anchor frame center coordinate y, anchor frame width w, and anchor frame height h; drivable area segmentation labels: directly drivable area, replaceable area, background (non-drivable area); multi-object tracking task labels: the frame of the video stream where the image is located, the ID assigned by the anchor frame, the anchor frame center coordinate x, the anchor frame center coordinate y, the anchor frame width w, the anchor frame height h, and the confidence conf that the anchor frame contains an object.

[0088] Furthermore, the design of the end-to-end neural network model for multi-perception task sharing based on YOLOv5 in step 3) is specifically as follows: the backbone network and feature pyramid network of the original YOLOv5 model are retained, and a branch network is added to the detection head network as the head network responsible for drivable area segmentation, so that the parameter weights of the backbone network and feature pyramid network can be shared by the head network structures of different tasks; the input of the multi-object tracking task depends on the output of the target detection task, therefore, a ReID head network for multi-object tracking is added after the detection head network to complete the multi-object tracking task.

[0089] During the training phase, the target detection labels, drivable area segmentation labels, and multi-object tracking labels of the same image need to be concatenated and used as input. The model extracts features from the original image through the backbone network, fuses feature maps of different scales through the feature pyramid network, and finally decodes the output through the head network of different tasks and calculates the loss function with the original image label. The loss function includes binary-cross-entropy and CIOU loss. The formulas for binary-cross-entropy and CIOU loss are as follows:

[0090] L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )]

[0091]

[0092] When the total loss value reaches the preset standard, a target model is obtained. The target model is used to obtain the image to be identified and output the detection target label and drivable area segmentation label identified by the target model. The drivable area segmentation label is directly post-processed and stored, and the detection target label is used as the input of the multi-object tracking task. The multi-object tracking branch uses the Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target. The Mahalanobis distance association metric calculation formula and the cosine distance association metric formula are as follows:

[0093]

[0094]

[0095] The loss function combining Mahalanobis distance and cosine association metric is as follows:

[0096] c i,j =λd(1) (i,j)+(1-λ)d (2) (i,j)

[0097] When the loss value reaches the preset standard, a multi-object tracking branch model is obtained, which is used to match the detection target with the tracking target, complete the multi-object tracking task, output the multi-object tracking label and store it.

[0098] Furthermore, the end-to-end structured iterative pruning based on cascaded knowledge distillation in step 4) is specifically performed as follows: for the end-to-end neural network model training process described in step 3), a penalty factor is added to the loss function to constrain the weights of the convolutional layer and the scaling coefficient of the BN layer. The constraint term λR(w) adopts the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling coefficient are as follows:

[0099]

[0100] R(w)=|α|

[0101] After training the end-to-end neural network model to convergence, the absolute values ​​of the filters are summed. The importance function is then calculated by combining the scaling factor and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows:

[0102]

[0103] m i =α i *E i

[0104] θ=sort p (M)

[0105] The pre-pruned network serves as the teacher network, and the pruned network serves as the student network for retraining and fine-tuning. This distills network information from three dimensions: feature maps in the intermediate layers, dark knowledge in the classification layer of the RPN / RCN, and dark knowledge in the regression layer of the RPN / RCN. This improves the student network's accuracy. By iteratively executing these steps, the computational complexity of the final end-to-end neural network model is reduced by approximately 40%, while maintaining a performance fluctuation within 1%.

[0106] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0107] Please refer to Figure 1, which is a flow chart of a lightweight multi-task video stream real-time reasoning method provided by an embodiment of the present invention. Among them, a lightweight multi-task video stream real-time reasoning method provided by an embodiment of the present invention specifically includes the following steps.

[0108] Step S101: Collect and download public edge scenario datasets. This example uses autonomous driving as an example and integrates multiple datasets into one dataset. The dataset labels include: anchor box category labels for object detection tasks, labels for drivable area segmentation tasks, and ReID labels for multi-object tracking tasks.

[0109] Step S102: pre-process the data set, convert the label files of different formats of different data sets into a standard format txt file. The converted txt file includes:

[0110] Target detection task labels: category, anchor frame center coordinate x, anchor frame center coordinate y, anchor frame width w, and anchor frame height h; drivable area segmentation labels: directly drivable area, replaceable area, background (non-drivable area); multi-object tracking task labels: the frame of the video stream where the image is located, the ID assigned by the anchor frame, the anchor frame center coordinate x, the anchor frame center coordinate y, the anchor frame width w, the anchor frame height h, and the confidence conf that the anchor frame contains an object.

[0111] The difference between multi-object tracking and target detection is that multi-object tracking requires assigning an ID to each anchor frame. If the anchor frame's content ID is the same in subsequent video frames, it is considered the same object. In other words, target detection only needs to output that the anchor frame is a "person," but tracking requires outputting that the anchor frame in the current frame is a "certain person" and is associated with the same person in the previous frame.

[0112] Step S103, the design of the end-to-end neural network model for multi-perception task sharing based on YOLOv5 is specifically as follows: the backbone network and feature pyramid network of the original YOLOv5 model are retained, and a branch network is added to the detection head network as the head network responsible for drivable area segmentation, so that the parameter weights of the backbone network and feature pyramid network can be shared by the head network structures of different tasks; the input of the multi-object tracking task depends on the output of the target detection task, therefore, the ReID head network for multi-object tracking is added after the detection head network to complete the multi-object tracking task. Among them, please refer to the schematic diagram of the end-to-end neural network model structure for multi-perception task sharing based on YOLOv5. Figure 2The detection head network comes with the original YOLOv5 model. The original YOLOv5 model consists of a backbone network, a feature pyramid network, and a detection head network. The relationship between the three is as follows: a preprocessed image is input into the backbone network, which extracts the multi-scale feature information of the image; the multi-scale feature information is input into the feature pyramid network, which fuses the multi-scale features; the fusion result is input into the detection head network, which then predicts and matches the anchor boxes and outputs the results. Figure 2 The neck network represents the Feature Pyramid Network. The backbone network of the model is called the backbone, the Feature Pyramid Network is the neck, and the detection head network is the head. The ReID head network performs object re-identification based on the output of the detection head network to complete the multi-object tracking task. The backbone network and neck network provide feature information extracted from the video image. The drivable area segmentation head network outputs the drivable area segmentation results. The detection head network outputs the road object detection results, which are input into the ReID network to complete the multi-object tracking task.

[0113] During the training phase, the target detection labels, drivable area segmentation labels, and multi-object tracking labels of the same image need to be concatenated and used as input. The model extracts features from the original image through the backbone network, fuses feature maps of different scales through the feature pyramid network, and finally decodes the output through the head network of different tasks and calculates the loss function with the original image label. The loss function includes binary-cross-entropy and CIOU loss. The formula for binary-cross-entropy is as follows:

[0114] L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )]

[0115] The formula of the CIOU Loss is specifically:

[0116]

[0117] α is defined as follows:

[0118]

[0119] The definition of IoU is as follows:

[0120]

[0121] The definition of v is as follows:

[0122]

[0123] Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the predicted anchor box A and the label box B, b gt is the center point of the label box, b is the center point of the predicted anchor box, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the predicted anchor box A and the labeled box B, c is the length of the diagonal of the minimum circumscribed rectangle between the predicted box A and the labeled box B, α is the weight, v is the distance between the aspect ratio of the predicted box A and the labeled box B; A is the predicted box, B is the labeled box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box;

[0124] When the binary-cross-entropy and CIOU Loss meet the preset standards, the target model is obtained. The target model is used to obtain the video frame to be identified and output the detection target label and driving area segmentation label identified by the target model. The driving area segmentation label can be directly post-processed and stored, and the detection target label is used as the input of the multi-object tracking task. The multi-object tracking branch uses the Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target. Among them, the Mahalanobis distance association metric calculation formula is specifically as follows:

[0125]

[0126] The cosine distance association metric formula is defined as follows:

[0127]

[0128] The loss function of the combined Mahalanobis distance and cosine association metric is as follows:

[0129] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0130] Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates dj with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric;

[0131] When the loss value reaches the preset standard, a multi-object tracking branch model is obtained, which is used to match the detection target with the tracking target, complete the multi-object tracking task, output the multi-object tracking label and store it.

[0132] Step S104, end-to-end structured iterative pruning based on cascaded knowledge distillation, the specific process is: for the end-to-end neural network model training process described in step S103, multiply the penalty factor R(w) after the binary-cross-entropy loss function, constrain the weights of the convolution layer and the scaling coefficient of the batch normalization BN layer, and the constraint term λR(w) adopts the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling coefficient are as follows:

[0133]

[0134] R(w)=|α|

[0135] Where α is a parameter in the batch normalization layer, which represents the scaling factor of the batch normalization layer. After training the above end-to-end neural network model to convergence, the absolute values ​​of the filters (referring to the convolution kernel of each channel. The numerical value is actually the weight of the convolution kernel of each channel) are summed, and then the importance function is calculated by combining the scaling factor and the sum of the absolute values ​​of the filters. Then, the importance is sorted in ascending order according to the preset pruning rate, and the pruning threshold is calculated by the preset pruning rate, and the filters below the threshold are pruned. Among them, the formula for summing the absolute values ​​of the filters, the importance judgment function, and the calculation formula for the pruning threshold are as follows:

[0136]

[0137] m i =α i *E i

[0138] θ=sort p (M)

[0139] Where W is the weight of the filter, E is the sum of the filter weights, α is the scaling factor, m is the importance calculated by the scaling factor and the weight, p is the pruning rate, and θ is the threshold for sorting the importance and retaining the top p% at the p pruning rate.

[0140] The pre-pruned network serves as the teacher network, and the pruned network serves as the student network for retraining and fine-tuning. This distills network information from three dimensions: feature maps in the intermediate layers, dark knowledge in the classification layer of the RPN / RCN, and dark knowledge in the regression layer of the RPN / RCN. This improves the student network's accuracy. By iteratively executing these steps, the computational complexity of the final end-to-end neural network model is reduced by approximately 40%, while maintaining a performance fluctuation within 1%.

[0141] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiment.

[0142] The present invention also proposes a lightweight multi-task video stream real-time reasoning system, which includes:

[0143] Module 1: Obtain a video dataset containing driving scenes, which includes multiple video streams. The video frames in the video streams have label files, which include target anchor box category labels, driving area labels, and object tracking labels;

[0144] Module 2: Obtain the original target detection model with a backbone network and a feature pyramid network. Add two branch networks to the target detection head network of the feature pyramid network, which serve as the driving area segmentation head network and the object tracking head network respectively, to obtain a multi-task detection model.

[0145] Module 3: Input the video dataset as training data into the multi-task detection model, and construct a loss function based on the target anchor box category, driving area, and object tracking results output by the multi-task detection model and the label file to train the multi-task detection model to obtain a teacher model. The teacher model is then subjected to end-to-end structured iterative pruning processing of cascaded knowledge distillation to obtain a student model.

[0146] Module 4: Input the real-time driving scene video stream to be identified, collected while the vehicle is driving, into the student model to obtain an inference result including target detection results, drivable area, and target tracking results; based on the inference result, control the vehicle to perform control tasks related to assisted driving.

[0147] The lightweight multi-task video stream real-time inference system, wherein the target anchor frame category label includes category, anchor frame center point coordinates, anchor frame width and anchor frame height; the driving area label includes drivable area and non-drivable area; the object tracking label includes the frame position of the video stream, anchor frame number, anchor frame center point coordinates, anchor frame width, anchor frame height, and confidence that the anchor frame contains an object.

[0148] The lightweight multi-task video stream real-time inference system, wherein the training of the multi-task detection model in module 3 specifically includes:

[0149] The multi-task detection model extracts features from video frames through the backbone network to obtain feature maps of different scales. The feature maps of different scales are fused through the feature pyramid network. The output is decoded by the head network of different tasks and the loss function binary-cross-entropy and CIOU loss is calculated with the label. The binary-cross-entropy loss is specifically:

[0150] L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )]

[0151] The formula of the CIOU Loss is specifically:

[0152]

[0153]

[0154]

[0155]

[0156] Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the prediction box A and the label box B, b gt is the center point of the label box, b is the center point of the prediction box, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the prediction box A and the annotation box B, c is the length of the diagonal of the minimum circumscribed rectangle between the prediction box A and the annotation box B, α is the weight, and is the distance between the aspect ratio of the prediction box A and the annotation box B; A is the prediction box, B is the annotation box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box;

[0157] When the total loss value reaches a preset standard, a target model is obtained. The target model is used to obtain a video frame to be identified and output a detection target label and a drivable area segmentation label identified by the target model, wherein the detection target label is used as the input of the object tracking head network; the object tracking head network uses Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target; wherein the Mahalanobis distance correlation is:

[0158]

[0159] The cosine distance association measure:

[0160]

[0161] The loss function combining Mahalanobis distance and cosine association metric is as follows:

[0162] c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j)

[0163] Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates d j with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric;

[0164] When the loss c i,j When the preset standards are reached, the teacher network is obtained.

[0165] In the lightweight multi-task video stream real-time inference system, the pruning process in module 3 includes:

[0166] For the training process of the multi-task detection model, the loss function L BCE (y' i ,y i ) adds a penalty factor to constrain the weight of the convolution layer and the scaling factor of the BN layer. The constraint term λR(w) uses the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling factor are as follows:

[0167]

[0168] R(w)=|α|

[0169] After training the end-to-end multi-task detection model until the loss function converges, the absolute values ​​of the filters are summed. Then, the importance function is calculated by combining the scaling coefficient and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows:

[0170]

[0171] m i =α i *E i

[0172] θ=sort p (M)

[0173] The network before pruning is used as the teacher network, and the network after pruning is used as the student network.

[0174] The present invention also proposes a storage medium for storing a program for executing any one of the lightweight multi-task video stream real-time inference methods.

[0175] The present invention also proposes a client for use in any of the aforementioned lightweight multi-task video stream real-time inference systems.

Claims

1. A lightweight multi-task real-time inference method for video streams, characterized by: include: Step 1: Obtain a video dataset containing driving scenes. The video dataset includes multiple video streams. The video frames in the video streams have label files. The label files include target anchor box category labels, driving area labels, and object tracking labels. Step 2: Obtain the original target detection model with a backbone network and a feature pyramid network. Add two branch networks to the target detection head network of the feature pyramid network, which serve as the driving area segmentation head network and the object tracking head network respectively, to obtain a multi-task detection model. Step 3: Input the video dataset as training data into the multi-task detection model, and construct a loss function based on the target anchor box category, driving area, and object tracking results output by the multi-task detection model and the label file to train the multi-task detection model to obtain a teacher model. The teacher model is subjected to end-to-end structured iterative pruning processing of cascaded knowledge distillation to obtain a student model. Step 4: Input the real-time driving scene video stream to be identified, which is collected while the vehicle is driving, into the student model to obtain an inference result including target detection results, drivable area, and target tracking results; According to the inference result, the vehicle is controlled to perform a control task related to assisted driving; Training the multi-task detection model in step 3 specifically includes: The multi-task detection model extracts features from video frames through the backbone network to obtain feature maps of different scales. The feature maps of different scales are fused through the feature pyramid network. The output is decoded by the head network of different tasks and the loss function binary-cross-entropy and CIOU loss is calculated with the label. The binary-cross-entropy loss is specifically: L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )] The formula of the CIOU Loss is specifically: Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the prediction box A and the label box B, b gt is the center point of the label box, b is the center point of the prediction box, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the prediction box A and the annotation box B, c is the length of the diagonal of the minimum circumscribed rectangle between the prediction box A and the annotation box B, α is the weight, and is the distance between the aspect ratio of the prediction box A and the annotation box B; A is the prediction box, B is the annotation box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box; When the total loss value reaches a preset standard, a target model is obtained. The target model is used to obtain a video frame to be identified and output a detection target label and a drivable area segmentation label identified by the target model. The detection target label is used as the input of the object tracking head network. The object tracking head network uses Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target. The Mahalanobis distance correlation is: The cosine distance association measure: The loss function combining Mahalanobis distance and cosine association metric is as follows: c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j) Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates d j with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric; When the loss c i,j When you reach the preset standards, you will get a teacher network.

2. The lightweight multi-task video stream real-time inference method according to claim 1, characterized in that: The target anchor frame category label includes the category, anchor frame center point coordinates, anchor frame width and anchor frame height; the driving area label includes the drivable area and the non-drivable area; the object tracking label includes the frame position of the video stream, the anchor frame number, the anchor frame center point coordinates, the anchor frame width, the anchor frame height, and the confidence that the anchor frame contains the object.

3. The lightweight multi-task video stream real-time inference method according to claim 1, characterized in that: The pruning process in step 3 includes: For the training process of the multi-task detection model, the loss function L BCE (y' i ,y i ) adds a penalty factor to constrain the weight of the convolution layer and the scaling factor of the BN layer. The constraint term λR(w) uses the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling factor are as follows: R(w)=|α| After training the end-to-end multi-task detection model until the loss function converges, the absolute values ​​of the filters are summed. Then, the importance function is calculated by combining the scaling coefficient and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows: m i =a i *E i θ=sort p (M) The network before pruning is used as the teacher network, and the network after pruning is used as the student network.

4. A lightweight multi-task video stream real-time reasoning system, characterized by: include: Module 1: Obtain a video dataset containing driving scenes, which includes multiple video streams. The video frames in the video streams have label files, which include target anchor box category labels, driving area labels, and object tracking labels; Module 2: Obtain the original target detection model with a backbone network and a feature pyramid network. Add two branch networks to the target detection head network of the feature pyramid network, which serve as the driving area segmentation head network and the object tracking head network respectively, to obtain a multi-task detection model. Module 3: Input the video dataset as training data into the multi-task detection model, and construct a loss function based on the target anchor box category, driving area, and object tracking results output by the multi-task detection model and the label file to train the multi-task detection model to obtain a teacher model. The teacher model is then subjected to end-to-end structured iterative pruning processing of cascaded knowledge distillation to obtain a student model. Module 4: Input the real-time driving scene video stream to be identified, collected while the vehicle is driving, into the student model to obtain the inference results including target detection results, drivable area and target tracking results; According to the inference result, the vehicle is controlled to perform a control task related to assisted driving; Training the multi-task detection model in module 3 specifically includes: The multi-task detection model extracts features from video frames through the backbone network to obtain feature maps of different scales. The feature maps of different scales are fused through the feature pyramid network. The output is decoded by the head network of different tasks and the loss function binary-cross-entropy and CIOU loss is calculated with the label. The binary-cross-entropy loss is specifically: L BCE (y′ i ,y i )=-w i [y i *logy′ i +(1-y i )*log(1-y′ i )] The formula of the CIOU Loss is specifically: Among them L BCE is the loss value calculated by the binary-cross-entropy formula, n is the total number of samples, a is the predicted output, and y is the true output; L CIoU is the loss value calculated by the CIOU Loss formula, IoU is the intersection and union of the prediction box A and the label box B, b gt is the center point of the label box, b is the center point of the prediction box, ρ 2 (b,b gt ) is the Euclidean distance between the center points of the prediction box A and the annotation box B, c is the length of the diagonal of the minimum circumscribed rectangle between the prediction box A and the annotation box B, α is the weight, and is the distance between the aspect ratio of the prediction box A and the annotation box B; A is the prediction box, B is the annotation box; w gt is the width of the annotation box, h gt is the length of the annotation box, w is the width of the prediction box, and h is the length of the prediction box; When the total loss value reaches a preset standard, a target model is obtained. The target model is used to obtain a video frame to be identified and output a detection target label and a drivable area segmentation label identified by the target model. The detection target label is used as the input of the object tracking head network. The object tracking head network uses Mahalanobis distance and cosine distance to measure the loss function between the tracking target and the detection target. The Mahalanobis distance correlation is: The cosine distance association measure: The loss function combining Mahalanobis distance and cosine association metric is as follows: c i,j =λd (1) (i,j)+(1-λ)d (2) (i,j) Among them, d j Indicates the position of the detection anchor box, y i Represents the predicted position of the target, S i Indicates d j with y i The covariance matrix between j Represents the feature vector of the detected target, represents the feature vector of the tracking target, and λ represents the weight of the Mahalanobis distance metric; When the loss c i,j When you reach the preset standards, you will get a teacher network.

5. The lightweight multi-task video stream real-time inference system according to claim 4, characterized in that: The target anchor frame category label includes the category, anchor frame center point coordinates, anchor frame width and anchor frame height; the driving area label includes the drivable area and the non-drivable area; the object tracking label includes the frame position of the video stream, the anchor frame number, the anchor frame center point coordinates, the anchor frame width, the anchor frame height, and the confidence that the anchor frame contains the object.

6. The lightweight multi-task video stream real-time inference system according to claim 4, characterized in that: The pruning process in module 3 includes: For the training process of the multi-task detection model, the loss function L BCE (y' i ,y i ) adds a penalty factor to constrain the weight of the convolution layer and the scaling factor of the BN layer. The constraint term λR(w) uses the L1 norm. The constraint formulas for the convolution kernel and the BN layer scaling factor are as follows: R(w)=|α| After training the end-to-end multi-task detection model until the loss function converges, the absolute values ​​of the filters are summed. Then, the importance function is calculated by combining the scaling coefficient and the sum of the absolute values ​​of the filters. The importance is then sorted in ascending order according to the preset pruning rate. The pruning threshold is calculated using the preset pruning rate, and filters below the threshold are pruned. The formula for summing the absolute values ​​of the filters, the importance judgment function, and the pruning threshold are as follows: m i =a i *E i θ=sort p (M) The network before pruning is used as the teacher network, and the network after pruning is used as the student network.

7. A storage medium for storing a program for executing any one of the lightweight multi-task video stream real-time inference methods according to claims 1 to 3.

8. A client, used for any lightweight multi-task video stream real-time inference system described in claim 4 or 6.

Images