Vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning

Abstract

The invention discloses a vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning, and the method comprises the steps: collecting an image through a vehicle-mounted image collection device, taking the image as the input through a dynamic neural network, and outputting the classification and bounding box regression of multiple targets. The dynamic neural network is composed of a multi-teacher network and a student network. The multi-teacher network is composed of a plurality of teacher models; a dynamic weight distribution system based on an attention mechanism is introduced into the multi-teacher network, and different weights are distributed to each teacher model according to the characteristics of the current input scene; a confidence coefficient threshold value is set in a multi-teacher network, and regional knowledge possibly containing a target is only taught to a student network. The student network is composed of a strategy network and a detection network, and the detection network is a two-stage target detection network obtained by integrating a feature extraction network and a target positioning network; the strategy network adaptively extracts task features according to knowledge provided by the multi-teacher network; and the detection network dynamically selects a route according to the policy network decision vector.

Description

Technical Field

[0001] The present invention belongs to the field of unmanned driving technology, and specifically is a vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning. Background Art

[0002] With the rapid development of driverless technology, vehicle-mounted multi-target detection has become a key research direction in the field of computer vision. Accurate and real-time detection of various objects on the road (such as vehicles, pedestrians, traffic signs, etc.) is crucial to ensure the reliability of driverless systems. Two-stage detectors based on deep neural networks (DNNs) have achieved remarkable results in handling the diversity of target objects. [1][2] .

[0003] In simple scenes with a single target, shallow networks can easily complete recognition through basic feature maps. As the scene complexity increases, such as the number of targets, size changes, and background diversity, the recognition ability of shallow networks is insufficient. [3] At this time, DNN needs a deeper network structure to enable the model to extract detailed features. The deep network structure requires a lot of computing resources and is difficult to meet the real-time requirements in the vehicle environment.

[0004] Model pruning and quantization are common methods to improve the compactness of DNN [4] This type of static compression technology can improve the model's reasoning speed by removing convolutional layers and neurons that have little impact on the results. However, the removed neurons may play a role in certain situations, causing the model's performance to degrade when processing complex scenarios. In addition, the fixed model structure is difficult to adapt to inputs of different complexities, which may cause a waste of computing resources in simple scenarios, and appear to be unable to cope with complex scenarios. [5] .

[0005] Dynamic Neural Networks (DyNN) is a potential solution [6] DyNN can adjust the network structure according to the input examples during the reasoning process to adapt to the changes in the scene and the frequently changing targets in the mobile process. Knowledge Distillation (KD) is a method for end-to-end learning of compact models. [7][8][9] One or more teacher models can transfer their learned knowledge to a lightweight student network. This approach can keep the student model at its original size and improve feature extraction capabilities.

[10] ～

[11] In the context of vehicle-mounted multi-target detection, KD provides an effective way to balance model size and detection performance. Based on the above considerations, DyNN and KD are combined to fully utilize the adaptive ability of DyNN and the model compression advantage of KD, so that the student network can achieve efficient and accurate target detection in a dynamically changing road environment. However, there are still some special challenges in the design and training of both, mainly including:

[0006] (1) The adjustment efficiency of dynamic neural networks and their adaptation to complex scenes. Dynamic neural networks are mainly used to solve image classification problems. They face many challenges in vehicle-mounted multi-target detection tasks under complex backgrounds, especially when the number of targets is unbalanced.

[12] ～13] During driving, vehicles face complex and simple backgrounds, signals and ambiguous signals, and [9] The intermittent switching between scenes such as dense traffic and dense pedestrians places high demands on the real-time reasoning of dynamic neural networks. Zuxuan Wu et al. proposed a strategy suitable for target classification scenarios, which aims to guide the detection network to selectively perform the subset of convolutional layers required for classification.

[14] The feature extraction network in the detection network is usually composed of several preset modules, and its network routing (i.e., the opening and closing of the modules) is determined by the policy network reasoning. The policy network is mostly trained using reinforcement learning to achieve the "dynamic switching" of the preset modules. However, the high computational cost of the reinforcement learning module may offset the advantage of the dynamic neural network in inference speed.

[15] ～

[18] In addition, on-board multi-target detection requires not only identifying objects in images, but also accurately locating the bounding boxes of these objects. Current dynamic neural network solutions still have limitations in this regard, and it is difficult to take into account both recognition accuracy and positioning accuracy at the same time.

[0007] (2) Improve the generalization and robustness of DyNN. Compared with static neural networks, the structure and reasoning of dynamic neural networks are more complex. Inappropriate structural design or imperfect training strategies may cause the dynamic network to experience "structural collapse", causing the model to lose its core advantage - adaptive adjustment ability.

[19] Qian X et al. proposed using an adjustable channel network to adjust the network structure according to the input data. However, experiments have found that when the policy network is poorly designed, the model tends to choose a fixed shallow structure, causing the network to lose the ability to adapt to different inputs.

[33] Although knowledge distillation can provide valuable learning guidance for dynamic neural networks, relying solely on a single teacher model has limitations.

[20] A single teacher model often focuses on a specific type of task or data distribution, which may cause the student model to underperform when faced with diverse inputs.

[0008] (3) Co-optimization of the policy-detection network. In the dynamic neural network architecture, there is an interdependent relationship between the policy network and the detection network. The policy network plays the role of a decision maker, responsible for dynamically regulating the activation mode of neurons; while the detection network acts as an executor, focusing on completing the target detection task.

[21] ～

[23] Wang J et al. proposed a layer selection strategy network (LSSN) to dynamically control the activation of different layers in the detection network. Experiments have found that if the strategy network tends to activate shallow features, it may cause high-level features to be lost, leading to insufficient information. In addition, excessive activation of irrelevant features may mistakenly favor some background features, thereby suppressing important target features.

[34] . Summary of the invention

[0009] In response to the above challenges, the present invention proposes a vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning. The method first uses an on-vehicle image acquisition device to collect images; then a dynamic neural network for multi-target detection uses the collected images as input and outputs the classification and bounding box regression of multiple targets;

[0010] The dynamic neural network is composed of a multi-teacher network for different task subsets and a student network;

[0011] The multi-teacher network is composed of multiple teacher models that focus on different instances and specific types of scenarios. A dynamic weight allocation system based on the attention mechanism is introduced into the multi-teacher network. This dynamic weight allocation system assigns different weights to each teacher model according to the characteristics of the current input scenario, so that the student network can learn the most relevant knowledge in a targeted manner. In the multi-teacher network, by setting a confidence threshold, only the regional knowledge that may contain the target is imparted to the student network.

[0012] The student network is composed of a policy network and a detection network, and the detection network is a two-stage target detection network based on Faster-RCNN, which is obtained by integrating the feature extraction network and the target localization network. The policy network adaptively extracts task features based on the professional knowledge provided by multiple teacher models. The detection network dynamically selects the feature extraction network route based on the decision vector provided by the policy network.

[0013] The multi-teacher assisted instance adaptive dynamic neural network framework of the present invention comprises a customizable teacher network and a student network consisting of a policy network and a detection network. The policy network adaptively extracts task features based on the professional knowledge provided by multiple teacher networks, while the detection network dynamically selects network routing based on the decision vector provided by the policy network. The main contributions and innovations of the present invention include:

[0014] First, an attention-based multi-teacher knowledge transfer framework is designed. Each teacher model focuses on a specific task subset or data distribution. By combining the knowledge of these teacher networks, the student can more comprehensively capture the target features in complex scenarios.

[0015] Second, a lightweight policy network based on curriculum learning is designed. For different inputs, the routing vector generated by the policy network is mapped to a feature extraction network stacked by residual blocks. Different from the traditional "one-step-one-decision" routing mode, the proposed policy network outputs a routing vector that matches the task attributes at one time during the policy generation phase to reduce the inference cost of the detection network.

[0016] Third, a joint training framework for dynamic routing is constructed. This framework allows the policy network and the detection network to achieve mutual adaptation and enhancement between the two networks through multi-teacher knowledge transfer. Under the guidance of the policy network, the detection network can better perform the dual tasks of classification and localization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Represents a dynamic neural network architecture for multi-object detection;

[0018] Figure 2 represents the student network model;

[0019] Figure 3 represents the improved feature extraction network;

[0020] Figure 4 The joint training and fine-tuning workflow for representation knowledge distillation;

[0021] Figure 5 Represents the detection accuracy and average FLOPs on the COCO dataset;

[0022] Figure 6 represents the probability distribution of Block14 route length;

[0023] Figure 7(a) and Figure 7(b) show representative examples of low FLOPs and high FLOPs, respectively, where:

[0024] Figure 7(a) corresponds to the Low FLOPs group;

[0025] Figure 7(b) corresponds to the High FLOPs group;

[0026] Figure 8(a) and Figure 8(b) show the routing strategy visualization for low FLOPs and high FLOPs images, respectively.

[0027] Figure 8(a) corresponds to the Low FLOPs group;

[0028] Figure 8(b) corresponds to the High FLOPs group;

[0029] Fig. 9 represents the probability distribution of Block31 route length;

[0030] Fig.10 Indicates representative image examples of MS-COCO;

[0031] Fig.11 Visualization of routing strategies for MS-COCO images (long routing case). DETAILED DESCRIPTION

[0032] 1 Overview

[0033] For the on-board multi-target detection task in unmanned driving systems, the present invention proposes a detection method that combines multi-teacher knowledge distillation with dynamic neural networks to cope with complex and changeable road environments, improving the reasoning speed while ensuring the reasoning accuracy.

[0034] Firstly, an attention-based multi-teacher knowledge transfer framework is designed to help the student network capture the target features in complex scenarios by focusing on a specific task subset of teacher models.

[0035] Then, a lightweight strategy network based on curriculum learning and a reward and punishment mechanism for multi-target detection were proposed to dynamically control the opening and closing of the residual blocks of the feature extraction network in the detection network.

[0036] Finally, under the joint training framework, the policy network and the detection network achieve mutual adaptation and enhancement through multi-teacher knowledge transfer, so that the student network can better complete the two tasks of classification and target detection.

[0037] Experimental results show that this method has higher detection accuracy than traditional model compression methods on datasets such as MS-COCO and PASCAL VOC, while significantly improving reasoning efficiency and resource utilization, providing an efficient solution for on-board multi-target detection in unmanned driving systems.

[0038] 2 Dynamic Neural Network Architecture for Multi-Target Detection

[0039] like Figure 1 As shown in Figure 1, the proposed framework consists of a multi-teacher network for different task subsets and a student network. The student network contains a policy network and a target detection network. The teacher model is responsible for capturing target features in complex scenarios and passing this knowledge to the student network to improve the student network's in-vehicle multi-target detection capabilities.

[0040] 2.1 Attention-based Multi-Teacher Network

[0041] The multi-teacher network consists of multiple teacher models that focus on different instances and specific types of scenes to cope with the rapid changes in scenes during vehicle movement, such as frequent switching between complex and simple backgrounds, clear and blurred lighting, dense traffic and dense pedestrians. A dynamic weight allocation system based on the attention mechanism is introduced to integrate this diverse knowledge. The system assigns different weights to each teacher model based on the characteristics of the current input scene, allowing the student network to learn the most relevant knowledge in a targeted manner. By setting a confidence threshold, the teacher network only teaches the student network knowledge about areas that may contain targets to reduce background noise and redundant information.

[0042] 2.2 Student Network Based on Dynamic Reasoning

[0043] like Figure 2 As shown in Figure 1, the student network consists of three parts: the strategy network, the feature extraction network, and the target localization network. The feature extraction network and the target localization network are integrated into a two-stage target detection network based on Faster-RCNN.

[24] .

[0044] The object localization network in the student network consists of a region proposal network (RPN), a RoIPooling layer, and a classification and detection network (R-CNN). The feature extraction network extracts features from the input image through convolution operations. The image features are processed by RPN to generate a series of prior boxes. These prior boxes are then classified and refined through the SoftMax function.

[24] .

[0045] In Faster R-CNN with ResNet as the feature extraction network, the first four layers of ResNet are used to generate feature maps of candidate regions, and the fifth layer is used for classification and regression to balance resolution and semantic information. [6] In view of this architecture, the present invention improves the feature extraction network of FasterR-CNN (refer to Figure 3 Specifically, the K residual blocks of the first four stages are stacked sequentially, and skip connections are added between each residual block.

[25] This improvement enables the network to adaptively extract features based on the needs of different instances.

[0046] For each input image x, the policy network in the student network outputs the discard decision U∈{0,1} of all residual blocks at once KThis is essentially a single-step Markov decision process given an input state.

[26] . Let s k ∈[0,1] is the kth element in the strategy vector S, and its value represents the residual block The probability of being turned on. Given an image x, the strategy for selecting the residual block is defined as a K-dimensional Bernoulli distribution, that is

[0047]

[0048] The policy network is described as a function f(·,W) with weights W. The image x is input to f and a feature map is obtained. The feature map is input to the activation function σ(x) = 1 / 1 + e -x , we get the following decision vector

[0049] S=f(x,W)={s1,s2,...,s K} (2)

[0050] The policy network generates an action vector according to (2). k =0 and u k = 1 means that the residual block k is discarded and retained respectively. The object detection network then generates classification and bounding box results based on the extracted features. For image x, the residual block usage is

[0051]

[0052] Removing blocks will increase the loss of feature information. In order to balance the feature loss and block usage, a reward-reward mechanism for object detection is designed.

[0053] In the target localization network, the proposed boxes output by the RPN are further screened according to the preset confidence. The average confidence M of the retained prior boxes and the true boxes is used as the judgment condition of the reward function. The training process revolves around the reward function, and the routing length (number of blocks used) of the detection network is minimized while ensuring that the prediction results are consistent with the true labels. Following the above rules, the reward function of the policy network is set to

[0054]

[0055] When M>a (correct detection), the shorter the route length, the more positive rewards are given to encourage the policy network to skip unnecessary residual blocks. When M<a (error detection), α1 and α2 reflect different degrees of punishment, driving the policy network to adjust the selection strategy of residual blocks to improve detection accuracy. The expected reward obtained by formula (5) is expressed as

[0056]

[0057] The policy network obtains the optimal parameters by maximizing (5).

[0058] 3 Joint Training and Fine-tuning Based on Knowledge Distillation

[0059] This section explains the joint training and fine-tuning strategy for multi-teacher knowledge distillation. Figure 4 The workflow is given, including the following parts:

[0060] Knowledge transfer: At the initial stage of training, the residual blocks in the feature extraction network are fully open. The parameters of the policy network are frozen and used to train the object detection network. Through multi-teacher knowledge distillation,

[0061] Give the target detection network a higher starting point.

[0062] Policy generation: The curriculum learning (CL) based policy network outputs a routing vector based on the input instance, which is subsequently mapped to the feature extraction network.

[0063] Routing path: The feature extraction network dynamically turns on or off its internal residual blocks according to the mapping results to achieve an efficient reasoning process.

[0064] Policy training (Back Propagation): The target localization network uses the intermediate inference parameters to generate detection results, which are input into the reward function as a key condition. The reward function then outputs a reward value, which optimizes the decision-making ability of the policy network through back propagation.

[0065] Object detection: The object detection task is decomposed into two subtasks: candidate region generation and object classification and bounding box regression. RPN outputs a set of anchor boxes with higher probability as candidate regions, and the candidate regions are mapped into fixed-size feature maps. These fixed-size feature maps are used by RCNN for object classification and precise positioning.

[0066] Joint training: After the target detection network and the policy network have each reached a certain level of performance, they enter the joint training phase. This phase uses multi-teacher knowledge distillation for joint training, while updating the parameters of the policy network and the detection network.

[0067] 3.1 Attention-based Multi-Teacher Feature Guidance

[0068] In the two-stage object detection network, RPN focuses on the local areas near the target object because the feature responses in these areas contain important information for effective detection of the target. Based on this feature, the suggestion boxes generated by the teacher network in the RPN stage are filtered by pre-set confidence, and only a small number of suggestion boxes are retained for the knowledge transfer process. This scheme focuses the knowledge transfer on the key areas near the target object to avoid the background areas with fewer features.

[0069] The feature maps of the student and teacher models m are represented as G s and A fully convolutional adaptation layer f a (·) is used to solve the problem of incompatibility of the number of feature channels of the student and teacher models. For the target anchor point position (x, y) on the feature map, the distance between the student model and the teacher model m at the anchor point is measured as

[0070]

[0071] The N estimated anchor points (imitation mask I) on a feature map with a width of W and a height of H are accumulated, and the distillation goal is to minimize the sum of the distance metrics of all anchor points, that is,

[0072]

[0073] The feature map output by the teacher network m is z m , and introduce the learning parameter w m and b m Through the attention mechanism, the weight of the teacher network m is

[0074]

[0075] The total distillation loss is obtained by weighted addition of the distillation losses of each teacher network.

[0076]

[0077] Among them, λ is the feature loss weight balancing factor.

[0078] The hard label loss of the student network is divided into RPN loss and R-CNN loss. The two parts of the loss are calculated in the same way, including classification loss (Classification Loss, Cls) and regression loss (Regression loss, Rls)

[24] The binary cross entropy loss L1(,·,) and Smooth L1 loss are used to calculate the classification loss and regression loss. The loss function of RPN is

[0079]

[0080] When calculating Cls, the probability that the i-th prior box is predicted as a target is expressed as p i When the GT intersection over union ratio IoU>0.7, the algorithm determines that the prior box contains an object and identifies the prior box as the foreground. is 1.

[0081] When IoU < 0.3, the algorithm determines that the prior frame does not contain the detection target and identifies the prior frame as the background. When calculating Rls, the predicted bounding box offset and the GT box offset corresponding to the foreground prior box are expressed as q i and When training the RPN network, N is the number of positive and negative samples sent to R-CNN. Among them, N / 2 positive samples participate in the training of the R-CNN network, and the R-CNN loss L is obtained using the same calculation method. c In summary, the total training loss of the student network is

[0082] L=λ1L gt +λ2L P +λ3L C (11)

[0083] The weight balancing factors λ1, λ2, λ3 ≥ 0 satisfy λ1 + λ2 + λ3 = 1. Maximizing formula (11) forces the student network to learn the compact knowledge of the multi-teacher detection model.

[0084] 3.2 Cl-based policy network training

[0085] The policy network needs to take into account both detection accuracy and inference cost, and provide dynamic routing that matches the task instance for the feature extraction network. In the policy generation, link jump, and policy training stages, the usage rate of the residual blocks in the feature extraction network and the prediction accuracy of the intermediate layers of the target detection network need to be considered to guide the training of the policy network.

[0086] RPN will give 300 proposal boxes for the input image. The confidence of all proposal boxes generated in this stage and the real target box is calculated and sorted, and the proposal boxes with confidence higher than 0.7 will be retained. The average confidence between the proposal boxes and the real boxes that are retained after NMS screening will be used as the prediction accuracy of the middle layer of the network to measure the error. According to formula (4), the target detection network can obtain the expected reward J of formula (5) through the penalty mechanism.

[0087] As a traditional reinforcement learning method, the policy gradient method

[27] Sampling from a multinomial distribution to obtain policy gradients is often used to search for the gradient that maximizes equation (5). In contrast, this paper samples training policy samples from a k-dimensional Bernoulli distribution. The policy gradient of the policy network parameters is calculated as

[0088]

[0089] The policy gradient (12) has a high variance, which causes the model to converge too slowly. To solve this problem, we use the critical baseline was introduced, where It is the most likely routing strategy to be selected under the current strategy. The policy gradient is recomputed as

[0090]

[0091] Through this method, the expectation of the policy gradient remains unchanged, but the variance is reduced, making the policy network model converge quickly and stably during training.

[0092] 3.3 Joint fine-tuning with the assistance of multiple teachers

[0093] The detection network and the policy network are trained independently. When the two networks update their parameters one after another, they will affect each other. It is necessary to jointly fine-tune the policy network and the object detection network through multi-teacher knowledge distillation so that the entire detection network can better adapt to the learned block discarding behavior. The joint fine-tuning assisted by multiple teachers is summarized as Algorithm 1. The parameters of the policy network are frozen. After the detection network is trained by multi-teacher knowledge distillation, the detection accuracy upper limit and parameter diversity are improved, which lays a high performance starting point for the object detection network (lines 4-6). The detection network is then frozen. The policy network sets the first Kh variables in the policy vector s to 1, and gradually increases h to start CL training (lines 7-16). After the student model explores a variety of routing strategies, the detection network and the policy network are jointly fine-tuned with the assistance of multiple teachers to achieve the best results (lines 17-19).

[0094]

[0095] 4 Experimental Preparation

[0096] 4.1 Dataset Selection

[0097] In order to evaluate the detection performance of the model, the present invention uses the public target detection dataset: MS-COCO

[28] 、PASCAL VOC

[29] . The PASCAL VOC dataset used contains 21,503 images from 2005 to 2012, of which 19,352 are used for training and 2,151 are used for testing. In order to better meet the needs of unmanned driving, categories in the dataset that are not related to unmanned driving (such as airplanes, bottles, and sofas) were removed, and finally 15 categories were retained. The MS-COCO dataset used contains 80 categories, mainly including 117,266 training sets, 4,952 validation sets, and 40,670 test sets. Categories that are not directly related to unmanned driving scenes are still eliminated. 10 categories (including vehicles, pedestrians, traffic signs and lights, bicycles and riders, etc.) are retained. Compared with PASCAL VOC, the image backgrounds in MS-COCO are more complex, the number of targets is greater, and the target size is smaller.

[0098] 4.2 Evaluation Metrics

[0099] In unmanned driving scenarios, it is inevitable to encounter various complex images, such as: bloated background, large number of targets, small target size, etc. For the detection of small targets, we use average precision (AP) as the evaluation indicator to intuitively reflect the detection accuracy of the detector in each category. For multi-target detection, mAP is used to measure the overall performance of the detector in all categories. The precision-recall curve and the number of categories are represented as P(r) and N. The average accuracy of target detection is

[0100]

[0101] The mAP evaluation index used is calculated based on IoU = 0.5. In-vehicle multi-target detection has strict requirements on the computational overhead of the model, and the number of floating-point operations (FLOPs) is introduced as an evaluation index.

[0102] FLOPs=2×C1×C2×H×W×K×K (15)

[0103] Among them, C1 is the number of input channels of the convolution layer, C2 is the number of output channels of the convolution layer, H and are the height and width of the feature map W respectively, and K is the size of the convolution kernel.

[0104] 4.3 Benchmark Methods

[0105] For the two-stage object detection network, the focus of model compression is on the feature extraction part. Two residual detection networks Block14 and Block31 with similar number of convolution kernels as ResNet50 and ResNet110 are constructed. As the basic feature extraction networks, they consist of 14 and 31 residual blocks respectively. VGG, DenseNet and ResNet are selected as reference benchmarks. The following three benchmark methods are selected for comparison with the proposed dynamic routing method based on multi-teacher knowledge distillation:

[0106] Slimming

[30] , is a static model compression method, the principle of which is to remove network channels below a specific threshold layer by layer according to a preset cropping ratio.

[0107] ·Early exiting

[31] , allowing the network to exit early when the detection result reaches the confidence threshold, and no longer perform calculations on subsequent layers.

[0108] Random depth

[32] , randomly skip some layers in the network during training, and the skipped layers will be replaced by identity mapping, but the complete network is used during inference.

[0109] 5 Experimental results analysis

[0110] To verify the effectiveness of multi-teacher knowledge transfer for two-stage detection networks, the PASCAL VOC dataset is introduced for algorithm evaluation. To verify the effectiveness of dynamic routing, MS-COCO is introduced to test the practicality of multi-teacher assisted joint fine-tuning.

[0111] 5.1 Analysis on the effectiveness of multi-teacher knowledge distillation

[0112] The first set of experiments examines the role of multi-teacher knowledge distillation in promoting model compression and detection accuracy. In order to build a multi-teacher network, three widely used deep learning models, Resnet110, VGG-19, and DenseNet121, were selected. The feature extraction network of the student network is stacked by residual blocks and is named Student. All models use Faster R-CNN as the basic framework. Table 1 shows the number of parameters of each model and the detection accuracy on the VOC dataset. VGG-19, which has the largest number of parameters, is used as a benchmark to measure the compression rate of other models. Although the size of Student is greatly compressed to 3.3M, its detection accuracy on the VOC dataset reaches 78.2%. This result is better than Resnet110 and VGG-19, and slightly worse than DenseNet121.

[0113] Table 1 Compression rate and detection accuracy of student model

[0114]

[0115] Table 2 shows the effect of teacher group size on the detection performance of the student network. The detection accuracy is positively correlated with the number of teacher networks, but the gain obtained by the student network shows a marginal benefit. The gap between single and dual teachers is 1.6%, which is larger than the performance gap between dual and triple teachers of 0.9%. Compared with no teacher guidance and single teacher guidance, the student network under multi-teacher guidance can produce more compact class-specific clusters, and the boundaries between different categories of objects become clearer.

[0116] Table 2 Compression rate and classification accuracy of student model

[0117]

[0118] Although the fixed-weight multi-teacher model Teacher3 outperforms the baseline model, its mAP is 0.8% lower than that of the dynamic weight model Teacher4 with attention mechanism. Especially when dealing with drastic changes in scenes, the fixed-weight model is obviously not adaptable enough and it is difficult to dynamically adjust the feature extraction strategy for different inputs.

[0119] 5.2 Dynamic Routing Effectiveness Analysis

[0120] The second set of experiments compares the proposed dynamic routing with pruning and early exit. Table 3 lists the model detection accuracy mAP and the average number of residual blocks used in the feature extraction network of Block14 and Block31 on VOC and COCO datasets. Assuming that the average length of the output routing is To ensure that the upper limit of the detection capability of the baseline model is not weaker than that of the proposed method, when running the cropping method, the cropping ratio of the model feature extraction network is set to When running early exit, the number of residual blocks used is set to The upper bound of

[0121] On the VOC dataset, Table 3 shows that the average length of the dynamic routing of the Block14 and Block31 models after CL training is 9.7 and 19.8, achieving an average accuracy of 77.9% and 81.3%, which is 53.6% and 60.4% higher than pruning, and 63.4% and 59.8% higher than early withdrawal. This result proves that knowledge transfer and CL training can improve detection accuracy and reduce the inference cost during model inference. Next, we investigate the effectiveness of multi-teacher assisted joint fine-tuning. On the VOC dataset, after combined fine-tuning, the Block14 and Block31 models, which are trained separately by knowledge transfer and CL training, not only improve the detection accuracy by 0.3% and 0.4%, but also reduce the routing length by 0.9 and 3.1 on average.

[0122] Table 3. Impact of model compression strategy on detection accuracy

[0123]

[0124]

[0125] 5.3 Dynamic Inference Model Acceleration Performance

[0126] The third set of experiments aims to compare the performance of the proposed dynamic routing method with network pruning, early retirement and random depth traditional model acceleration techniques. The parameters a and b in (4) are adjusted to balance the routing length and target detection accuracy, with the aim of finding the best balance between mAP and FLOPs under different detection requirements. Figure 5 Each point on the curve represents the model performance under a given set of parameters a and b. The vertical axis represents the model complexity, measured by average FLOPs. The horizontal axis represents the detection accuracy, measured by mAP.

[0127] Figure 5 The figure shows the average FLOPs and detection accuracy of different methods on MS-COCO when Block31 is used as the feature extraction network. The proposed model is significantly better than other baseline models. With the same detection accuracy, the proposed method reduces 23.8% of FLOPs (1.6×10 11 2.1×10 11 ); it only takes 76% of the FLOPs (1.3×10 11 1.7×10 11 ) can achieve the same detection accuracy of 75.2%. At the same model complexity (red dotted line), the proposed method improves the accuracy of cropping by 2.1% (76.1% vs. 78.2%).

[0128] 5.4 Impact of instance complexity on routing and inference cost

[0129] From the above results, we can see that the proposed scheme produces very different inference costs when detecting images of different complexity. Images with obvious features usually require fewer convolutional layer embeddings than complex or atypical images. This subsection analyzes the impact of instance complexity on routing selection to understand the correlation between routing length and image complexity. In order to visualize the dynamic reasoning, this group of experiments collected 1000 test instances of FLOPs and recorded the routing length corresponding to each test instance. Figure 6 It can be seen that block 14 uses an average of 8.8 residual modules when testing instances, and the average usage of the modules is 63%.

[0130] The threshold of the average routing length is 9. Test instances with less than 9 residual blocks are selected as the low FLOPs group, and those with 9 or more residual blocks are selected as the high FLOPs group. As shown in Figure 7(a) and Figure 7(b), this experiment selects 8 representative images from the low and high FLOPs groups for visual analysis. The objects in each image have been marked according to the label. The visual difference between the two groups of images is quite obvious. The low FLOPs images have a simple background, large objects, and easy-to-identify objects; while the high FLOPs images have a complex background, small and numerous objects, and difficult-to-detect objects.

[0131] The two groups of images are input into the policy network respectively, and the generated routing vectors are visualized as Figure 8(a) and Figure 8(b). The horizontal axis represents the residual module number and the vertical axis represents the image number; gray (blank) represents the residual block that participates (does not participate) in routing. It can be seen that the routing strategies of the two groups of images show different characteristics. The routing strategies of the same category instances in each group are relatively similar, suggesting that the features of the same category are stored in similar filters; the average number of residual modules used in the low FLOPs group is lower than that in the high FLOPs group, which is consistent with the proposed inference, that is, the complexity of the image is positively correlated with the routing length, and complex images will generate more inference overhead; images that are easy to distinguish or have clear features only need short routing to meet the requirements, while images with small and numerous objects rely on long routing.

[0132] Next, this section examines the routing node distribution and inference cost under long routing. Compared with VOC, the image types in MS-COCO are richer and the image resolution is higher, so the processing of such images relies on a larger model to store complex parameters. Fig. 9 It can be seen that in the feature extraction network composed of 31 residual modules, the test instance uses an average of 22 residual blocks, and the average block usage is 70.1%. Compared with the average usage of 63% in VOC, the block usage under MS-COCO has increased by 7.1%. Fig.10The 7 image examples in are selected from MS-COCO. After these images are input into Block31, the generated routing strategy is visualized as Fig.11 . Combining Figure 8(a), Figure 8(b) and Fig.11 ,Facing instances of different complexity, the proposed dynamic network improves the ,adaptability of the model.,Whether it is simple or complex images, the proposed scheme can generate ,routes matching them according to the features of the examples, balancing the detection ,accuracy and inference cost.

[0133] 6 Conclusion

[0134] In order to achieve accurate and efficient multi-target detection in autonomous driving scenarios, the present invention proposes a dynamic neural network method based on multi-teacher distillation. This scheme enables the dynamic neural network to flexibly adapt to various complex tasks by combining the knowledge of multiple teacher models with different advantages. This mechanism not only combines the advantages of multiple models, makes up for the shortcomings of a single model in some aspects, but also achieves more efficient target detection. In addition, the detection network and the policy network learn feature expression and routing strategy from the combined knowledge of multiple pre-trained teacher models, respectively, so that the student network builds an instance-path mapping relationship similar to a routing table, thereby enhancing its robustness. This method not only provides more valuable guidance for the training of dynamic neural networks, but also enables the model to show higher stability and accuracy when facing complex and changeable tasks. Through simulation experiments, it is verified that the scheme proposed in this paper is superior to other benchmark schemes in improving model detection accuracy, reasoning efficiency, robustness and compressing the number of model parameters.

[0135] References

[0136] [1]Du L, Zhang R, Wang

[0137] [2] Maleki H, M,Durak-Ata L.Handover-enabled dynamic computationoffloading for vehicular edge computing networks[J].IEEE Transactions onVehicular Technology,2023,72(7):9394-9405.

[0138] [3]Meir Y, Tevet O, Tzach Y, et al. Efficient shallow learning as an alternative to deep learning [J]. Scientific Reports, 2023, 13(1):5423.

[0139] [4] Zeng Huanqiang, Hu Haolin, Lin Xiangwei, et al. A review of deep neural network compression and acceleration[J]. Signal Processing, 2022, 38(1): 183-194.

[0140] [5]He J, Koric S, Kushwaha S, et al. Novel DeepONet architecture to predict stresses in elastoplastic structures with variable complex geometries and loads [J]. Computer Methods in Applied Mechanics and Engineering, 2023, 415: 116277.

[0141] [6]Mehnatkesh H, Jalali SMJ, Khosravi A, et al. An intelligent drivendeep residual learning framework for brain tumor classification using MRIimages[J]. Expert Systems with Applications, 2023, 213:119087.

[0142] [7]Gou J, Yu B, Maybank SJ, et al.Knowledge distillation:A survey[J].International Journal of Computer Vision, 2021,129(6):1789-1819.

[0143] [8]Zhang L,Ma K.Structured knowledge distillation for accurate andefficient object detection[J].IEEE Transactions on Pattern Analysis andMachine Intelligence,2023.

[0144] [9]Park W,Kim D,Lu Y,et al.Relational knowledge distillation[C] / / Proceedings of the IEEE / CVF conference on computer vision and patternrecognition.2019:3967-3976.

[0145]

[10] Zhang H,Chen D,Wang C.Confidence-aware multi-teacher knowledgedistillation[C] / / ICASSP 2022-2022IEEE International Conference on Acoustics,Speech and Signal Processing(ICASSP).IEEE,2022:4498-4502.

[0146]

[11] Sun N,Zhao J,Shi Q,et al.Moving Target Tracking by UnmannedAerial Vehicle:A Survey and Taxonomy[J].IEEE Transactions on IndustrialInformatics,2024.

[0147]

[12] Khentout N,Magrotti G.Fault supervision of nuclear researchreactor systems using artificial neural networks:A review with results[J].Annals of Nuclear Energy,2023,185:109684.

[0148]

[13] Zhou L,Kumar V.Robust multi-robot active target tracking againstsensing and communication attacks[J].IEEE Transactions on Robotics,2023,39(3):1768-1780.

[0149]

[14] Hu X,Huang Z,Huang A,et al.A dynamic multi-scale voxel flownetwork for video prediction[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2023:6121-6131.

[0150]

[15] Han Y,Huang G,Song S,et al.Dynamic neural networks:A survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,44(11):7436-7456.

[0151]

[16] Li Y.Deep reinforcement learning:An overview[J].arXiv preprintarXiv:1701.07274,2017.

[0152]

[17] Milani S,Topin N,Veloso M,et al.Explainable reinforcementlearning:A survey and comparative review[J].ACM Computing Surveys,2024,56(7):1-36.

[0153]

[18] Han D,Mulyana B,Stankovic V,et al.A survey on deep reinforcementlearning algorithms for robotic manipulation[J].Sensors,2023,23(7):3762.

[0154]

[19] Zhu Y F,Yao Y,Huang Y,et al.Machine learning applications forassessment of dynamic progressive collapse of steel moment frames[C] / / Structures.Elsevier,2022,36:927-934.

[0155]

[20] Zhao B,Cui Q,Song R,et al.Decoupled knowledge distillation[C] / / Proceedings of the IEEE / CVF Conference on computer vision and patternrecognition.2022:11953-11962.

[0156]

[21] Hosseini Shirvani M,Akbarifar A.A Survey Study on IntrusionDetection Systemin Wireless Sensor Network:Challenges and Considerations[J].Journal of Electrical and Computer Engineering Innovations(JECEI),2024,12(2):449-474.

[0157]

[22] Yi Z,Lian J,Liu Q,et al.Learning rules in spiking neuralnetworks:A survey[J].Neurocomputing,2023,531:163-179.

[0158]

[23] Papaleo F,Lizarraga-Seijas X,Font F.Evaluating Neural NetworksArchitectures for Spring Reverb Modelling[J].arXiv preprint arXiv:2409.04953,2024.

[0159]

[24] Vaishnavi K,Reddy G P,Reddy T B,et al.Real-time object detectionusing deep learning[J].Journal of Advances in Mathematics and ComputerScience,2023,38(8):24-32.

[0160]

[25] Wang S,Jiu M,Chen L,et al.A deep encoder-decoder based primal-dual proximal network for image restoration[C] / / Fifteenth InternationalConference on Graphics and Image Processing(ICGIP 2023).SPIE,2024,13089:312-322.

[0161]

[26] Iqbal S,N.Qureshi A,Li J,et al.On the analyses of medical imagesusing traditional machine learning techniques and convolutional neuralnetworks[J].Archives of Computational Methods in Engineering,2023,30(5):3173-3233.

[0162]

[27] Dang T,Nguyen T T,McCall J,et al.Two-layer Ensemble of DeepLearning Models for Medical Image Segmentation[J].Cognitive Computation,2024:1-20..

[0163]

[28] Zimmermann E,Szeto J,Pasquero J,et al.Benchmarking a Benchmark:How Reliable is MS-COCO?[J].arXiv preprint arXiv:2311.02709,2023.

[0164]

[29] Prakash S,Shah P,Agrawal A.Exploiting CNNs for SemanticSegmentation with Pascal VOC[J].arXiv preprint arXiv:2304.13216,2023.

[0165]

[30] Li L,Zhao P.Protective Self-Adaptive Pruning to Better CompressDNNs[J].arXiv preprint arXiv:2303.11881,2023.

[0166]

[31] Sun B,Ye X,Wang Z,et al.Exploring coarse-to-fine action tokenlocalization and interaction for fine-grained video action recognition[C] / / Proceedings of the 31st ACM International Conference on Multimedia.2023:5070-5078.Sun B,Ye X,Wang Z,et al.Exploring coarse-to-fine action tokenlocalization and interaction for fine-grained video action recognition[C] / / Proceedings of the 31st ACM International Conference on Multimedia.2023:5070-5078.

[0167]

[32] Ernst D,Louette A.Introduction to reinforcement learning[J].Feuerriegel,S.,Hartmann,J.,Janiesch,C.,and Zschech,P,2024:111-126.

[0168]

[33] Qian X,Liu F,Jiao L,et al.Knowledge transfer evolutionary searchfor lightweight neural architecture with dynamic inference[J].PatternRecognition,2023,143:109790.

[0169]

[34] Wang J,Su K,Yang G,et al.Research on multi-layer network routingselection strategy based on cooperative evolutionary game in IoT environment[J].Frontiers in Physics,2023,11:1274578.

Claims

1. A vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning, characterized by First, the vehicle-mounted image acquisition device acquires images; then the dynamic neural network for multi-target detection uses the acquired images as input and outputs the classification and bounding box regression of multiple targets; The dynamic neural network is composed of a multi-teacher network for different task subsets and a student network; The multi-teacher network is composed of multiple teacher models that focus on different instances and specific types of scenarios. A dynamic weight allocation system based on the attention mechanism is introduced into the multi-teacher network. This dynamic weight allocation system assigns different weights to each teacher model according to the characteristics of the current input scenario, so that the student network can learn the most relevant knowledge in a targeted manner. In the multi-teacher network, by setting a confidence threshold, only the regional knowledge that may contain the target is imparted to the student network. The student network is composed of a policy network and a detection network, and the detection network is a two-stage target detection network based on Faster-RCNN, which is obtained by integrating the feature extraction network and the target localization network. The policy network adaptively extracts task features based on the professional knowledge provided by the multi-teacher network. The detection network dynamically selects the feature extraction network routing according to the decision vector provided by the policy network; The target localization network consists of a region proposal network (RPN), a RoIPooling layer, and a classification and detection network (R-CNN). The feature extraction network extracts features from the input image through convolution operations. The image features are processed by RPN to generate a series of prior boxes; these prior boxes are then classified and refined through the SoftMax function; The feature extraction network in Faster R-CNN is ResNet. The first four layers of ResNet are used to generate feature maps of candidate regions, and the fifth layer is used for classification and regression. In ResNet, the K residual blocks of the first four stages are stacked in sequence, and jump connections are added between each residual block. For each input image x, the policy network outputs the discard decision U∈{0,1} of all residual blocks at once K , which is essentially a single-step Markov decision process for a given input state; let s k ∈[0,1] is the kth element in the strategy vector S, and its value represents the residual block The probability of being turned on; u k =0 and u k =1 indicates that the residual block k is discarded and retained respectively; For a given image x, the strategy for selecting the residual block is defined as a K-dimensional Bernoulli distribution, that is, The policy network is described as a function f(·,W) with weights W; the image x is input to f to obtain a feature map; the feature map is input to the activation function σ(x) = 1 / 1 + e -x , we get the following decision vector S=f(x,W)={s1,s2,...,s K } (2) The policy network generates an action vector according to formula (2); the target detection network then generates classification and bounding box results based on the extracted features; for image x, the residual block usage is Removing blocks will increase the loss of feature information, so a reward-reward mechanism for target detection is used to balance feature loss and block usage; In the target positioning network, the suggested boxes output by the RPN are further screened according to the preset confidence; the average confidence M of the retained prior boxes and the true boxes is used as the judgment condition of the reward function; The training process revolves around the reward function. Under the premise of ensuring that the prediction results are consistent with the true labels, the routing length of the detection network, that is, the number of residual blocks used, is minimized; the reward function of the policy network is set to When M>a, that is, the detection is correct, the shorter the route length, the more positive rewards are given to encourage the policy network to skip unnecessary residual blocks; when M<a, that is, the detection is wrong, α1 and α2 reflect different degrees of punishment, driving the policy network to adjust the selection strategy of the residual block to improve the detection accuracy; The expected reward obtained by formula (5) is expressed as The policy network obtains the optimal parameters by maximizing (5).

2. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 1 is characterized in that The joint training and fine-tuning strategy of the policy network and the detection network includes the following workflow: Knowledge transfer: In the initial stage of training, the residual blocks in the feature extraction network are fully open; the parameters of the policy network are frozen and used to train the object detection network; Policy generation: The policy network based on curriculum learning (CL) outputs a routing vector based on the input instance, which is then mapped to the feature extraction network; Link jump: The feature extraction network dynamically turns on or off the residual block inside it according to the mapping results; Policy training: The target localization network uses the intermediate inference parameters to generate detection results, which are input into the reward-penalty function as a key condition; the reward-penalty function then outputs a reward value, which optimizes the decision-making ability of the policy network through back propagation; Object detection: The object detection task is decomposed into two subtasks: candidate region generation and object classification and bounding box regression. RPN outputs a set of prior boxes with higher probability as candidate regions, which are mapped into feature maps of fixed size. These feature maps are used by R-CNN for object classification and precise positioning. Joint training: After the target detection network and the policy network have each reached a certain level of performance, they enter the joint training phase; this phase uses multi-teacher knowledge distillation for joint training, while updating the parameters of the policy network and the detection network.

3. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 2 is characterized by Attention-based multi-teacher feature guidance: In the two-stage object detection network, RPN focuses on the local area near the target object, and the suggestion boxes generated by the multi-teacher network in the RPN stage are filtered by the preset confidence, and only a small number of suggestion boxes are retained for the knowledge transfer process; The feature maps of the student network and the teacher model m are represented as G s and Teacher model m is the mth teacher model in the multi-teacher network; A fully convolutional adaptation layer f a (·) is used to solve the problem of incompatibility of the number of feature channels between the student and teacher models; for the target anchor point position (x, y) on the feature map, the distance between the student network and the teacher model m at the anchor point is measured as The N estimated anchor points on a feature map with a width of W and a height of H are accumulated, and the distillation goal is to minimize the sum of the distance metrics of all anchor points, that is, The feature map output by the teacher model m is z m , and introduce the learning parameter w m and b m ; Through the attention mechanism, the weight of the teacher model m is The total distillation loss is obtained by weighted addition of the distillation losses of each teacher model. Among them, λ is the feature loss weight balancing factor; The hard label loss of the student network is divided into RPN loss and R-CNN loss. The calculation methods of these two parts of loss are the same, including classification loss Cls and regression loss Rls respectively. Binary cross entropy loss L1(,·,) and Smooth L1 loss are used to calculate Cls and Rls; The loss function of RPN is When calculating Cls, the probability that the i-th prior box is predicted as a target is expressed as p i ; Through the GT intersection over union (IoU) judgment: the prior frame contains the target, that is, the prior frame is the foreground. is 1; the prior frame does not contain the detection target, that is, the prior frame is the background. When calculating Rls, the predicted bounding box offset and the real box GT box offset corresponding to the foreground prior box are expressed as q i and When training RPN, N is the number of positive and negative samples sent to R-CNN, of which N / 2 positive samples participate in the training of R-CNN, and R-CNN loss L c ; Then, the total training loss of the student network is L=λ1L gt +λ2L P +λ3L C (11) Among them, the weight balancing factors λ1, λ2, λ3 ≥ 0 satisfy λ1+λ2+λ3=1; the maximization formula (11) prompts the student network to learn the compact knowledge of the multi-teacher detection model.

4. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 3 is characterized by When the ground truth box GT intersection over Union (IoU) > 0.7, it is determined that the prior box contains the target; when IoU < 0.3, it is determined that the prior box does not contain the detected target.

5. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 2 is characterized by CL-based policy network training: In the policy generation, link jumping and policy training stages, the usage rate of residual blocks in the feature extraction network and the prediction accuracy of the intermediate layers of the object detection network should be considered to guide the training of the policy network. RPN gives multiple suggested boxes for the input image. First, the confidence of all suggested boxes and real boxes is calculated and sorted, and the suggested boxes with confidence that meets the requirements are retained. Then, the average confidence between the proposed box and the real box is filtered by the non-maximum suppression (NMS) method. The average confidence is used as the prediction accuracy of the middle layer of the network to measure the error. According to formula (4), the target detection network obtains the expected reward J of formula (5) through the penalty mechanism. The training policy samples are collected from the k-dimensional Bernoulli distribution, and the policy gradient of the policy network parameters is calculated as Reintroduction of self-critical baseline in Is the most likely routing strategy to be selected under the current strategy; using the self-critical baseline Recalculate the policy gradient as 6. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 2 is characterized by Joint fine-tuning with multi-teacher assistance: The parameters of the policy network are first frozen; the detection network is trained by multi-teacher knowledge distillation; The parameters of the detection network are frozen again; the policy network sets the first Kh variables in the policy vector s to 1, and gradually increases h to start CL training; After the student network explores diverse routing strategies, the detection network and the strategy network are jointly fine-tuned with the assistance of the multi-teacher network to achieve the best results.

7. The vehicle-mounted multi-target classification and detection method based on multi-teacher distillation and dynamic reasoning according to claim 1 is characterized by The vehicle-borne multi-target classification and detection method is used for vehicle-borne multi-target detection tasks in an unmanned driving system.

Images