Self-distillation assisted dynamic neural network based autonomous driving target detection method

Abstract

An automatic driving target detection method based on self-distillation assisted dynamic neural network, the collected image is used as the input of the target detection model to realize target classification. The target detection model is based on a self-distillation assisted dynamic neural network, which is composed of a strategy network, a detection network and a hybrid distillation network. The detection network is composed of a series of multi-branch residual blocks. The detection network takes the collected image as the input and outputs the target classification result. The strategy network generates corresponding routing vectors for different inputs to determine the opening and closing of the residual blocks in the detection network. The hybrid distillation network includes two subnetworks. The two subnetworks are differentiated from the detection network in the initial training stage and are randomly initialized. The subnetworks after hybrid distillation are fused into a fusion network. Through knowledge distillation, the knowledge of the fusion network is compressed and migrated back to the detection network, completing a closed-loop optimization.

Description

TECHNICAL FIELD

[0001] The application relates to the application of artificial intelligence technology in automatic driving, in particular to an automatic driving target detection method based on a self-inter-distillation auxiliary dynamic neural network. BACKGROUND

[0002] Deep neural networks (DNNs) play a crucial role in autonomous driving systems, particularly in key areas such as perception, decision-making, and execution. Autonomous driving systems need to process sensor data in real-time and respond quickly. This requires neural network models that balance accuracy and real-time performance. Dynamic neural networks and knowledge distillation can effectively improve the inference speed of neural networks while maintaining model accuracy. Dynamic neural networks can dynamically adjust their structure and parameters based on input data to improve computational efficiency and representation ability. Knowledge distillation improves the performance of student models by transferring knowledge from large and complex teacher models. Combining knowledge distillation and dynamic neural networks can further enhance the performance and adaptability of dynamic networks.

[0003] Existing dynamic neural network solutions mostly rely on pre-trained detection networks based on traditional supervised learning, and their classification ability (in the case of executing full network inference) has been fixed. In the knowledge distillation framework, the student model as a detection network can break through this performance bottleneck by learning the generalization ability and feature representation of the teacher model without increasing the model size. However, in many cases, the detection model deployed on autonomous vehicles or drones cannot obtain guidance from the teacher model or model download and update from the edge network. To adapt to this harsh environment, the detection network as a student must have the ability of self-learning and self-enhancement. This puts higher requirements on the design and strategy of dynamic neural network distillation methods, which are the collaborative optimization of detection networks, reflected in the following aspects:

[0004] (1) Endogenous knowledge transfer and complementarity. In many cases, mobile platforms represented by vehicles or drones are limited by environmental factors and their own computing resources, and cannot download or update models from the edge side or find suitable teacher models for support. As emphasized in [5] and [6], traditional knowledge distillation is difficult to choose a suitable teacher model. From the perspective of vehicle mobility and security threats, mutual learning between vehicles is also difficult to implement in reality. This drives us to explore a self-learning dynamic neural network solution to adapt to the multi-classification tasks of mobile platforms without teacher guidance and infrastructure assistance for model updates.

[0005] (2) Coordination of network structure flexibility and reusability. Dynamic inference requires the detection network to maintain flexibility while ensuring the rationality and effectiveness of the structure. In order to improve inference efficiency and scalability, the design of dynamic network often needs to be modularized, so that the same network module can be reused on different tasks and datasets. LC-Net [7] is a reusable framework that reduces the inference cost of deep neural networks in resource-constrained scenarios by dynamically skipping redundant layers and channels. However, there is a big difference in computing resources among different types of autonomous vehicles and unmanned aerial vehicle platforms. The input instances of mobile platforms also vary greatly in complexity. This requires the detection network to be customizable and able to quickly customize a model suitable for itself. Therefore, designing a flexible and reusable modular structure and ensuring its stability in inference performance under different instances is worth further exploring.

[0006] (3) Trade-off between inference cost and classification accuracy. Dynamic networks need to adjust their structure or parameters during the inference stage to adapt to the clarity of the input image and the change of the category. Alex et al. [8] introduced an additional network layer (called "gate mechanism") to determine the switching of each layer network. This gate mechanism evaluates the current input and past calculation results, and then decides whether to continue more calculation steps. However, the strategy network inference also requires computational cost, and overly complex routing decisions will affect model efficiency. Researchers need to develop new optimization techniques to train networks containing discrete decisions. SUMMARY

[0007] For the environment of updating the model without teacher guidance and infrastructure assistance, the application takes the image collected by the automatic driving vehicle (through the vehicle-mounted image acquisition device) as the input of the vehicle-mounted target detection model, and realizes target detection. The architecture of the target detection model of the application is based on a self-inter-distillation assisted dynamic neural network. The technical solutions of the application are as follows.

[0008] An automatic driving target detection method based on a self-distillation assisted dynamic neural network takes the image collected by the automatic driving mobile platform as the input of the mobile platform target detection model, and realizes target recognition classification;

[0009] The architecture of the target detection model is based on a self-inter-distillation assisted dynamic neural network, which is composed of a strategy network, a detection network, and a hybrid distillation network;

[0010] The detection network is composed of multiple branch residual blocks in series, takes the collected image as the input, and outputs the target classification result, and strengthens the strategy network through back propagation, and the strategy network generates a routing vector according to the input to determine the routing selection of the detection network;

[0011] The mixed distillation network comprises two sub-networks; the two sub-networks are differentiated from the detection network in the initial training stage, and the parameters are randomly initialized; the mixed distillation sub-network is fused into a fusion network; through knowledge distillation, the knowledge (i.e., the parameter weight and feature representation in the model) of the fusion network is compressed and migrated back to the detection network, completing a closed-loop optimization.

[0012] The mobile platform comprises a vehicle or a drone.

[0013] Compared with the prior art, the target detection model of the instance adaptive dynamic neural network architecture based on mixed distillation is adopted, the detection accuracy, inference speed and scalability of the multi-instance classification task are considered under the condition of resource limitation.

[0014] The main contributions and innovative points of the application include:

[0015] Firstly, the application designs a training and parameter fusion framework based on mixed distillation, which comprises two stages of mutual distillation and self-fusion distillation.

[0016] Secondly, in order to realize dynamic inference on the detection network, a lightweight strategy network based on curriculum learning is developed.

[0017] Thirdly, the experimental results on the CIFAR and ImageNet data sets prove that, compared with the traditional mutual learning and ensemble learning, the mixed distillation proposed in the application can more effectively improve the classification performance of the detection network. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 An example adaptive dynamic neural network framework representing hybrid distillation empowerment;

[0019] Figure 2 An example mutual distillation framework;

[0020] Figure 3 An example self-fusion distillation framework;

[0021] Figure 4 An example hybrid distillation framework;

[0022] Figure 5 An example multi-branch residual structure;

[0023] Figure 6 An example policy-detection network joint training framework;

[0024] Figure 7 An example representative picture instance;

[0025] Figure 8 An example routing visualization of representative pictures.

[0026] Figures 1-6 Involving main term translation:

[0027] Policy network based on CL: policy network based on CL.

[0028] Multi-branch residual detection network: multi-branch residual detection network.

[0029] Hybrid distillation network: hybrid distillation network.

[0030] Feature Fusion Module: feature fusion module. DETAILED DESCRIPTION

[0031] The application is further described below in conjunction with the specific embodiments and the accompanying drawings.

[0032] 1 SUMMARY

[0033] For the resource-constrained vehicle target classification task, the application proposes a detection model of instance adaptive dynamic neural network based on mixed distillation, which can adapt to the harsh environment without teacher network guidance and edge network support. In the scene without large teacher model supervision and guidance, the scheme combines diversity enhancement strategies (mutual distillation and self-fusion distillation), integrates and refines the knowledge of the sub-network with the same structure as the detection network into the detection network, and further fuses the dynamic routing to realize the dual optimization of detection accuracy and inference speed.

[0034] The application designs a weakly supervised joint training framework, which realizes the efficient cooperation of the strategy network and the detection network through the joint fine-tuning strategy.

[0035] The experimental results verify that the proposed scheme improves the classification accuracy and reduces the model inference cost compared with other schemes, and the visualization experiment verifies the effectiveness of the dynamic strategy.

[0036] The subsequent content is arranged as follows:

[0037] Section 2 introduces the instance adaptive dynamic neural network architecture assisted by mixed distillation, the detection network and the strategy network it contains, and proposes a joint training method.

[0038] Section 3 describes the performance evaluation simulation experiment.

[0039] Section 4 summarizes the research and discusses the future prospects.

[0040] 2. Architecture and scheme design

[0041] This section introduces the design idea and implementation details of the instance adaptive dynamic neural network assisted by mixed distillation. As shown in Figure 1 The proposed framework consists of a curriculum learning (CL) based strategy network, a multi-branch residual detection network and a mixed distillation network. The mixed distillation network contains two sub-networks (called subnet-1 and subnet-2), which are differentiated from the detection network at the initial training stage and are randomly initialized parameters. The sub-networks after mixed distillation are fused into a more powerful model. Through knowledge distillation, the knowledge of the fused model is compressed and migrated back to the detection network, completing a closed-loop optimization. This scheme relies on a closed-loop optimization mode of self-differentiation, self-supervision and self-enhancement to improve classification performance without the assistance of teacher models or infrastructure to update parameters.

[0042] 2.1 Mixed distillation detection network

[0043] Initially, the mobile platform's dynamic neural network consists of only a policy network and a detection network. During the training preparation phase, the detection network differentiates into two sub-networks with the same structure as itself, and then begins training based on hybrid distillation. The main process includes:

[0044] Mutual Distillation: The two sub-networks learn from each other using the mutual distillation method to enhance each other’s parameter diversity, thereby improving the classification performance of the two sub-networks.

[0045] Self-fusion distillation: Using autoencoders to fuse two enhanced sub-networks, a single fusion network with even greater performance is formed. By fusing the parameters of the two sub-networks, its size and computational complexity are significantly reduced.

[0046] Endogenous knowledge transfer: The fusion network compresses and transfers knowledge back to the detection network of the dynamic neural network through self-fusion distillation.

[0047] Next, we will introduce the implementation details of mutual distillation and self-fusion distillation respectively.

[0048] (1) Mutual distillation

[0049] The two sub-networks in the mutual distillation are denoted as θ1 and θ2, as Figure 2 As shown. Let z c Represents the predicted probability of the network on category c. The prediction result of the network model θ1 is expressed as

[0050]

[0051] Where τ represents the distillation temperature. The regularized cross entropy between the soft labels generated by the network model θ1 and the true labels is calculated as

[0052]

[0053] According to Equation (2), traditional supervised learning enables the network to correctly predict instance labels. In order to improve the generalization performance of θ1, we introduce another peer network θ2 to provide training experience for θ1. The prediction results of these two networks are expressed as σ1(z c ) and σ2(z c ), their matching degree is quantified by KL divergence. From σ1(z c ) to σ2(z c ) is calculated as

[0054]

[0055] The total loss functions of networks θ1 and θ2 are expressed as

[0056]

[0057] and

[0058]

[0059] In this way, sub-networks θ1 and θ2 both master the correct true labels and learn the probability estimates of the parallel networks.

[0060] (2) Self-fusion distillation

[0061] The two sub-networks after mutual distillation are used to generate a fusion model. A self-fusion distillation method is introduced to maximize the knowledge absorption of the two sub-networks. As shown in Figure 3 , the feature maps of θ1 and θ2 are concatenated into a fusion model. Compared with a single sub-network, this fusion model has more diversified parameters and stronger classification performance. Then, the classification knowledge of the fusion model is distilled and transferred to the detection network S0. The feature maps (F 1L and F 2L ) of the Lth convolutional layer of the two sub-networks are concatenated, and the concatenated result is denoted as F e . The auto-encoder ω1 encodes the concatenated feature maps into a meaningful compact feature map, denoted as F f , which has the same size as the feature map of the Lth layer of the detection network.

[0062] The output result of F e is denoted as F f After F e is passed to the fusion classifier, we use the true labels y and z f to supervise the training of the classifier. The soft output of the fusion classifier is denoted as z fusion . Based on equations (1), (2) and (3), the training objective of the fusion classifier is to minimize

[0063] L CE = L f (y,z 2 )+τ KL D e (z f ||z 0L ) (6)

[0064] Self-fusion distillation includes two stages of compressing the knowledge of the fused model and transferring the high-value knowledge therein. The output of the last layer of the detection network is a feature map, denoted as F 0L . We encourage F f to learn the feature distribution from F e . Then, we calculate the feature map of the output of the detection network after the inverse decoding and the concatenated feature map FThe cross entropy loss is used to quantify the effectiveness of the feature map learned by the detection network. Regarding the feature map output by the detection network, the optimization goal is to minimize

[0065]

[0066] where ||·||2 is the norm and r(·) is used to align F 0L and F e channel size.

[0067] Based on Equation (7), the final goal of the detection network is expressed as minimizing

[0068]

[0069] Among them, z0 is the soft output of the detection network, λ f is the weight factor of the fusion feature matching degree.

[0070] 2.2 Detection Network and Policy Network

[0071] Detection network:

[0072] Figure 4 The detection network in is composed of a set of multi-branch residual blocks connected in series. Figure 5 As shown in the figure, a 1×1 convolution kernel is used to reduce the dimensionality of the input, and a multi-branch structure is used to extract data features. Branch one and branch two use convolution kernels of sizes 3×3 and 5×5, respectively. By summing the feature matrices obtained from each branch, the residual block can obtain a sum matrix, and then continue to use the 1×1 convolution kernel to transform the dimension of the sum matrix. Then, using the quick connection in the residual network, the matrix is ​​superimposed with the original features after downsampling to obtain the output of the multi-branch residual block. When y i When input to the i-th multi-branch residual block, the output of the residual block is y i+1 =F i (y i )+y i , as the input of the next residual block.

[0073] For a residual network, skipping a residual block does not result in excessive accuracy loss. Even when some residual blocks are removed, low-dimensional feature information can still be partially retained [9]. Compared with single-path static networks (such as AlexNet

[10] and VGGNet

[11] ), there are many optional paths in the proposed detection network. When the residual module receives the "skip" instruction, the convolution kernel in the residual block will not participate in the reasoning, which is equivalent to y i+1 =y i .

[0074] Policy Network:

[0075] To solve the above problems, a CL-based policy network is introduced to determine the action sequence step by step through exploratory search.

[0076] In the early stage of training, CL enables the model to start learning from easy samples, and gradually progresses to complex samples and knowledge in the later stage of training. According to the complexity, CL assigns different weights to the training samples. In the initial training stage, simple samples are assigned the highest weight, and the weights of more difficult samples are gradually increased. For training samples, each optimization of CL is weighted with different weights. Assuming that the original distribution of the training samples is P(z), the weight given to each sample at the λth optimization is 0≤W λ (z)≤1, where 0≤λ≤1 and W1(z)=1. At the λth optimization, the sample distribution is represented as

[0077]

[0078] where ∫Q λ (z)dz=1. When λ=1, Q1(z)=P(z). Inequality (10) is used to ensure that the information entropy of Q λ (z) is monotonically increasing;

[0079]

[0080] Inequality (11) is used to ensure that W λ (z) is monotonically non-decreasing.

[0081]

[0082] 2.3 Weakly supervised joint training

[0083] Mixed distillation enhances the diversity of detection network parameters and the upper limit of classification accuracy. The vector generated by the policy network determines the routing of the detection network, and the classification results of the detection network are used to strengthen the policy network through backpropagation. After CL training, the model learns a variety of routing strategies, but the detection accuracy is slightly lower than performing complete model inference. This drives us to explore a weakly supervised (teacher-free) joint training framework. The purpose is to improve the synergy of dynamic routing and detection network, and balance detection accuracy and inference speed. Figure 6 The workflow of the proposed joint training is given, including:

[0084] · Knowledge transfer: In the initial training stage, the detection network is divided into two subnetworks with the same structure, and a mixed distillation network is used to enhance the diversity of detection network parameters.

[0085] · Strategy generation: The CL-based policy network is responsible for extracting the features of the input instance and outputting the routing vector, where the CL training requires the joint participation of the policy and detection networks;

[0086] • Linkage jump: The guided multi-branch residual network of policy vector realizes dynamic jump through shortcut connection;

[0087] • Dynamic inference: The residual block in detection network is selectively skipped to realize dynamic inference;

[0088] • Collaborative training: The detection network fine-tunes its parameters according to the classification results, and inputs the results into the reward function in CL. The output reward value is back-propagated to the policy network, prompting it to learn a more suitable route.

[0089] Unlike random depth

[12] , the selection of residual blocks in the proposed scheme is controlled by the policy network to enhance the matching degree with the detection task instance. For an input image, the policy network outputs all residual network routes at once, i.e., predicting all actions of the detection network, which is essentially a single-step Markov decision process given the input state. Let s k ∈[0,1] be the kth element in the policy vector s, whose value represents the probability of the kth residual block being turned on. Given an image x and a pre-trained detection network consisting of K multi-branch residual blocks, the policy for selecting residual blocks is defined as a K-dimensional Bernoulli distribution, i.e.

[0090]

[0091] The policy network here is described as a function f(x; W) with respect to image x and weight W. The output of the image x after being processed by the activation function σ(x) = 1 / (1+e -x ) is represented as

[0092] s = f(x; W) (13)

[0093] A lightweight Resnet-8 model is used to construct the policy network. The inference overhead is largely determined by the number of convolution kernels. Due to the small number of convolution kernels, the inference cost generated by the policy network accounts for only 8% of the overall network. The policy network generates an action vector u according to s, which determines the blocks involved in inference. The 0-1 variable u k corresponds to the kth element in u, u k = 1(u k = 0) represents turning on (off) the kth residual block.

[0094] We quantify the benefits brought by the action vector u by customizing a reward function, and guide the policy network to find a route with high accuracy and low cost. The training objective around this reward function is to minimize the length of the detection network route (block usage) as much as possible while ensuring that the predicted results are consistent with the true labels. The reward function is defined as

[0095]

[0096] where, represents the proportion of the residual blocks that are turned on in the whole detection network. When the result is correctly predicted, the shorter the route length, the more positive reward is given to encourage the policy network to skip more residual blocks. γ is used to punish the prediction with low similarity to the teacher network, balancing the inference speed and detection accuracy. Selecting more residual blocks is beneficial to improve the detection accuracy, but will reduce the inference speed. We maximize the reward expectation, i.e.

[0097]

[0098] to obtain the optimal policy network parameters.

[0099] In summary, the route generated by formula (13) determines which blocks in the detection network perform forward propagation to predict the classification result; the policy network calculates the reward value according to whether the prediction is correct and the number of blocks used.

[0100] As a traditional reinforcement learning method, the policy gradient method samples from a multi-dimensional distribution to obtain the policy gradient, which is often used to search the gradient that can maximize formula (15). Unlike this, the present application collects training policy samples from a k-dimensional Bernoulli distribution. For u k ∈{0,1}, the policy gradient is calculated as

[0101]

[0102] In small batches of samples, Monte Carlo sampling is used to obtain the expected gradient of formula (16).

[0103] After the detection network is trained by hybrid distillation, the upper limit of the detection accuracy and the parameter diversity are improved. After the CL training of the dynamic model framework, a diversified routing strategy can be learned, but the detection accuracy is slightly lower than that of performing complete model inference. To this end, after the CL training is completed, the policy network and the multi-branch residual network must be jointly fine-tuned to balance the detection accuracy and the inference speed. The execution of joint training is summarized as algorithm 1. The policy network sets the first K-h variables in the strategy vector s to 1 and gradually increases h, starting the CL training (lines 9-12). After the model is trained by CL, a diversified routing strategy is explored (line 17). To further optimize the model, the policy network and the detection network are jointly trained (lines 18-21).

[0104]

[0105]

[0106] 3 Experimental design and result analysis

[0107] To evaluate the detection and inference performance of the proposed method under different image complexity and instances, three authoritative datasets, CIFAR-10

[13] , CIFAR-100

[13] and IMAGE-NET

[14] , are selected. The CIFAR dataset contains 60,000 32x32 RGB images, of which 50,000 and 10,000 are used for training and testing. The ImageNet dataset contains 1.2 million training images of 1000 classes, of which 50,000 are used as a validation set to test the top-1 accuracy.

[0108] The proposed method is implemented under the PyTorch platform, and the ADAM optimizer is used to train the model. The model training server is configured with an Intel i9-13900k CPU and an NVIDIA GeForce RTX 4090 GPU. The distillation temperature τ in equation (1) is set to 2, and the hyperparameter λ in equation (8) is set to 10. The learning rate is 1x10-3. In the training of the policy network (CL), the batch size is set to 2048. In joint training, the batch size is adjusted to 256, and the learning rate is adjusted to 1x10-5. f

[0109] Two multi-branch residual detection networks with similar number of convolution kernels to Resnet50

[15] and Resnet110

[15] are constructed, named Inception15 and Inception54. As the basic student detection network, they are composed of 15 and 54 multi-branch residual blocks respectively. We build detection networks with this and test the effectiveness of dynamic routing.

[0110] 3.1 Analysis of the effectiveness of mixed distillation

[0111] ​The first group of experiments analyzes the promotion effect of mixed distillation on the classification accuracy of the detection network. The mixed distillation experiment results on CIFAR-100 are shown in Table 1. The initial detection network without distillation (named: nodistillation), the subnetworks after mutual distillation (named: subnet-1 and subnet-2), the network after self-fusion (named: fusion), and the detection network enhanced by mixed distillation (named: hybrid distillation) are compared to verify the effectiveness of the proposed mixed distillation. Among them, the classification accuracy of fusion is better than that of subnet-1 and subnet-2, which shows that fusing the feature maps of the two subnetworks is very helpful to the classification accuracy. However, this improvement requires the preservation of all subnetwork parameters, increasing the storage and inference cost. Therefore, the knowledge in fusion is compressed and migrated back to the detection network through knowledge distillation. On the Resnet20, Resnet50 and DenseNet40 models, the classification accuracy of hybrid distillation is improved by 2.44%, 1.81% and 2.59% respectively compared with nodistillation.

[0112] Table 1 Performance gain brought by mixed distillation

[0113]

[0114] The present experiment selects traditional knowledge distillation (named: KD), mutual learning (named: DML) and ensemble learning (named: ONE) as benchmark methods, and compares them with the mixed distillation method proposed in the present application. As shown in Table 2, under ResNet-32, the classification accuracy of the proposed mixed distillation method is 71.70%, higher than that of KD, DML, ONE, which is 70.32%, 71.15%, 71.07%, proving the superiority of mixed distillation.

[0115] Table 2 Comparison of mixed distillation and benchmark distillation methods on CIFAR100

[0116]

[0117] Further experiments are conducted on ImageNet dataset to demonstrate the effectiveness of the proposed method facing complex image inputs. As shown in Table 3, the hybrid distillation on ResNet50 achieves 70.9% top-1 accuracy, which is better than DML and ONE. Compared with directly supervised training with ResNet50, the proposed hybrid distillation method can obtain a gain of 1.2%. This gain is better than other distillation models. The proposed scheme obtains satisfactory results on CIFAR and Imagenet datasets with different complexities, indicating that the proposed hybrid distillation method has the potential to be applied to datasets containing other types of instances.

[0118] Table 3 Hybrid distillation and benchmark distillation methods on ImageNet

[0119]

[0120] 3.2 Dynamic routing effectiveness analysis

[0121] The second set of experiments compares the proposed dynamic routing with early- exit

[16] and random depth

[17] . Table 4 lists the results of Iception15 and Iception54 on CIFAR10 and CIFAR100 datasets, respectively, where Acc represents the model detection accuracy, and L represents the average residual block usage of the model.

[0122] Suppose the average length of the output routing of the proposed method is L1. The residual block usage of the early- exit and random depth models is set to , which is the upper bound of L1, to ensure that the upper limit of the benchmark model detection ability is not weaker than the proposed method. When running the early- exit model, the first residual blocks are kept on. When running the random depth model, randomly select residual blocks and keep them on.

[0123] On CIFAR-10 dataset, the average length of CL trained Inception15 dynamic routing is 10.6, achieving 88.1% average accuracy, which is 72.9% and 68.8% higher than early exit and random depth, respectively. It is worth noting that when running Inception54, nearly 15% of the images use less than 10 blocks, and some even less than 3. These results confirm that the proposed CL method not only improves the classification accuracy but also significantly reduces the inference cost. Neither pruning, random depth nor early exit can achieve fine-grained dynamic adjustment of neural networks. Next, we examine the performance improvement of joint training. On the CIFAR10 dataset, the Inception15 and Inception54 models trained jointly are not only 3.5% and 17.1% more accurate than the models relying solely on CL, but also 3.2 and 3.9 less in average routing length, proving the efficiency and practicality of joint training.

[0124] Table 4 Influence of routing strategy on detection accuracy

[0125]

[0126] The proposed strategy network outputs the complete routing vector each time. The model inference process does not need to refer to the intermediate output results, which helps to reduce the strategy execution cost. To verify this inference, this subsection compares the proposed dynamic routing strategy with the one-step decision-making benchmark inference method (named Single), which uses a traditional reinforcement learning training strategy network. To ensure fairness, all methods are configured with the same number of residual blocks to observe the differences in inference speed under the same accuracy.

[0127] Table 5 Influence of routing strategy on inference speed

[0128]

[0129] Table 5 summarizes the average inference delay and speedup effect of different models on CIFAR-10 data. In the case of achieving the same detection accuracy as Iception15, the proposed scheme Proposed improves the detection speed by 14.6% compared to Full-Net. On the other hand, Single reduces the detection speed by 28.9% compared to Full-Net, which is due to the one-step decision-making inference method of Single, which brings additional computational cost to multi-step inference, resulting in negative acceleration. These results confirm that the "one-time generation of routing" of the proposed strategy network has a significant improvement on the model inference speed.

[0130] 3.3 Dynamic routing visualization analysis

[0131] The present experiment aims to visualize the routing selection of dynamic neural networks during inference. The present application demonstrates the dynamic routing strategy of the Iception15 model on ImageNet. As shown in Figure 7 The present application selects 8 representative picture examples from the Image-Net dataset, and the ID and true label of the picture are shown below the picture. The above-mentioned 8 picture examples are input into the dynamic neural network model, and the routing vector generated by the strategy network is visualized as Figure 8 . Among them, the horizontal axis represents the number of multi-branch residual blocks, and the vertical axis is the picture ID; gray (blank) represents the residual block participating (not participating) in routing.

[0132] As can be seen from Figure 8 , in the face of different input picture instances, the dynamic neural network can give diverse routing strategies, which proves the effectiveness and generalization of the dynamic routing strategy of the present application.

[0133] 4. Conclusion

[0134] The present application proposes an instance adaptive dynamic neural network method based on hybrid distillation. The method can dynamically select the optimal routing path in the multi-branch residual network according to the specific characteristics of the input instance.

[0135] In the early stage of training, the present application adopts a hybrid distillation strategy combining mutual distillation and self-fusion distillation to enhance the basic classification ability of the detection network.

[0136] Subsequently, a strategy network is trained, which is responsible for generating a routing strategy suitable for the multi-branch residual network. This strategy significantly reduces the inference cost of the network while maintaining high detection accuracy. The present application also jointly fine-tunes the strategy network with the multi-branch residual network to increase the diversity of the routing strategy, thereby further improving the detection accuracy and inference speed of the model.

[0137] The experimental results on the CIFAR and ImageNet datasets show that the hybrid distillation proposed by the present application can significantly improve the performance of the detection network without the guidance of a large teacher model. In addition, the visualization experiment further confirms the sparsity characteristics exhibited by the network channels when processing different images, which verifies the effectiveness and adaptability of the dynamic routing method. Under the premise of maintaining the same inference cost, the proposed method surpasses the mainstream benchmark methods such as early exit, random depth and pruning in terms of accuracy level. After proper customization, our method has the potential to support most existing target classification applications, providing intelligent support for mobile platforms, thereby promoting the development of intelligent mobile systems.

[0138] References

[0139] [1] Han Y, Huang G, Song S, et al. Dynamic neural networks: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(11): 7436-7456.

[0140] [2] Hinton G, Vinyals O, Dean J. Distilling the Knowledge in a Neural Network[J]. Stat, 2015, 1050: 9.

[0141] [3] Li W, Wang J, Ren T, et al. Learning accurate, speedy, lightweight CNNs via instance-specific multi-teacher knowledge distillation for distracted driver posture identification[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10): 17922-17935.

[0142] [4] An S, Liao Q, Lu Z, et al. Efficient semantic segmentation via self-attention and self-distillation[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(9): 15256-15266.

[0143] [5] Jin X, Peng B, Wu Y, et al. Knowledge distillation via route constrained optimization[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 1345-1354.

[0144] [6] Cho J H, Hariharan B. On the efficacy of knowledge distillation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 4794-4802.

[0145] [7] Xia W, Yin H, Dai X, et al. Fully dynamic inference with deep neural networks [J]. IEEE Transactions on Emerging Topics in Computing, 2021, 10(2): 962-972.

[0146] [8] Graves A. Adaptive Computation Time for Recurrent Neural Networks [J]. 2016.

[0147] [9] Huang G, Sun Y, Liu Z, et al. Deep networks with stochastic depth [C] / / Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer International Publishing, 2016: 646-661.

[0148]

[10] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks [J]. Advances in Neural Information Processing Systems, 2012, 25.

[0149]

[11] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition [C] / / 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.

[0150]

[12] Yim J, Joo D, Bae J, et al. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 4133-4141.

[0151]

[13] Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images [J]. 2009.

[0152]

[14] Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database [C] / / 2009 IEEE Conference on Computer Vision and Pattern Recognition. Ieee, 2009: 248-255.

[0153]

[15] He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.

[0154]

[16] Figurnov M, Collins M D, Zhu Y, et al. Spatially adaptive computation time for residual networks [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1039-1048.

[0155]

[17] Huang G, Sun Y, Liu Z, et al. Deep networks with stochastic depth [C] / / Computer Vision - ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14. Springer International Publishing, 2016: 646-661.

Claims

1. A self-distillation-assisted dynamic neural network-based object detection method for autonomous driving. This method uses images collected by an autonomous driving mobile platform as input to the platform's object detection model to achieve object recognition and classification. Its characteristics are The architecture of the object detection model is based on a self-distillation-assisted dynamic neural network, which consists of a policy network, a detection network and a hybrid distillation network; The detection network is composed of a series of multi-branch residual blocks. It takes the collected image as input and outputs the target classification result. The policy network is strengthened by backpropagation. The policy network generates a routing vector based on the input and determines the routing selection of the detection network. The hybrid distillation network consists of two subnetworks, which are differentiated from the detection network in the initial training stage and have randomly initialized parameters. The subnetworks that have undergone hybrid distillation are fused into a fusion network. Through knowledge distillation, the knowledge of the fusion network is compressed and transferred back to the detection network, completing a closed-loop optimization.

2. The automatic driving target detection method according to claim 1, characterized in that The mobile platform includes a vehicle or a drone.

3. The automatic driving target detection method according to claim 1, characterized in that During the training of a dynamic neural network: a. In the initial stage, the dynamic neural network only contains a policy network and a detection network; b. During the training preparation phase, the detection network differentiates into two sub-networks with the same structure as itself; c. In the training phase based on hybrid distillation, the process includes: 1) Mutual distillation: The two sub-networks learn from each other using the mutual distillation method to obtain an enhanced sub-network; 2) Self-fusion distillation: Use autoencoders to fuse the two enhanced sub-networks into a fusion network; 3) Endogenous knowledge transfer: The fusion network compresses knowledge through self-fusion distillation and transfers it back to the detection network.

4. The automatic driving target detection method according to claim 3, characterized in that In the mutual distillation of step 1), The two sub-networks in the mutual distillation are denoted as θ1 and θ2; let z 1,c and z 2,c Represent the predicted probabilities of θ1 and θ2 on category c respectively; The prediction result of θ1 is expressed as Where τ represents the distillation temperature, c' represents the category index, and C represents the set of categories; The regular cross entropy formula of the predicted result and the true label of category c is where y c represents the true label of category c; z c Represents the predicted probability of the subnetwork on category c; Then, the regular cross entropy between the soft label generated by θ1, that is, the predicted result and the true label of category c is In order to improve the generalization performance of θ1, another equivalent sub-network θ2 is introduced to provide training experience for θ1; The same method as θ1 is used to obtain the prediction result σ2(z 2,c ), use formula (2) to get the regularized cross entropy of the soft label generated by θ2 and the real label of category c σ1(z 1,c ) and σ2(z 2,c ) is quantified by KL divergence; from σ1(z 1,c ) to σ2(z 2,c ) is calculated as Similarly: The total loss functions of θ1 and θ2 are expressed as and Then, θ1 and θ2 not only grasp the correct true labels, but also learn the probability estimates of the parallel network.

5. The automatic driving target detection method according to claim 3, characterized in that In the self-fusion distillation of step 2), the feature maps of the two sub-networks θ1 and θ2 are connected in series to form a fusion network; the classification knowledge of the fusion network Feature Fusion is refined and infused into the detection network; specifically: The feature map F of the Lth convolutional layer of θ1 and θ2 1L and F 2L are connected, and the result after series connection is recorded as F e ; The autoencoder ω1 in the fusion network converts F e Encoded into a meaningful compact feature map F f ; F f The size of is the same as the feature map of the Lth layer of the detection network; F e The output of the fusion network is expressed as F f After being passed to the fusion classifier, the true label y and the predicted probability z are used e The ensemble classifier is trained under supervision; the soft output of the ensemble classifier is represented as z f ; The training goal of the fusion classifier is to minimize the fusion loss L fusion =L CE (y,z f )+τ 2 D KL (With e ||with f ) (6) Self-fusion distillation consists of two stages: compressing the fused model knowledge and transferring the high-value knowledge in the model knowledge. The detection network finally outputs the feature map F 0L , calculate F after the inverse decoder ω2 0L With F e The cross entropy loss is used to quantify the effectiveness of the feature maps learned by the detection network; Regarding the feature map output by the detection network, the optimization goal is to minimize F 0L With F e Cross entropy loss where ||·||2 is the norm and r(·) is used to align F 0L and F e Channel size; Based on Equation (7), the final goal of the detection network is expressed as minimizing in, represents the overall loss of the detection network, z0 is the soft output of the detection network, and λ f is the weight factor of the fusion feature matching degree.

6. The automatic driving target detection method according to claim 1, Its characteristics are that for the detection network: The 1×1 convolution kernel in the residual block is used to reduce the dimensionality of the input, and the multi-branch structure is used to extract data features. Branch 1 and branch 2 use convolution kernels of sizes 3×3 and 5×5, respectively. The feature matrices obtained from each branch are summed to obtain a sum matrix, and then the 1×1 convolution kernel is used to transform the dimension of the sum matrix. Using the shortcut connection in the residual network, the sum matrix and the downsampled original features are superimposed to obtain the output of the multi-branch residual block; the output y of the previous residual block is i When input to the i-th multi-branch residual block, the output of the residual block is y i+1 =F i (y i )+y i , as the input of the next residual block; The adjustment of the detection network architecture is determined by routing selection. That is, under the guidance of the strategy vector, the multi-branch residual network realizes dynamic routing through the shortcut connection in the residual structure; when the residual block receives the "skip" instruction, the convolution kernel in the residual block will not participate in the reasoning, that is, y i+1 =y i .

7. The automatic driving target detection method according to claim 1, Its characteristic is that the policy network is a policy network based on curriculum learning CL; for the policy network: During training, CL selects the last h residual blocks from all K residual blocks for training. In the first round, h is 1. As h increases, it gradually associates and optimizes the switching of more residual blocks until all blocks are covered when h equals K. When the policy network associates h residual blocks, it keeps the first Kh blocks open and only learns the switching strategy for the last h blocks. At the beginning of training, start learning from easy samples; In the middle and late stages of training, the training samples and knowledge are gradually advanced to complex ones. CL assigns different weights to training samples based on their complexity. In the initial training phase, easy samples are assigned the highest weights, and the weights of difficult samples are gradually increased. For each training sample, CL uses different weights for each optimization. Assume that the original distribution of the training samples is P(z), and the weight given to each sample during the λth optimization is 0≤W λ (z)≤1, where 0≤λ≤1 and W1(z)=1; at the λth optimization, the sample distribution is expressed as where ∫Q λ (z)dz=1; when λ=1, Q1(z)=P(z); Inequality (10) is used to ensure that Q λ The information entropy of (z) increases monotonically, Inequality (11) is used to ensure that W λ (z) monotonically non-decreasing, 8. The automatic driving target detection method according to claim 1, characterized in that A weakly supervised joint training method is used to improve the synergy between dynamic routing and detection networks, balancing detection accuracy and inference speed. The steps of joint training include: Knowledge transfer: In the initial training phase, the detection network is differentiated into two sub-networks with the same structure, and a hybrid distillation network is used to enhance the diversity of detection network parameters. Policy generation: The CL-based policy network extracts features of the input instance and outputs a policy vector as the routing vector of the detection network. CL training is jointly performed by the policy network and the detection network. Link jump: The strategy vector guides the multi-branch residual network to achieve dynamic jumps through shortcut connections; Dynamic reasoning: Residual blocks in the detection network are selectively skipped to achieve dynamic reasoning; Collaborative training: The detection network adjusts its parameters based on the classification results and uses the results as the input of the reward function in the CL; the reward value output by the reward function is back-propagated to the policy network, prompting it to learn more appropriate routes.

9. The autonomous driving target detection method according to claim 8, wherein for joint training: For an input image, the policy network outputs all residual network routes at once, that is, predicting all actions of the detection network. This is a single-step Markov decision process given the input state. Order s k ∈[0,1] is the kth element in the strategy vector s, and its value represents the probability that the residual block k is turned on. Given an image x and a pre-trained detection network consisting of K multi-branch residual blocks, the strategy for selecting the residual block is defined as a K-dimensional Bernoulli distribution, i.e. The policy network here is described as a function f(x; W) about the image x and the weight W; the image x is activated by the activation function σ(x) = 1 / (1+e -x ) The output after processing is represented as s=f(x;W)(13) The lightweight Resnet-8 model is used to build the policy network; the policy network generates an action vector u based on s and determines the block involved in reasoning; the 0-1 variable u k Corresponding to the kth element in u, u k =1 and u k =0 represents turning on and off the kth residual block respectively; The reward function is customized to quantify the benefits of the action vector u and guide the policy network to find a high-precision and low-cost route. The reward function is defined as in, Represents the proportion of enabled residual blocks in the entire detection network. When the result is predicted correctly, the shorter the route length, the more positive rewards are given to encourage the policy network to skip more residual blocks. γ is used to penalize predictions with low similarity to the teacher network, balancing inference speed and detection accuracy; Selecting more residual blocks is beneficial to improving detection accuracy, but it will reduce the reasoning speed. The optimal policy network parameters are obtained by maximizing the reward expectation. The reward expectation is expressed as Then, the routing generated by formula (13) determines which blocks in the detection network perform forward propagation to predict the classification results; the policy network calculates the reward value based on whether the prediction is correct and the number of blocks used; The training strategy samples are collected from the k-dimensional Bernoulli distribution, for u k ∈{0,1}, the policy gradient is calculated as In the mini-batch samples, Monte Carlo sampling is used to obtain the expected gradient of Equation (16).

Images