Automatic driving multi-target detection method based on instance adaptive dynamic neural network

Abstract

Aiming at the multi-instance detection task in automatic driving, the application provides an automatic driving multi-target detection method based on an instance adaptive dynamic neural network, first, an image is captured by a vehicle-mounted device; then, the image as an instance is detected and the target in the image is recognized by a dynamic neural network model. The construction process of the dynamic neural network model is as follows: first, a strategy network training framework based on curriculum learning (CL) is developed, which is used for extracting task features and generating a routing vector matched with the task attribute; then, the routing vector is mapped into a detection network composed of multiple branch residual blocks, which is used for guiding the opening and closing of the residual blocks; finally, a joint training framework is constructed, the routing vector generated by the strategy network is used to adjust the detection network routing and optimize the parameters; the output of the detection network is fed back to the strategy network, so as to optimize the generation of the routing vector. The method takes the instance adaptive dynamic neural network method as the core, and considers the detection accuracy and reasoning speed.

Description

Technical Field

[0001] The present invention relates to a machine vision technology that can be used for multi-instance detection tasks in autonomous driving, and specifically to a multi-target detection method for autonomous driving based on an instance-adaptive dynamic neural network. Background Art

[0002] With the rapid development of the Internet of Vehicles, a series of in-vehicle applications covering information services, driving safety, and traffic efficiency will emerge. Self-driving cars represent the mainstream of future intelligent transportation and are expected to occupy about 75% of vehicles on the road by 2035. It is estimated that a single self-driving car will generate about 1G of data per second

[31] , which includes a large number of computationally intensive and delay-sensitive services. However, considering the limited processing power of in-vehicle equipment and the low latency requirements of in-vehicle services, traditional neural network detection frameworks are unable to meet the requirements of real-time performance and demand differentiation.

[0003] There are two main ways to improve the inference speed of the model. One is to improve the compactness of the model through model pruning [1-3] or to generate a compact student network through knowledge distillation [4-6]. The other is to use weight quantization [7] to improve the inference response speed. However, these two methods have some inherent defects when dealing with multi-target detection tasks. First, they permanently remove layers and neurons with low activity to build a dense network, but these removed neurons may be very important for accurately classifying complex input instances. Although the model inference speed is improved, the inference accuracy is weakened. Second, even layers and neurons with high activity may have limited help in improving the detection effect when processing some different types of input instances. For example, pedestrians do not have the headlights of trucks, and trucks do not have the limbs of pedestrians, and the degree of feature overlap between the two is minimal. Therefore, the convolution filters related to truck features are negligible for predicting pedestrians. Finally, in reality, most simple images only require shallow embedding to meet the classification accuracy requirements, and only a small number of complex image classifications rely on deep network embedding.

[0004] Dynamic neural networks[8] are a potential solution that can selectively perform the subset of computations required for accurate classification based on input attributes. Figure 1For example, dynamic neural networks primarily consist of a policy network and a detection network. Faced with different instances, the neural network automatically adjusts its structure, improving inference speed without sacrificing recognition accuracy. Unlike static pruning, which permanently removes neurons, this dynamic inference mode temporarily suppresses computation based on input instances. Due to its superior inference efficiency and expressiveness, dynamic neural networks have opened up new avenues for addressing the timeliness and safety issues of autonomous driving object detection. However, they still face some unique challenges, including the following:

[0005] (1) Taking into account both the lightweight and high efficiency of the policy network. The detection network is usually composed of several preset modules, and its network routing (i.e., the opening and closing of blocks) is determined by the policy network reasoning [9-11]. Following this idea, Leroux et al.

[12] used parameter sharing and adaptive computation time (ACT) to reduce the number of parameters and adaptive execution layers. Odena et al. introduced a controller based on reinforcement learning, adaptively constructing a computation graph from submodules

[13] to check filter activation. Liu et al.

[14] improved the control module in the policy network and selectively activated specific modules through Q-learning to achieve dynamic routing. These methods all use reinforcement learning to train the policy network to achieve the "dynamic switching" of preset modules, but the computationally intensive reinforcement learning may offset the advantages of dynamic neural networks.

[0006] (2) Balancing inference speed and detection accuracy. The inference speed of a dynamic neural network is determined by the size of the policy network model and the depth of the detection network. Detection accuracy does not increase linearly with the depth of the detection network. Fignov et al.

[15] used ACT to implement a dynamic Resnet network for specific instances, but it only allowed the continuous use of shallow Resnet predefined modules. When extracting features, using only the shallow layers of the neural network will reduce detection accuracy

[16] . Fusion of shallow and deep features [17,18] can help reduce accuracy loss, but at a higher computational cost.

[0007] (3) Jointly Debugging the Policy Network and the Detection Network. Pruning the detection network under the guidance of the policy network helps reduce inference costs. However, there is a dependency between specific instances and filters. Simply pruning specific filters will inevitably lead to a mismatch between the policy network and some input instances, thereby reducing detection accuracy. It is worth further exploring how to give neural networks fine-grained tuning capabilities and implement instance-adaptive dynamic routing of neural networks. Summary of the Invention

[0008] For multi-instance detection tasks in autonomous driving, this paper proposes an instance-adaptive dynamic neural network method that dynamically selects detection network routes based on task features extracted by a policy network, balancing detection accuracy and inference speed. The main innovations of this paper include:

[0009] First, to address the aforementioned problem (1), we construct a lightweight policy network training framework based on curriculum learning (CL). Unlike the traditional "one-step-one-decision" dynamic routing model, this network directly outputs a policy vector that matches the task attributes, while reducing the inference cost of both the policy network and the detection network.

[0010] Second, to address the aforementioned problem (2), a detection network consisting of multi-branch residual blocks is developed to customize routing for tasks based on the strategy vector, thereby enhancing the flexibility of the multi-target detection network architecture and the adaptability to instance features.

[0011] Third, to address the aforementioned issue (3), we construct a joint training framework for dynamic routing to further improve detection accuracy and speed. The policy vector generated by the policy network is used to adjust the detection network architecture and optimize parameters. Simultaneously, the output of the detection network is fed back to the policy network to optimize routing selection.

[0012] Experimental results on the CIFAR and ImageNet datasets demonstrate the effectiveness and superiority of the proposed method. At the same inference cost, the proposed method achieves higher detection accuracy than early-falling and random depth. At the same accuracy level, the proposed method reduces inference latency by 50% and 59% compared to early-falling and pruning. Compared to routing generation based solely on CL, the proposed joint training achieves an 18% improvement in accuracy and a 16% improvement in speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of dynamic reasoning of dynamic neural networks for simple and complex examples.

[0014] Figure 2 It is a schematic diagram of the multi-branch residual network dynamic routing joint training framework of the present invention.

[0015] Figure 3 It is a schematic diagram of the course training process of the strategy network of the present invention.

[0016] Figure 4 Schematic diagram of the multi-branch residual structure of the detection network of the present invention.

[0017] Figure 5(a) and Figure 5(b) show the detection accuracy and average FLOPs of each method on different datasets in the experiment, where:

[0018] Figure 5(a) shows the detection accuracy and average FLOPs on the CIFAR dataset;

[0019] Figure 5(b) shows the detection accuracy and average FLOPs on the ImageNet dataset.

[0020] Figure 6 This is a schematic diagram of the probability distribution of route length in the experiment.

[0021] Figure 7(a) and Figure 7(b) are representative examples of low FLOPs and high FLOPs in the experiment, where:

[0022] Figure 7(a) shows the low FLOPs group;

[0023] Figure 7(b) shows the high FLOPs group.

[0024] Figure 8(a) and Figure 8(b) are visualization diagrams of the routing strategies for low-FLOPs and high-FLOPs images in the experiment, where:

[0025] Figure 8(a) shows the low FLOPs group;

[0026] Figure 8(b) shows the high FLOPs group. DETAILED DESCRIPTION

[0027] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] 1 Overview

[0029] Aiming at the multi-instance detection task in autonomous driving, this paper proposes an instance-adaptive dynamic neural network method that balances detection accuracy and inference speed.

[0030] First, a lightweight policy network training framework based on curriculum learning (CL) is developed to extract task features and generate routing vectors matching the task attributes.

[0031] The routing vectors are then mapped into a detection network consisting of multi-branch residual blocks to guide the opening and closing of residual blocks to adapt to the diversity of instances.

[0032] Finally, under the constructed joint training framework, the routing vectors generated by the policy network are used to adjust the detection network routing and optimize parameters; the output of the detection network is fed back to the policy network to optimize the generation of routing vectors.

[0033] Experimental results on the CIFAR and ImageNet datasets show that, compared to early-falling and pruning, the proposed method reduces inference latency by 50% and 59% while maintaining the same accuracy. The proposed dynamic routing strategy can adapt to images of varying complexity, balancing detection accuracy and inference cost.

[0034] 2 Design Plan

[0035] This section introduces the design ideas and implementation details of the adaptive dynamic neural network. Figure 2 As shown in Figure 1, the proposed dynamic reasoning framework consists of a policy network based on curriculum learning (CL) and a multi-branch residual detection network. The workflow after the image captured by the vehicle device is input into the policy network and the detection network includes:

[0036] The policy network extracts features of the input instance and outputs a policy vector.

[0037] Under the guidance of the strategy vector, the multi-branch residual network dynamically jumps through shortcut connections to form a detection network route.

[0038] Based on whether the result is correct or not, the detection network generates a reward function and back-propagates it to the policy network to optimize the policy network so that it produces a more appropriate policy vector.

[0039] Next, we will explain the design of the policy network, the multi-branch residual detection network, and the joint training framework for dynamic routing.

[0040] 2.1 CL-based policy network design

[0041] The policy network helps the pre-trained detection network find the best combination of residual blocks to achieve accurate classification with the least number of blocks. A detection network composed of k residual blocks has 2 k The on / off strategy of the blocks, that is, there are 2 k -1 optional route. As can be seen, the dimension of the policy search space grows exponentially with the number of residual blocks. If the initial search strategy is randomly set when the dimension is large, policy network training will become extremely difficult and may even fail to converge.

[0042] To address the above issues, a CL-based policy network training strategy is designed, which uses a progressive search mode to gradually determine the action sequence. During training, CL selects h residual blocks from all k residual blocks for training. In the first round, h is 1. As h increases, the switching of more residual blocks is gradually associated and optimized until all blocks are covered when h equals k. When associating h residual blocks, the policy network keeps the first kh blocks open and only learns the switching strategy of the last h blocks, accelerating model convergence while reducing training cost. This method optimizes the "on and off" of each block based on the input features to evaluate the utility of the block. Faced with different instances, the policy network trained by CL can flexibly identify and skip redundant residual blocks.

[0043] by Figure 3 Take the training process of CL as an example. CL allows the model to start learning from easy samples and gradually advance to complex samples and knowledge. According to the complexity, CL assigns different weights to training samples. In the initial training stage, simple samples are assigned the highest weights, and the weights of more difficult samples are gradually increased. CL is an optimization sequence and can be regarded as a special form of continuous learning. For each optimization of CL, different weights are used for weighting. Assuming that the original distribution of the training samples is P(z), the weight assigned to each sample during the λth optimization is 0≤W λ (z)≤1, where 0≤λ≤1 and W1(z)=1. At the λth optimization, the sample distribution is expressed as

[0044]

[0045] where ∫Q λ (z)dz=1. When λ=1, Q1(z)=P(z). Inequality (2) is used to ensure that Q λ The information entropy of (z) increases monotonically;

[0046]

[0047] Inequality (3) is used to ensure that W λ (z) Monotonically non-decreasing.

[0048]

[0049] 2.2 Multi-branch residual detection network construction

[0050] As the basic component of the detection network, the multi-branch residual block

[20] consists of residual connections and multi-branch structures. Figure 4As shown in the figure, a 1×1 convolution kernel is used to reduce the dimensionality of the input, and a multi-branch structure is used to extract data features. Branch one and branch two use convolution kernels of size 3×3 and 5×5 respectively, and branch three is connected to the pooling layer. By summing the feature matrices obtained from each branch, the residual block can obtain a sum matrix, and then continue to use the 1×1 convolution kernel to transform the dimension of the sum matrix. Then, through the shortcut connection (short path) in the residual network, the matrix is ​​superimposed with the original features after downsampling to obtain the output of the multi-branch residual block. When y i When input to the i-th multi-branch residual block, the output of the residual block is y i+1 =F i (y i )+y i , as the input of the next residual block.

[0051] The multi-branch residual detection network consists of k multi-branch residual blocks connected in series. The feature extraction of the multi-branch residual blocks combines convolution and pooling operations, ensuring efficient feature extraction while preserving the key features of the data. This design reduces the number of network parameters and the cost of model inference.

[0052] For a residual network, skipping a residual block will not bring too much accuracy loss. Even when some residual blocks are removed, low-dimensional feature information can still be partially retained

[24] . Compared with single-path static networks (such as AlexNet

[22] and VGGNet

[23] ), there are many optional paths in the proposed detection network. Inspired by this, the present invention realizes the adjustment of the detection network architecture by selecting paths. Specifically, under the guidance of the strategy vector, the multi-branch residual network realizes dynamic routing through shortcut connections in the residual structure. When the residual module receives the "skip" instruction, the convolution kernel in the residual block will not participate in the reasoning, which is equivalent to y i+1 =y i .

[0053] However, skipping more residual blocks during inference will inevitably increase classification errors. Therefore, choosing the best residual block combination and network routing for different instances is crucial, as it determines both inference speed and detection accuracy.

[0054] 2.3 Joint Training for Dynamic Routing

[0055] The vectors generated by the policy network determine the routing of the detection network; the classification performance of the detection network, in turn, optimizes the generation of the policy vectors through backpropagation. These two networks jointly determine routing selection, which motivates this paper to explore a joint training framework for the policy and detection networks to achieve instance-adaptive dynamic routing.

[0056] Unlike random depth

[24] , the selection of residual blocks in the joint training framework is controlled by the policy network to enhance the matching between the policy network and the detection task instance. In this framework, the policy vector outputs a string of binary decision variables to indicate whether to retain or delete a residual block, where each decision to delete a residual block corresponds to deleting a path from the network. During the joint training process, the residual block specified by the policy vector is run in the detection network to generate rewards based on the usage of the residual block and the prediction accuracy of the detection network.

[0057] The policy network outputs all residual network routes at once, that is, predicts all actions of the detection network. It is essentially a single-step Markov decision process given an input state. Given an image x and a pre-trained detection network consisting of K multi-branch residual blocks, the strategy for selecting residual blocks is defined as a K-dimensional Bernoulli distribution, that is

[0058]

[0059] The policy network is described as a function f(x; W) about the image x and the weight W. The image x is activated by the activation function σ(x) = 1 / (1+e -x ) The output after processing is represented as

[0060] s=f(x;W) (5)

[0061] The policy network is constructed using a lightweight Resnet-8 model. The inference overhead is largely determined by the number of convolution kernels. Due to the small number of convolution kernels, the inference overhead generated by the policy network accounts for only 8%. Let s k ∈[0,1] is the kth element in the policy vector s, and its value represents the probability that the residual block k is turned on. The policy network generates an action vector u based on s and determines the block involved in reasoning. 0-1 determines the variable u k Corresponding to the kth element in u, u k =1(u k =0) represents turning on (off) the kth residual block.

[0062] The present invention develops a reward function to quantify the benefits of the action vector u and guide the policy network to find a high-precision and low-cost route. Specifically, during the training process, the present invention minimizes the length of the detection network route (the use of blocks) while ensuring that the prediction results are consistent with the true labels. The reward function is formalized as

[0063]

[0064] in, Represents the proportion of residual blocks that are enabled in the entire detection network; when the result is correctly predicted, the shorter the route length, the more positive rewards are given to encourage the policy network to skip more residual blocks. γ is used to penalize incorrect predictions and balance inference speed and detection accuracy. Generally, the more residual blocks are selected, the higher the detection accuracy is, but the inference speed will be reduced. This invention maximizes the reward expectation, that is,

[0065]

[0066] To obtain the optimal policy network parameters for training.

[0067] In summary, the routing strategy generated by Equation (5) determines which blocks in the detection network perform forward propagation to generate prediction results. At the same time, the policy network calculates the reward value based on whether the prediction is correct and the number of blocks used.

[0068] The policy gradient method

[25] is used to search for the gradient that maximizes (7). Traditional reinforcement learning samples from multiple distributions to obtain policy gradients. In contrast, the present invention samples training policy samples (selecting the residual block strategy) from a k-dimensional Bernoulli distribution. For u k ∈{0,1}, the policy gradient is calculated as:

[0069]

[0070] Monte Carlo sampling is used to obtain the expected gradient of Eq. (8) in the mini-batch. These gradient estimates are unbiased. Self-critical baseline is used in formula (8) to reduce the variance. Based on this, we get

[0071]

[0072] in, is the routing strategy most likely to be selected under the current strategy. i <1,u i =1, otherwise u i =0.

[0073] In order to encourage the policy network to further explore, this paper introduces the weight factor α∈[0,1] and updates formula (5) to

[0074] s=α·s+(1-α)·(1-s) (10)

[0075] Make s i ∈[1-α,α], so as to sample diverse policy vectors.

[0076] ​After CL training, the model can learn a variety of routing strategies, but the detection accuracy is slightly lower than that of performing full model inference. To address this, after CL training, the policy network and the multi-branch residual network are jointly fine-tuned to simultaneously optimize detection accuracy and inference speed. The joint training algorithm is summarized in Algorithm 1. The policy network sets the first Kh variables in the policy vector s to 1 and gradually increases h to begin CL training (Lines 7-9). After CL training, the model explores a variety of routing strategies (Line 14). To further optimize the model, the policy network and the detection network are jointly trained (Lines 15-17).

[0077]

[0078] 3 Experimental design and result analysis

[0079] To evaluate the detection and inference performance of the proposed method, this experiment selected three authoritative datasets: CIFAR-10

[27] , CIFAR-100

[27] , and IMAGENET

[28] . The CIFAR dataset contains 60,000 32×32 RGB images, of which 50,000 and 10,000 images are used for training and testing respectively. The ImageNet dataset contains 1.2 million training images from 1,000 categories, of which 50,000 images are used as a validation set to test the top-1 accuracy.

[0080] In this experiment, the proposed algorithm is implemented in the PyTorch platform, and the ADAM optimizer is used to train the model. During the model training process, the weight factor α (in formula (10)) is set to 0.8, and the learning rate is 1×10 -4 In the policy network (CL) training, the batch size is set to 2048. In the joint training, the batch size is adjusted to 256 and the learning rate is adjusted to 1×10 -5 .

[0081] In this experiment, we constructed two multi-branch residual detection networks, named Inception15 and Inception54, with similar numbers of convolution kernels to Resnet50

[21] and Resnet110

[21] . As basic static models, they consist of 15 and 54 multi-branch residual blocks, respectively. This experiment generates dynamic routing based on these basic models to demonstrate the effectiveness of the proposed solution.

[0082] 3.1 Routing Strategy Effectiveness Analysis

[0083] The first set of experiments compares the proposed dynamic routing with the early retirement network

[15] and random depth

[24] . Table 1 lists the results of Iception15 and Iception54 on the CIFAR10 and CIFAR100 datasets, respectively. Acc represents the model detection accuracy; L represents the average number of residual blocks used by the model.

[0084] Assume that the mean length of the route output by the proposed method is L. The number of residual blocks used in the early withdrawal network model and the random depth model is set to That is, the upper bound of L, to ensure that the upper limit of the baseline model's detection ability is not weaker than the proposed method. When running the early exit network model, keep the previous When running the random depth model, randomly select residual blocks and keep them turned on.

[0085]

[0086]

[0087] On CIFAR-10, the average dynamic routing length of the Inception-15 trained with CL was 9.4, achieving an average accuracy of 88.6%. This represents improvements of 72% and 68.1% compared to the early-leaving mechanism and random depth, respectively. Notably, when running with Inception-54, nearly 15% of images used fewer than 10 blocks, with some using fewer than 3. These results confirm that the proposed CL method significantly reduces inference cost while improving detection accuracy. Neither pruning, distillation, nor early-leaving methods can achieve fine-grained dynamic adjustments. Next, we examine the performance improvement achieved by joint training. On the CIFAR-10 dataset, the jointly trained Inception-15 and Inception-54 models not only improved detection accuracy by 2.7% and 18.2% compared to models relying solely on CL, but also reduced routing length by an average of 2.5 and 3.2, respectively, demonstrating the efficiency and practicality of joint training.

[0088] The proposed policy network outputs a complete routing vector each time. The model inference process does not need to refer to intermediate output results, which helps reduce the cost of policy execution. To verify this inference, this section compares the proposed routing generation strategy with the following baseline method: a training policy network using traditional reinforcement learning

[15] , using a one-step-one-decision reasoning model (named Single).

[0089] To ensure fairness, all methods use the same number of residual blocks to observe the differences in inference speed of different methods when the detection capability is the same (achieving the same accuracy).

[0090] Table 2 Impact of routing strategy on inference speed

[0091]

[0092] Table 2 summarizes the average inference latency and speedup of different models on the CIFAR-10 dataset. While achieving the same detection accuracy on Iception15, the proposed solution improves detection speed by 14.9% compared to Full-Net. Meanwhile, Single reduces detection speed by 28.7% compared to Full-Net. This is due to Single's decision-making inference method, which uses a one-step-per-decision approach. Multi-step reasoning incurs additional computational overhead, resulting in negative speedup. These results confirm that the proposed policy network's ability to generate all selected actions simultaneously significantly improves model inference speed.

[0093] 3.2 Dynamic Inference Model Acceleration Performance

[0094] The second set of experiments adjusted the parameter γ in R(u) to balance route length and detection accuracy, observing the optimal balance under different detection requirements. Each point on the curves in Figures 5(a) and 5(b) represents a set of model parameters for a given γ. The average FLOPs generated by classifying images in the test set was used to evaluate model complexity. Three benchmark algorithms were selected to compare model acceleration performance:

[0095] ACT

[15] : Early exit mechanism, automatically exiting model inference when the confidence requirement is met.

[0096] SACT

[15] : adds a gating function to ACT, using different inference depths for different regions of the image (such as background, contour, etc.).

[0097] PFEC

[30] : Model pruning, which reduces the parameters of convolutional layers by pruning static networks.

[0098] Figure 5(a) shows the average FLOPs and detection accuracy of different methods on the CIFAR dataset. Compared with the ResNet-110 model, the best model of the proposed method (corresponding to the point pointed by the arrow in the figure) improves the accuracy by 0.4% on average (93.6% vs. 93.2%) and reduces the FLOPs by 65% ​​on average (1.73×10 8 5.08×10 8 The proposed method achieves the same 93.0% detection accuracy as ACT, using only 50% of the FLOPs. At the same 93.6% detection accuracy, the proposed method reduces FLOPs by 59% compared to PFEC. PFEC can be used in conjunction with the framework of the present invention to accelerate convolution operations.

[0099] Figure 5(b) shows the average FLOPs and detection accuracy of different methods on the ImageNet dataset. The best model of the proposed scheme outperforms ResNet-110 in detection accuracy (76.8% vs. 76.4%) and is 6% faster (1.47×10 10 For 1.56×10 10 FLOPs). After slightly reducing the detection accuracy (see the rectangular box in the figure), the performance of the dynamic inference framework is comparable to that of the full ResNet110, with a 20% reduction in inference cost (1.25×10 10 For 1.56×10 10 FLOPs). A 20% speedup without sacrificing detection accuracy is quite significant. For example, in a high-precision image recognition service that receives 1 billion API calls per day, the method proposed in this paper saves approximately 1,000 hours of computation (0.024 seconds per image) on a single P6000 GPU.

[0100] 3.3 Impact of Instance Complexity on FLOPs

[0101] The above results show that the proposed scheme incurs significantly different computational costs when classifying images of varying complexity. Images with distinct features typically require fewer network embeddings than complex or atypical images. This section analyzes the impact of instance complexity on routing length to understand the relationship between routing length and image complexity. To facilitate observation of dynamic reasoning, this set of experiments collected 10,000 test instances with FLOPs and recorded the routing length used for each test instance. Figure 6 The probability density of the routing length is given. In the case where the detection network consists of 15 multi-branch residual modules, the test instance uses an average of 6.8 multi-branch residual blocks.

[0102] Using an average routing length of 6.8 as a threshold, test examples with fewer than seven residual blocks were selected as the low-FLOPs group, while those with seven or more were selected as the high-FLOPs group. As shown in Figures 7(a) and 7(b), eight representative images were selected from the low-FLOPs and high-FLOPs groups, respectively, for visualization analysis. The ground-truth labels are shown below each image. The visual differences between the two groups are quite distinct. The low-FLOPs images have clearer and more complete features, making objects easier to identify. In contrast, the high-FLOPs images have incomplete outlines and atypical features, making them difficult to distinguish. For example, in the high-FLOPs group, example b lacks the complete outline of the car, showing only the rear end, while example c only shows the back half. In the remaining examples from the high-FLOPs group, the outlines are blurred and the contrast with the background is low, increasing the probability of recognition errors.

[0103] Next, we visualize the routes generated for the two groups of images in Figures 7(a) and 7(b). The route vectors correspond to Figures 8(a) and 8(b), where the horizontal axis represents the residual module number and the vertical axis represents the image number. Gray and blank areas represent residual blocks that participated in and did not participate in the routing, respectively. As can be seen, the routing strategies of the two groups of images exhibit different characteristics. First, the routing strategies for instances of the same category within each group are relatively similar, suggesting that features of the same category are stored in similar filters. Second, the low-FLOPs group uses a lower average number of residual modules than the high-FLOPs group, which is consistent with the proposed inference that image complexity is positively correlated with route length, with complex images incurring more inference overhead. Finally, images that are easily distinguishable or have clear features require shorter routes, while atypical or blurry images rely on longer routes.

[0104] 4 Conclusion

[0105] This paper proposes a multi-branch residual network dynamic routing detection framework based on CL. This architecture can be extended to most existing mainstream automatic object detection frameworks. It is a method for dynamically selecting multi-branch residual network routing based on input examples. This paper trains a policy network to generate multi-branch residual network routing policies, significantly reducing network inference costs while maintaining detection accuracy. The policy network is also fine-tuned jointly with the multi-branch residual network to increase the diversity of routing policies, further improving the detection accuracy and inference speed of the model.

[0106] References

[0107] [1] Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through networks limming. ICCV, 2017.

[0108] [2] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Softfilter pruning for accelerating deep convolutional neural networks. IJCAI, 2018.

[0109] [3]Yihui He,Xiangyu Zhang,and Jian Sun.Channel pruning foraccelerating very deep neural networks.ICCV,2017.

[0110] [4]Jimmy Ba and Rich Caruana.Do deep nets really need to be deep?InNeurIPS,2014.

[0111] [5]Geoffrey E.Hinton,Oriol Vinyals,and Jeffrey Dean.Distilling theknowledge in a neural network.arXiv:1503.02531,2015.

[0112] [6]Adriana Romero,Nicolas Ballas,Samira Ebrahimi Kahou,AntoineChassang,Carlo Gatta,and Yoshua Bengio.Fitnets:Hints for thin deep nets.InICLR,2015.

[0113] [7]Benoit Jacob,Skirmantas Kligys,Bo Chen,Menglong Zhu,Matthew Tang,Andrew Howard,Hartwig Adam,and Dmitry Kalenichenko.Quantization and trainingof neural networks for efficient integer-arithmetic-only inference.In CVPR,2018.

[0114] [8]Yizeng Han,Gao Huang,Shiji Song,Le Yang,Honghui Wang,and Yulinwang.Dynamic Neural Networks:ASurvey.arXiv:2102.04906,2021.

[0115] [9]E.Bengio,P.-L.Bacon,J.Pineau,and D.Precup.Conditional computationin neural networks for faster models.In ICML Workshop on Abstraction inReinforcement Learning,2016.

[0116]

[10] L.Liu and J.Deng.Dynamic deep neural networks:Optimizingaccuracy-efficiency trade-offs by selective execution.arXiv:1701.00299,2017.

[0117]

[11] L.Denoyer and P.Gallinari.Deep sequential neural network.arXiv:1410.0510,2014.

[0118]

[12] S.Leroux,P.Molchanov,P.Simoens,B.Dhoedt,T.Breuel,and J.Kautz,“IamNN:Iterative and adaptive mobile neural network for efficient imageclassification,”in Proc.ICLR Workshop,2018.

[0119]

[13] A.Odena,D.Lawson,and C.Olah,“Changing model behavior at test-timeusing reinforcement learning,”in Proc.ICLR Workshop,2017.

[0120]

[14] L.Liu and J.Deng,“Dynamic deep neural networks:Optimizingaccuracy-efficiency trade-offs by selective execution,”in Proc.AAAIConference on Artificial Intelligence,2018.

[0121]

[15] M.Figurnov,M.D.Collins,Y.Zhu,L.Zhang,J.Huang,D.Vetrov,andR.Salakhutdinov.Spatially adaptive computation time for residual networks.InCVPR,2017.

[0122]

[16] A.Veit,M.J.Wilber,and S.Belongie.Residual networks behave likeensembles of relatively shallow networks.In NIPS,2016.

[0123]

[17] M.McGill and P.Perona.Deciding how to decide:Dynamic routing inartificial neural networks.In ICML,2017.

[0124]

[18] G.Huang,D.Chen,T.Li,F.Wu,L.van der Maaten,andK.Q.Weinberger.Multi-scale dense convolutional networks for efficientprediction.arXiv preprint arXiv:1703.09844,2017.

[0125]

[19] X.Gao,Y.Zhao, Dudziak,R.Mullins,and C.-Z.Xu,“Dynamic channelpruning:Feature boosting and suppression,”in Proc.International Conference onLearning Representations,2019.

[0126]

[20] SU Tian-tian,WANG Hui-min,ZHANG Xiao-feng.Research on LightweightImage Classification Algorithm Based on Multi-Branch BottleneckStructure.Acta Electronica Sinica,2023

[0127]

[21] HE K,ZHANG X,REN S,et al.Deep residual learning for imagerecognition.Proceedings of the IEEE Confer-ence on Computer Vision andPattern Recognition.Piscat-away:IEEE,2016.

[0128]

[22] A.Krizhevsky,I.Sutskever,and G.E.Hinton.Imagenet classificationwith deep convolutional neural networks.In NIPS,2012.

[0129]

[23] K.Simonyan and A.Zisserman.Very deep convolutional networks forlarge-scale image recognition.arXiv preprint arXiv:1409.1556,2014.

[0130]

[24] G.Huang,Y.Sun,Z.Liu,D.Sedra,and K.Q.Weinberger.Deep networks withstochastic depth.In ECCV,2016.

[0131]

[25] R.S.Sutton and A.G.Barto.Reinforcement learning:Anintroduction.MIT press Cambridge,1998.

[0132]

[26] SJRennie,E.Marcheret,Y.Mroueh,J.Ross,and V.Goel.Self-criticalsequence training for image captioning.In CVPR,2017.

[0133]

[27] A.Krizhevsky and G.Hinton.Learning multiple layers of featuresfrom tiny images.2009.

[0134]

[28] J.Deng,W.Dong,R.Socher,L.-J.Li,K.Li,and L.FeiFei.Imagenet:Large-scale hierarchical image database.In CVPR,2009.

[0135]

[29] M.Figurnov,MDCollins,Y.Zhu,L.Zhang,J.Huang,D.Vetrov,andR.Salakhutdinov.Spatially adaptive computation time for residual networks.InCVPR,2017.

[0136]

[30] H.Li,A.Kadav,I.Durdanovic,H.Samet,and HPGraf.Pruning filtersfor efficient conventions.In ICLR,2017.

[0137]

[31] LIU Lei,CHEN Chen,FENG Jie,XIAO Ting-ting,Qing-qi,“A Survey ofComputation Offloading in Vehicular Edge Computing Networks,”in ACTAELECTRONICA SINICA,2021.

Claims

1. A multi-target detection method for autonomous driving based on an instance-adaptive dynamic neural network. First, an on-board device captures an image. Then, a dynamic neural network model detects and identifies targets in the image as an instance. The method is characterized by: The construction process of the dynamic neural network model is as follows: 1) Design a lightweight policy network based on curriculum learning (CL) to extract task features and generate routing vectors that match task attributes; 2) The routing vector is mapped to a detection network consisting of k multi-branch residual blocks connected in series, which is used to guide the opening and closing of each multi-branch residual block; "multi-branch residual block" is referred to as "block" for short; the detection network has 2 k The on / off strategy of the seed block, that is, there are 2 k -1 optional route; 3) Establishing a joint training framework in which routing vectors generated by the policy network are used to adjust detection network routing and optimize parameters; the output of the detection network is fed back to the policy network to optimize the generation of routing vectors; In step 1), the CL-based policy network design: The policy network is constructed from a neural network model; the policy network helps the pre-trained detection network find the best combination of blocks; The policy network is trained using the CL method: h blocks are selected from all k blocks in the detection network for training. In the first round, h is 1. As h increases, the switching of more blocks is gradually associated and optimized until all blocks are covered when h equals k. When the policy network associates h residual blocks, it keeps the first kh blocks open and only learns the switching strategy for the last h blocks. Assign different weights to training samples based on their complexity: In the initial training phase, simple samples are assigned the highest weights, and the weights of samples are gradually increased from easy to difficult. For the training samples, each optimization of CL is weighted with different weights; Assume that the original distribution of the training samples is P(z), and the weight given to each sample during the λth optimization is 0≤W λ (z)≤1, where 0≤λ≤1 and W1(z)=1; at the λth optimization, the distribution of training samples is expressed as: where ∫Q λ (z)dz=1; when λ=1, Q1(z)=P(z); Using inequality (2) to ensure Q λ The information entropy H(Q λ ) monotonically increasing; Use inequality (3) to ensure W λ (z) monotonically non-decreasing; In step 2), for the k blocks connected in series, y i Input to the i-th block to get F i (y i ), put y i+1 =F i (y i )+y i As the input of the i+1th block; if the block is closed, the reasoning process of the detection network skips the block and goes to the next open block to obtain an inference route; where the subscript i represents the block number; In step 3), a joint training framework is used to jointly train the strategy network and the detection network; The selection of blocks in the joint training framework is controlled by a policy network. The policy vector outputs a series of binary decision variables to indicate whether to keep or delete a block, where each decision to delete a block corresponds to removing a path from the detection network. During the joint training process, the blocks specified by the policy vector are run in the detection network to generate rewards based on the block usage and the prediction accuracy of the detection network. The policy network outputs all the inference routes of the detection network at once to predict all the actions of the detection network, which can be regarded as a single-step Markov decision process given a given input state: Given an image x and a pre-trained detection network consisting of K patches, the strategy for selecting patches is defined as a K-dimensional Bernoulli distribution, i.e. The policy network is described as a function f(x; W) about the image x and the weight W; the image x is activated by the activation function σ(x) = 1 / (1+e -x ) The output after processing is represented as the strategy vector s s=f(x;W) (5) Order s k ∈[0,1] is the kth element in the strategy vector s, and its value represents the probability that the kth block is opened; The policy network generates an action vector u based on s and determines the block involved in reasoning; Let 0-1 determine the variable u k Corresponding to the kth element in u, u k =1 and u k =0 represents the opening and closing of the kth block respectively; The reward function is designed to quantify the benefits brought by the action vector u and guide the policy network to find a better reasoning route. On the basis of ensuring that the prediction results are consistent with the true labels, the length of the detection network route is minimized as much as possible. The reward function is formalized as in, Indicates the ratio of enabled blocks to all blocks in the entire detection network. When the prediction result is correct, the shorter the route length, the more positive rewards are given to encourage the policy network to skip more blocks. γ is used to penalize incorrect predictions. The optimal policy network parameters are obtained by maximizing the reward expectation. The reward expectation is expressed as: In summary, the routing strategy generated by formula (5) determines which blocks in the detection network perform forward propagation to generate prediction results; at the same time, the policy network calculates the reward value based on whether the prediction is correct and the number of blocks used; The policy gradient method is used to search for the gradient that maximizes the reward expectation. For u k ∈{0,1}, the policy gradient is calculated as: Monte Carlo sampling is used to obtain the expected gradient of Equation (8), the self-critical baseline is used in formula (8) to reduce the variance, and we get in, is the routing strategy most likely to be selected under the current strategy; if and only if 0.5 i <1,u i =1, otherwise u i =0;​ Introduce the weight factor α∈[0,1] and update formula (5) to s=α·s+(1-α)·(1-s) (10) Make s i ∈[1-α,α], to sample diverse policy vectors.

2. The method for multi-target detection in autonomous driving based on instance-adaptive dynamic neural network according to claim 1 is characterized by: The detection network consists of a multi-branch residual block, and the structure of the multi-branch residual block is as follows: First, a 1×1 convolution kernel is used to reduce the dimensionality of the input; Then, a multi-branch structure is used to extract the data features of the results of the previous step; one of these branches is connected to the pooling layer, and the remaining branches are convolution kernels; Next, the characteristic matrices obtained from each branch structure are summed to obtain a sum matrix; Then, use a 1×1 convolution kernel to transform the dimension of the sum matrix; Finally, the matrix obtained by dimension transformation in the previous step and the original features after downsampling are superimposed through the shortcut connection short path and output.

3. The method for multi-target detection in autonomous driving based on instance-adaptive dynamic neural network according to claim 1 or 2, characterized in that The policy network is constructed by Resnet-8.

Images