Multi-task detection method, electronic device, vehicle and medium

Abstract

The application provides a multi-task detection method, an electronic device, a vehicle and a medium. The multi-task detection method comprises: acquiring multi-task perception data; inputting the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein the lightweight multi-task model is obtained by optimizing a baseline multi-task model based on the sensitivity level of each unit; and the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information. By optimizing the baseline multi-task model based on the sensitivity level of each unit and using different optimization methods for different units in the model, the multi-task model can be more lightweight, the deployment time (i.e. the occupied computing resources) can be reduced, and the multi-task detection efficiency and the accuracy of the detection result can be improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a multi-task detection method, electronic device, vehicle, and medium. Background Technology

[0002] In the overall autonomous driving technology solution, perception, as the most fundamental link, often needs to output multiple types of information to downstream processes (planning and control). After receiving perception data (such as data from cameras, LiDAR, and millimeter-wave radar), the perception model needs to output multiple pieces of information simultaneously. For example, by detecting or identifying various types of elements in the road—lane lines, traffic lights, signs, road markings, and drivable areas—each element detection or identification can be considered a perception task. Through a multi-task model, these tasks share a backbone network and utilize the feature maps obtained from it. Based on these feature maps, different tasks then perform their respective processing, thereby improving overall detection efficiency. However, multi-task models are complex, time-consuming to deploy, and consume significant computational resources. Related technologies employ model lightweighting to reduce the computational load or size of multi-task models. Lightweighting methods for multi-task models typically involve pruning the entire model or performing Neural Architecture Search (NAS). However, since each module in the model structure has a different sensitivity level, applying the same processing method to each module, such as pruning every module or performing NAS on the entire model, can lead to insufficient or excessive lightweighting of the multi-task model, affecting detection efficiency and results. Summary of the Invention

[0003] This invention provides a multi-task detection method, electronic device, vehicle, and medium to address the shortcomings of traditional multi-task model lightweighting methods that use the same lightweighting approach for each unit or module in the model structure, which affects the model output effect and output efficiency.

[0004] This invention provides a multi-task detection method, comprising:

[0005] Acquire multi-task awareness data;

[0006] The multi-task perception data is input into the trained lightweight multi-task model to output the detection information corresponding to each task;

[0007] The lightweight multi-task model is obtained by optimizing the baseline multi-task model based on the sensitivity level of each unit.

[0008] The baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information.

[0009] According to a multi-task detection method provided by the present invention, the method for calculating the sensitivity hierarchy of each unit includes:

[0010] Calculate the sensitivity of each layer;

[0011] Take the average sensitivity of each layer contained in each block, and determine the sensitivity level of the corresponding block based on the average layer sensitivity;

[0012] Calculate the proportion of blocks in each unit whose sensitivity level is lower than a preset threshold, and determine the sensitivity level of each unit based on the calculated proportion of blocks.

[0013] or,

[0014] Calculate the sensitivity of each block;

[0015] The sensitivity of each block contained in each unit is averaged, and the sensitivity level of each unit is determined based on the average block sensitivity.

[0016] According to a multi-task detection method provided by the present invention, the optimization of the baseline multi-task model based on the sensitivity level of each unit includes:

[0017] If the proportion of blocks with the first sensitivity level in a certain unit exceeds the proportion threshold, then the unit will be pruned.

[0018] If the proportion of blocks with sensitivity level 1 in a certain unit does not exceed the proportion threshold, and the proportion of blocks with sensitivity level 1 and 2 in a certain unit exceeds the proportion threshold, then the unit is optimized by model search.

[0019] If the proportion of blocks with sensitivity levels of first and second level in a certain unit does not exceed the aforementioned proportion threshold, then the unit will not be optimized.

[0020] According to a multi-task detection method provided by the present invention, the baseline multi-task model includes a backbone network unit, wherein the proportion of blocks with a sensitivity level of the first level in the backbone network unit does not exceed a proportion threshold, and the proportion of blocks with a sensitivity level of the first level and the second level exceeds the proportion threshold. The method further includes model search optimization of the backbone network unit, comprising:

[0021] The backbone network units in the baseline multi-task model are converted into a supernet structure based on multiple search dimensions.

[0022] The supernet structure is trained;

[0023] Search for the optimal subnet in the trained supernet structure and use the optimal subnet as the new backbone network unit of the baseline multi-task model.

[0024] By training a baseline multi-task model that includes new backbone network units, a multi-task model with lightweight backbone network units is obtained.

[0025] According to a multi-task detection method provided by the present invention, the step of searching for the optimal subnet in the trained supernet structure includes:

[0026] Randomly generate all subnets that meet preset constraints;

[0027] The inference result of each subnet is compared with the manually calibrated result to calculate the inference loss, and the inference loss value corresponding to each subnet is obtained. The subnet with the smallest inference loss value is selected as the best subnet.

[0028] According to a multi-task detection method provided by the present invention, the backbone network units in the baseline multi-task model are converted into a supernet structure, including:

[0029] Based on the sensitivity of each layer in the backbone network unit, the corresponding layer of the backbone network unit is converted into a subnet in the supernet structure.

[0030] According to a multi-task detection method provided by the present invention, the baseline multi-task model includes multiple detection heads, and the proportion of blocks with a sensitivity level of the first level in the multiple detection heads exceeds the proportion threshold. Model pruning is performed on the multiple detection heads, including:

[0031] Set the pruning rate of the layer to be pruned for each detection head;

[0032] The convolution kernels and filters of the layer to be pruned are pruned according to the pruning rate;

[0033] Calculate the pruning rate of each convolutional kernel and the pruning rate of the filter;

[0034] If the pruning rate of the convolutional kernels in the current layer exceeds the first preset threshold, then all convolutional kernels in the current layer will be pruned.

[0035] If the pruning rate of the current layer's filters exceeds the second preset threshold, then all filters in the current layer will be pruned to obtain a multi-task model with a lightweight detection head.

[0036] The multi-task detection method provided by the present invention further includes:

[0037] A lightweight multi-task model is obtained based on the multi-task model with lightweight backbone network units and the multi-task model with lightweight detection head.

[0038] The lightweight multi-task model is fine-tuned and trained to obtain a well-trained lightweight multi-task model.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-task detection method as described above.

[0040] The present invention also provides a vehicle including the electronic equipment described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the obstacle depth detection method as described above.

[0042] The multi-task detection method, electronic device, vehicle, and medium provided by this invention acquire multi-task perception data; input the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein, the lightweight multi-task model is obtained by optimizing a baseline multi-task model based on the sensitivity level of each unit; the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information, and is optimized based on the sensitivity level of each unit. Different optimization methods are used for different units in the model, which can make the multi-task model more lightweight, reduce deployment time and computational resource consumption, and improve multi-task detection efficiency and detection result accuracy. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is one of the flowcharts of the multi-task detection method provided by the present invention;

[0045] Figure 2 This is the second flowchart of the multi-task detection method provided by the present invention;

[0046] Figure 3 This is the third flowchart of the multi-task detection method provided by the present invention;

[0047] Figure 4 This is the fourth flowchart of the multi-task detection method provided by the present invention;

[0048] Figure 5 This is a functional structure diagram of the lightweight multi-task model provided by the present invention;

[0049] Figure 6This is a schematic diagram of the structure of the multi-task detection device provided by the present invention;

[0050] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Figure 1 A flowchart of the multi-task detection method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the multi-task detection method provided in this embodiment of the invention includes:

[0053] Step 101: Obtain multi-task awareness data;

[0054] Step 102: Input the multi-task perception data into the trained lightweight multi-task model to output the detection information corresponding to each task; wherein, the lightweight multi-task model is obtained by optimizing the baseline multi-task model based on the sensitivity level of each unit; the baseline multi-task model is trained based on the multi-task perception training data and multi-task target detection information.

[0055] In some embodiments of the present invention, the method for calculating the sensitivity level of each unit includes:

[0056] Calculate the sensitivity of each layer;

[0057] In this embodiment of the invention, the sensitivity analysis method for the layer is as follows:

[0058] First, sort the channel weights of this layer, then prune the channel with the smallest weight. Calculate the percentage increase in loss on the test set compared to the model before pruning; this percentage is the sensitivity of this layer. For example, if the loss before pruning is loss_1 and the loss after pruning is loss_2, then the sensitivity is (loss_2 - loss_1) / loss_1.

[0059] The sensitivity of each layer contained in each block is averaged, and the sensitivity level of the corresponding block is determined based on the average layer sensitivity. A block refers to a structure composed of multiple layers stacked together and cut or connected. A unit refers to multiple blocks stacked together and becoming part of the model architecture, such as backbone, decoder, neck, head, etc.

[0060] Calculate the proportion of blocks in each unit whose sensitivity level is lower than a preset threshold, and determine the sensitivity level of each unit based on the calculated proportion of blocks.

[0061] The baseline multi-task model is optimized based on the sensitivity level of each unit, including:

[0062] If the proportion of blocks with the first sensitivity level in a certain unit exceeds the proportion threshold, then the model of that unit will be pruned.

[0063] If the proportion of blocks with sensitivity level 1 in a certain unit does not exceed the proportion threshold, and the proportion of blocks with sensitivity level 1 and 2 in a certain unit exceeds the proportion threshold, then model search optimization is performed on that unit.

[0064] If the proportion of blocks with sensitivity levels of first and second level in a certain unit does not exceed the aforementioned proportion threshold, then that unit will not be optimized.

[0065] For example, if a unit has n blocks and k blocks have sensitivity levels of 1 and 2, accounting for k / n, if the percentage exceeds 80%, the unit will proceed to the next round of analysis (model pruning or model search optimization); otherwise, the unit will neither be pruned nor subjected to model search optimization.

[0066] In some embodiments of the present invention, the method for calculating the sensitivity level of each unit includes:

[0067] Calculate the sensitivity of each block;

[0068] In this embodiment of the invention, the channel weights of each layer in the block are sorted, and then the channel with the smallest weight value in each layer is pruned. The percentage increase in the inference result and the ground truth loss of the model on the test set after pruning compared to before pruning is calculated, and this percentage is the sensitivity of that layer. For example, if the loss before pruning is loss_1 and the loss after pruning is loss_2, then the sensitivity is (loss_2-loss_1) / loss_1.

[0069] The sensitivity of each block contained in each unit is averaged, and the sensitivity level of each unit is determined based on the average block sensitivity.

[0070] The baseline multi-task model is optimized based on the sensitivity level of each unit, including:

[0071] If the proportion of blocks with the first sensitivity level in a certain unit exceeds the proportion threshold, then the model of that unit will be pruned.

[0072] In this embodiment of the invention, if the calculated sensitivity of a block is <15%, the block's sensitivity level is the first level; if the block's sensitivity is <60%, the block's sensitivity level is the second level; and if the block's sensitivity is >60%, the block's sensitivity level is the third level. It should be noted that this application does not limit the division of sensitivity levels, and those skilled in the art can set them according to actual needs.

[0073] If the proportion of blocks with sensitivity level 1 in a certain unit does not exceed the proportion threshold, and the proportion of blocks with sensitivity level 1 and 2 in a certain unit exceeds the proportion threshold, then model search optimization is performed on that unit.

[0074] If the proportion of blocks with sensitivity levels of first and second level in a certain unit does not exceed the aforementioned proportion threshold, then the unit will not be optimized.

[0075] For example, if the proportion of blocks with a sensitivity level of 1 in a unit exceeds 70%, then the unit is pruned; if the proportion of blocks with a sensitivity level of 1 in a unit does not exceed 70%, and the proportion of blocks with sensitivity levels of the first and second levels in a unit exceeds 70%, then the unit is optimized by model search; if the proportion of blocks with sensitivity levels of the first and second levels in a unit does not exceed 70%, then the unit is neither pruned nor optimized by model search.

[0076] In some embodiments of the present invention, the baseline multi-task model includes a backbone, a decoder, a neck, and a head. Constructing the baseline multi-task model means determining the specific structure of the backbone, decoder, neck, and head. After the above-mentioned sensitivity level calculation, the optimization methods for each unit are selected as follows: model search optimization is performed on the backbone, no optimization is performed on the decoder and neck, and model pruning is performed on the head.

[0077] Traditional multi-task detection methods are lightweighting approaches that involve pruning the entire model or performing NAS on the entire model. However, both of these methods have their own drawbacks, resulting in insufficient or excessive lightweighting of the multi-task model, which affects detection efficiency and results.

[0078] The multi-task detection method provided in this invention acquires multi-task perception data; inputs the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein, the lightweight multi-task model is obtained by optimizing the baseline multi-task model based on the sensitivity level of each unit; the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information, and is optimized based on the sensitivity level of each unit. Different optimization methods are used for different units in the model, which can make the multi-task model more lightweight, reduce deployment time and computational resource consumption, and improve the efficiency and accuracy of multi-task detection results.

[0079] Based on any of the above embodiments, each unit can be composed of a classic model structure: for example, the backbone can be ResNet50, MobileNetV3, etc., the decoder can be a Unet-style upsampling structure, the neck can be an FPN structure, and the head can be selected as a multi-layer convolutional layer.

[0080] In some embodiments of the present invention, training the baseline multi-task model includes training the baseline multi-task model with multi-task perception training data and multi-task object detection information in the training dataset, and determining the model weights after the output converges, denoted as M_p.

[0081] In some embodiments of the present invention, multi-task detection for autonomous driving includes joint detection of traffic targets, drivable areas, and lane lines.

[0082] For example, in a parking scenario, information such as parking space detection, wall and pillar detection, drivable space detection, and obstacle (dynamic and static) detection is required. Therefore, after receiving perception data (such as perception data from cameras, LiDAR, and millimeter-wave radar), the perception model needs to output multiple detection information simultaneously.

[0083] Based on any of the above embodiments, the baseline multi-task model includes a backbone network unit. In the embodiments of the present invention, such as... Figure 2 As shown, model search optimization is performed based on the baseline multi-task model, including:

[0084] Step 201: Convert the backbone network units in the baseline multi-task model into a supernet structure based on multiple search dimensions;

[0085] In this embodiment of the invention, the search dimensions include: the range of convolutional kernel size is (3x3, 5x5, 7x7), and the range of the number of channels is (0.5, 0.8, 1, 1.5, 2 times the original number of channels). Therefore, if the backbone network unit has n layers, each layer has 3*5 = 15 choices, and the entire backbone network unit has a total of 15 choices. n There are several choices, which is the size of the entire search space.

[0086] In this embodiment of the invention, converting the backbone network units in the baseline multi-task model into a supernet structure includes:

[0087] Based on the sensitivity of each layer in the backbone network unit, the corresponding layer of the backbone network unit is converted into a subnet in the supernet structure;

[0088] The sensitivity of each layer is determined based on the magnitude of its impact on the output of the baseline multi-task model after pruning. The sensitivity analysis method for each layer is as follows:

[0089] First, sort the channel weights of this layer, then prune the channel with the smallest weight. Calculate the percentage increase in loss of the baseline multi-task model on the test set after pruning compared to the baseline before pruning; this percentage is the sensitivity of this layer. For example, if the loss before pruning is loss_1 and the loss after pruning is loss_2, then the sensitivity is (loss_2 - loss_1) / loss_1.

[0090] In this embodiment of the invention, when determining the sensitivity of a layer, the layer can be pruned first. The accuracy of the entire model's output is then assessed by determining if pruning one channel of the layer reduces its overall accuracy. If so, it indicates that the layer has high sensitivity and is more suitable for search optimization. In this case, the layer can be converted into a subnet within a supernet structure for optimization. In this embodiment, search optimization was performed on the backbone network units, and the detection head was pruned.

[0091] Step 202: Train the supernet structure;

[0092] In this embodiment of the invention, training the supernet structure includes:

[0093] Load the pre-trained model weights M_p of the baseline multi-task model: Since the backbone network unit was later replaced by a supernet, the pre-trained weights are not suitable for the supernet, so the weights of the backbone network unit do not need to be loaded. Load the weights of other modules such as the decoder, neck, and head modules. For non-backbone weights, just load them.

[0094] Based on the defined search dimensions, the backbone network units are converted into a supernet structure.

[0095] For example, the original fixed convolutional kernels in each layer of the backbone network unit can be expanded to a variety of selectable convolutional kernel sizes, and the original fixed number of channels in each layer can be expanded to a variety of selectable number of channels, etc.

[0096] The specific training method is as follows:

[0097] Step 1: Initialize iter_i = 0 (iter_i is the number of training iterations);

[0098] Step 2: If iter_i is less than iters_all (iters_all is the target total number of training iterations), then proceed with the following steps:

[0099] Step (1): First, randomly generate the current subnet;

[0100] Step (2): Perform forward computation on the subnets and update the model weights based on the inference loss values ​​calculated from the inference results of each subnet and the manual calibration results;

[0101] Step (3): iter_i = iter_i + 1.

[0102] Training ends once the target total number of training sessions has been reached.

[0103] Step 203: Search for the best subnet in the trained supernet structure and use the best subnet as the new backbone network unit of the baseline multi-task model;

[0104] In this embodiment of the invention, searching for the optimal subnet in the trained supernet structure includes:

[0105] Step 2031: Randomly generate all subnets that meet the preset constraints;

[0106] In embodiments of the present invention, limiting conditions include, for example, the amount of model computation and the number of parameters.

[0107] Step 2032: Calculate the inference result of each subnet and the manual calibration result to calculate the inference loss, obtain the inference loss value corresponding to each subnet, and select the subnet with the smallest inference loss value as the best subnet.

[0108] In this embodiment of the invention, this process can be performed in parallel using multiple processes to speed up the search.

[0109] In this embodiment of the invention, the inference loss values ​​of all subnets are sorted from smallest to largest, and the subnet with the smallest inference loss value is selected as the best subnet.

[0110] Step 204: Train the baseline multi-task model, which includes the new backbone network units, to obtain a multi-task model with lightweight backbone network units.

[0111] In this embodiment of the invention, the model after replacing the backbone network unit with the best subnet is trained, and the training method refers to the baseline multi-task model. After training, the optimized multi-task model is obtained.

[0112] Traditional NAS model optimization methods optimize the entire model. Due to the large search space, the optimization problem of model search becomes very complex. It is difficult to find the ideal structure with limited resources and time, which affects the efficiency and accuracy of model output results.

[0113] In this embodiment of the invention, model search optimization is performed only on the backbone network units in the baseline multi-task model to reduce the search space and improve search efficiency.

[0114] Furthermore, in this embodiment of the invention, the search optimization is not performed on all layers in the network structure. That is, all layers in the backbone network unit are converted into subnets in the supernet structure. Due to the differences in the sensitivity of each layer in the model structure, it is difficult to optimize each layer accurately without affecting the model performance.

[0115] In this embodiment of the invention, by filtering the layers to be optimized based on the differences in sensitivity levels of each layer, the search space is reduced, the optimization problem of model search is simplified, and the search efficiency is improved compared to traditional NAS.

[0116] Based on any of the above embodiments, the baseline multi-task model includes multiple detection heads. In the embodiments of the present invention, such as... Figure 3 As shown, model compression processing based on the baseline multi-task model includes:

[0117] Step 301: Set the pruning rate of the layer to be pruned for each detection head;

[0118] Step 302: Prune the convolution kernels and filters of the layer to be pruned according to the pruning rate to obtain a multi-task model with a lightweight detection head.

[0119] In this embodiment of the invention, after pruning the weights of each layer according to a preset pruning rate, the method further includes:

[0120] Calculate the pruning rate of each convolutional kernel and the pruning rate of the filter;

[0121] If the pruning rate of the convolutional kernels in the current layer exceeds the first preset threshold, then all convolutional kernels in the current layer will be pruned.

[0122] If the pruning rate of the current layer's filters exceeds the second preset threshold, then all filters in the current layer will be pruned.

[0123] For example, the pruning rate is set to 0.5, and the pruning rate range is (0,1). The model weights of the detector head are sorted according to the L1Norm value. The smaller the value, the lower the importance is considered. The weights are pruned according to the preset pruning rate, that is, the corresponding weight value is set to 0.

[0124] A pruning expansion strategy is adopted to increase the pruning rate. The pruning expansion strategy includes: if the pruning rate of the current kernel (convolution kernel) exceeds 80%, the entire kernel is pruned to further increase the pruning rate. If the pruning rate of the current filter exceeds 90%, the entire filter is pruned, that is, the entire output channel is pruned.

[0125] In this embodiment of the invention, the pruning rate is further improved by the pruning expansion strategy, and the pruning of the backbone network units with high sensitivity is avoided, thereby increasing the upper limit of the model's lightweight design and ensuring output accuracy.

[0126] Based on any of the above embodiments, such as Figure 4 As shown, the lightweight multi-task model is obtained by optimizing and compressing the baseline multi-task model, including:

[0127] Step 401: Obtain a lightweight multi-task model based on the multi-task model with lightweight backbone network units and the multi-task model with lightweight detection heads;

[0128] Lightweight multi-task model structure such as Figure 5 As shown, it includes: a backbone network unit, a decoder, and a multi-task module, which consists of a neck and a head structure corresponding to each task, and each task module shares the same backbone network unit and decoder structure.

[0129] The structure of the backbone, decoder, neck, and head can be composed of classic model structures.

[0130] Figure 5 The upper half (dashed line) describes a lightweight approach for this multi-task model. This involves NAS optimization of the backbone network units and pruning the detection head.

[0131] Step 402: Fine-tune the lightweight multi-task model to obtain a trained lightweight multi-task model.

[0132] By fine-tuning the training, lightweight multi-task models can be better suited for autonomous driving scenarios.

[0133] The multi-task detection method provided in this embodiment of the invention optimizes only the backbone network units, compared to NAS which optimizes the entire model. This reduces the search space, simplifies the optimization problem of model search, and improves search efficiency. Furthermore, compared to pruning the entire model, it avoids pruning the backbone network units with high sensitivity, increases the upper limit of model lightweighting, and makes the model lighter while maintaining the output performance before lightweighting.

[0134] The multi-task detection device for autonomous driving provided by the present invention is described below. The multi-task detection device for autonomous driving described below can be referred to in correspondence with the multi-task detection method described above.

[0135] Figure 6 This is a schematic diagram of a multi-task detection device for autonomous driving provided in an embodiment of the present invention, such as... Figure 6 As shown, the multi-task detection device for autonomous driving provided in this embodiment of the invention includes:

[0136] Module 601 is used to acquire multi-task awareness data;

[0137] The output module 602 is used to input multi-task perception data into a lightweight multi-task model to output detection information corresponding to each task. The lightweight multi-task model is obtained by optimizing and compressing the baseline multi-task model. The baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information.

[0138] The multi-task detection device for autonomous driving provided in this invention acquires multi-task perception data; inputs the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein, the lightweight multi-task model is obtained by optimizing the baseline multi-task model based on the sensitivity level of each unit; the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information, and is optimized based on the sensitivity level of each unit. Different optimization methods are used for different units in the model, which can make the multi-task model more lightweight, reduce deployment time and computational resource consumption, and improve multi-task detection efficiency and detection result accuracy.

[0139] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-task detection method as described above.

[0140] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a multi-task detection method. This method includes: acquiring multi-task perception data; inputting the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein the lightweight multi-task model is obtained by optimizing a baseline multi-task model based on the sensitivity levels of each unit; and the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information.

[0141] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] This invention also provides a vehicle including the electronic equipment described in the above embodiments.

[0143] In this embodiment of the invention, the electronic device is an autonomous driving controller, and the vehicle is a vehicle capable of autonomous driving.

[0144] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the outbound call methods provided by the above methods. The method includes: acquiring multi-task perception data; inputting the multi-task perception data into a trained lightweight multi-task model to output detection information corresponding to each task; wherein the lightweight multi-task model is obtained by optimizing a baseline multi-task model based on the sensitivity levels of each unit; and the baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information.

[0145] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-task detection method, characterized in that, include: Acquire multi-task perception data, which includes perception data from cameras, lidar, and millimeter-wave radar; The multi-task perception data is input into the trained lightweight multi-task model to output the detection information corresponding to each task; The lightweight multi-task model is obtained by optimizing the baseline multi-task model based on the sensitivity level of each unit. The method for calculating the sensitivity level of each unit includes: Calculate the sensitivity of each layer; Take the average sensitivity of each layer contained in each block, and determine the sensitivity level of the corresponding block based on the average layer sensitivity; Calculate the proportion of blocks in each unit whose sensitivity level is lower than a preset threshold, and determine the sensitivity level of each unit based on the calculated proportion of blocks. or, Calculate the sensitivity of each block; The sensitivity of each block contained in each unit is averaged, and the sensitivity level of each unit is determined based on the average block sensitivity. The optimization of the baseline multi-task model based on the sensitivity level of each unit includes: If the proportion of blocks with the first sensitivity level in a certain unit exceeds the proportion threshold, then the unit will be pruned. If the proportion of blocks with sensitivity level 1 in a certain unit does not exceed the proportion threshold, and the proportion of blocks with sensitivity level 1 and 2 in a certain unit exceeds the proportion threshold, then the unit is optimized by model search. If the proportion of blocks with sensitivity levels of first and second level in a certain unit does not exceed the proportion threshold, then the unit will not be optimized. The baseline multi-task model is trained based on multi-task perception training data and multi-task target detection information.

2. The multi-task detection method according to claim 1, characterized in that, The baseline multi-task model includes backbone network units. The proportion of blocks with sensitivity level 1 in the backbone network units does not exceed a certain threshold, and the proportion of blocks with sensitivity levels 1 and 2 exceeds the threshold. Model search optimization is performed on the backbone network units, including: The backbone network units in the baseline multi-task model are converted into a supernet structure based on multiple search dimensions. The supernet structure is trained; Search for the optimal subnet in the trained supernet structure and use the optimal subnet as the new backbone network unit of the baseline multi-task model. By training a baseline multi-task model that includes new backbone network units, a multi-task model with lightweight backbone network units is obtained.

3. The multi-task detection method according to claim 2, characterized in that, The process of searching for the optimal subnet within the trained supernet structure includes: Randomly generate all subnets that meet preset constraints; The inference result of each subnet is compared with the manually calibrated result to calculate the inference loss, and the inference loss value corresponding to each subnet is obtained. The subnet with the smallest inference loss value is selected as the best subnet.

4. The multi-task detection method according to claim 2, characterized in that, Converting the backbone network units in the baseline multi-task model into a supernet structure includes: Based on the sensitivity of each layer in the backbone network unit, the corresponding layer of the backbone network unit is converted into a subnet in the supernet structure.

5. The multi-task detection method according to claim 1, characterized in that, The baseline multi-task model includes multiple detection heads. If the proportion of blocks with a sensitivity level of first level in these multiple detection heads exceeds a certain threshold, model pruning is performed on these multiple detection heads, including: Set the pruning rate of the layer to be pruned for each detection head; The convolution kernels and filters of the layer to be pruned are pruned according to the pruning rate; Calculate the pruning rate of each convolutional kernel and the pruning rate of the filter; If the pruning rate of the convolutional kernels in the current layer exceeds the first preset threshold, then all convolutional kernels in the current layer will be pruned. If the pruning rate of the current layer's filters exceeds the second preset threshold, then all filters in the current layer will be pruned to obtain a multi-task model with a lightweight detection head.

6. The multi-task detection method according to claim 2, characterized in that, Also includes: A lightweight multi-task model is obtained based on the multi-task model with lightweight backbone network units and the multi-task model with lightweight detection head. The lightweight multi-task model is fine-tuned and trained to obtain a well-trained lightweight multi-task model.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-task detection method as described in any one of claims 1 to 6.

8. A vehicle, characterized in that, Including the electronic device as described in claim 7.

Images