An automatic driving multi-task perception method based on an adaptive hybrid expert model

Abstract

This application provides an adaptive hybrid expert-based multi-task perception method for autonomous driving, comprising: acquiring raw data collected by multiple sensors of a vehicle; processing the raw data using a pre-trained multi-task perception model to obtain perception results for N tasks; wherein the multi-task perception model includes N task branches, each task branch including a backbone network and a detection head connected in sequence, each backbone network including M convolutional units connected in sequence, and each level of convolutional unit in each task branch being connected to a gating network, the gating network being used to control whether the output of each level of convolutional unit is used as the input of the next level of convolutional unit. This application improves the understanding and accuracy of the surrounding environment by the panoramic driving perception system.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to an autonomous driving multi-task perception method based on an adaptive hybrid expert model. Background Technology

[0002] With the rapid development of autonomous driving technology, panoramic driving perception systems are widely used in autonomous driving to identify and interpret information about the surrounding environment. Traditional panoramic driving perception systems typically deploy dedicated perception models for each task to accomplish different perception tasks. However, as perception demands increase, the total model size grows linearly with the number of perception tasks, exceeding the deployment capabilities of autonomous driving edge devices. Multi-task learning techniques reduce the width and size of the model by widely sharing components within the model, reducing the number of parameters while increasing the accuracy and robustness of the perception model, providing an effective solution for deploying panoramic driving perception systems. However, traditional multi-task learning methods have the following problems:

[0003] First, existing methods are limited by specific domain configurations, and the input data for various tasks remains the same, which cannot effectively cope with the perception characteristics of autonomous vehicles using multiple vision cameras for multi-vision input.

[0004] Second, existing methods use static model structures, which cannot flexibly adapt to the complexity of different tasks and changes in data characteristics when faced with scenarios where perception tasks vary greatly, thus causing a decrease in the accuracy of individual tasks.

[0005] Third, there are parameter competitions and training conflicts between different perception tasks. When managing multiple tasks in different dataset domains, domain transfer will exacerbate the conflicts, further affecting performance and leading to a decline in model performance and training difficulties. Summary of the Invention

[0006] In view of this, this application proposes an autonomous driving multi-task perception method based on an adaptive hybrid expert model to solve the above-mentioned technical problems.

[0007] In a first aspect, embodiments of this application provide an autonomous driving multi-task perception method based on adaptive hybrid experts, including:

[0008] Acquire raw data collected by multiple sensors on the vehicle;

[0009] The raw data is processed using a pre-trained multi-task perception model to obtain perception results for N tasks.

[0010] The multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each convolutional unit at the same level of each task branch is connected to a gating network. The gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level convolutional unit.

[0011] Furthermore, the gating network is a multilayer perceptron.

[0012] Furthermore, in a task branch, when the gating network controls the m-th convolutional unit not to output a result, the output of the (m-1)-th convolutional unit is used as the input of the (m+1)-th convolutional unit; where, when m=1, the input of the second convolutional unit is the raw data collected by multiple sensors.

[0013] Furthermore, each convolutional unit includes at least one convolutional layer.

[0014] Furthermore, the method also includes the step of training a multi-task perception model, specifically including:

[0015] Establish a training sample set, which includes: multiple sets of simultaneous spatiotemporal data samples from multiple sensors and real perception results of N tasks;

[0016] The raw data collected by multiple sensors is processed using a multi-task perception model to obtain the predicted perception results for N tasks.

[0017] By using the actual perception results and predicted perception results for each task, the loss function for each task is obtained;

[0018] Determine the total loss function based on the loss functions of N tasks;

[0019] The parameters of the convolutional layers of all convolutional units in each task branch, as well as the parameters of each gating network, are updated based on the total loss function.

[0020] Furthermore, based on the loss functions of the N tasks, the total loss function is determined, including:

[0021] Total loss function L total for:

[0022]

[0023] in, Let λ be the loss function for the k-th task. k This represents the weight of the k-th task.

[0024] Furthermore, the parameters of the convolutional layers of the peer convolutional units of the N task branches are divided into shared parameters and specific parameters, wherein the shared parameters of the convolutional layers of the peer convolutional units of the N task branches are all the same.

[0025] Secondly, embodiments of this application provide an autonomous driving multi-task perception device based on adaptive hybrid experts, comprising:

[0026] The acquisition unit is used to acquire raw data collected by multiple sensors of the vehicle.

[0027] The processing unit is used to process the raw data using a pre-trained multi-task perception model to obtain the perception results of N tasks.

[0028] The multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each convolutional unit at the same level of each task branch is connected to a gating network. The gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level convolutional unit.

[0029] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0030] Fourthly, an embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the embodiment of this application.

[0031] This application aims to improve the understanding and accuracy of the surrounding environment by the panoramic driving perception system. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 A flowchart of an autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of the structure of the multi-task perception model provided in the embodiments of this application;

[0035] Figure 3 This is a schematic diagram of the parameter partitioning of the convolutional layer provided in an embodiment of this application;

[0036] Figure 4 This is a functional structure diagram of an autonomous driving multi-task perception device based on an adaptive hybrid expert model provided in an embodiment of this application.

[0037] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0040] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0041] The current multi-task learning methods have the following drawbacks: parameter competition and training conflicts exist between different perceptual tasks. When managing multiple tasks in different dataset domains, domain transfer exacerbates the conflicts, further affecting performance and leading to model performance degradation and training difficulties.

[0042] To address this, this application provides a multi-task visual perception method based on an adaptive hybrid expert framework. The method includes: constructing a multi-task learning network framework with multiple inputs and multiple outputs; introducing hybrid expert layers into the backbone network, replacing the basic modules in ResNet with sparse activation hybrid expert layers; introducing an adaptive expert activation technique (AEAT) to adaptively adjust the number of experts involved in each task, ensuring the model structure meets the needs of multiple tasks; introducing a gradient pruning technique to average the gradients between different tasks to control the maximum norm of the gradients; and employing a joint optimization strategy to combine the training losses of multiple perception tasks for joint optimization, improving the overall performance and efficiency of the model.

[0043] First, the hybrid expert layer is defined as consisting of a set of N experts f1, f2, K, f NA network structure consisting of a router R (or a gated network), where router R is configured as a top-K gate, specifically as follows:

[0044]

[0045] Here, G represents the learnable network in the router, employing a multilayer perceptron (MLP). The softmax(·) method combined with TopK(·,K) sets all elements in the vector to zero, except for the element with the largest K value. By using the router to select the number of experts, a hybrid expert layer achieves a dynamic balance between network capacity and efficiency, effectively improving the perceptual performance of the multi-task model.

[0046] Secondly, this application iterates based on the model's perception results on the dataset to search for the number of experts when the model achieves the optimal result, thereby realizing the automatic finding and determination of the number of experts in each model and improving the overall performance and training efficiency of the model.

[0047] This application includes two aspects: firstly, it introduces a hybrid expert layer to effectively separate the parameter spaces of different tasks and reduce training conflicts; secondly, it introduces an adaptive expert activation technique that can adaptively adjust the number of experts involved in each task, preventing overfitting of simple tasks and underfitting of complex tasks.

[0048] Compared with existing technologies, the advantages of this application are:

[0049] 1. Construct a multi-task learning network framework with multiple inputs and multiple outputs, which can adaptively process multiple visual inputs and effectively utilize information from different perspectives, thereby improving the panoramic driving perception system's understanding and accuracy of the surrounding environment;

[0050] 2. By using an adaptive hybrid expert framework and adaptive expert activation technology, the model structure and parameters can be dynamically adjusted, and resources can be allocated and model complexity adjusted adaptively according to the needs of different tasks, thereby effectively improving the performance of each task and avoiding the problem of accuracy degradation.

[0051] 3. By introducing a hybrid expert layer and gradient pruning techniques, the parameter spaces of different tasks can be effectively separated, reducing training conflicts.

[0052] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0053] This application provides an autonomous driving multi-task perception method based on an adaptive hybrid expert model, comprising the following steps:

[0054] Step 101: Acquire data collected by multiple sensors of the vehicle at the current moment and the target task;

[0055] The system includes one or more target tasks, which can be lane segmentation, semantic segmentation, object detection, etc. Multiple sensors are used, including RGB cameras, millimeter-wave radar, and 4D millimeter-wave radar.

[0056] Step 102: Process the raw data using a pre-trained multi-task perception model to obtain perception results for N tasks; wherein, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each level of convolutional unit in each task branch is connected to a gating network, the gating network being used to control whether the output of each level of convolutional unit is used as the input of the next level of convolutional unit.

[0057] like Figure 2 As shown, the multi-task perception model includes three task branches, and the backbone network is ResNet50, which includes five convolutional units: stage1, stage2, stage3, stage4 and stage5.

[0058] In some embodiments, the gating network is a multilayer perceptron. In a task branch, when the gating network controls the m-th convolutional unit not to output a result, the output of the (m-1)-th convolutional unit is used as the input of the (m+1)-th convolutional unit; wherein, when m=1, the input of the second convolutional unit is the raw data collected by multiple sensors.

[0059] As can be seen, the gating network is used to control whether the results of multiple corresponding convolutional units are output. For example, the result of the first convolutional unit in the first task branch is not output, while the result of the first convolutional unit in the second task branch is output. The N convolutional units represent the N experts f1, f2, K, f... N The number of experts is selected through a router, that is, the gating network controls which of the N convolutional units output and which do not.

[0060] In some embodiments, each convolutional unit includes at least one convolutional layer. For example... Figure 2 As shown, each convolutional unit of ResNet50 includes a convolutional layer CONV.

[0061] In some embodiments, the method further includes a step of training a multi-task perception model, specifically including:

[0062] Establish a training sample set, which includes: multiple sets of simultaneous spatiotemporal data samples from multiple sensors and real perception results of N tasks;

[0063] The raw data collected by multiple sensors is processed using a multi-task perception model to obtain the predicted perception results for N tasks.

[0064] By using the actual perception results and predicted perception results for each task, the loss function for each task is obtained;

[0065] Determine the total loss function based on the loss functions of N tasks;

[0066] The parameters of the convolutional layers of all convolutional units in each task branch, as well as the parameters of each gating network, are updated based on the total loss function.

[0067] Furthermore, based on the loss functions of the N tasks, the total loss function is determined, including:

[0068] Total loss function L total for:

[0069]

[0070] in, Let λ be the loss function for the k-th task. k Let λ be the weight of the k-th task. The weight λ is adjusted by... k This allows for control over the relative importance of each task, enabling model training to flexibly adapt to different task requirements and dataset characteristics.

[0071] This application employs a joint optimization strategy, which combines the training losses of multiple perception tasks for joint optimization, thereby improving the overall performance and efficiency of the model.

[0072] For gradient estimation during training, the score s of the k-th task is used. k,j The gradient formula is as follows:

[0073]

[0074] Furthermore, to indicate that the gradient of the function Θ(·) is learnable, a direct gradient estimator is employed, which modifies backpropagation and uses:

[0075]

[0076] This enables the model to effectively maintain high robustness and accuracy across a variety of different tasks.

[0077] In some embodiments, the parameters of the convolutional layers of the peer convolutional units in the N task branches are divided into shared parameters and specific parameters, wherein the shared parameters of the convolutional layers of the peer convolutional units in the N task branches are all the same. For example... Figure 3As shown, the convolutional layer is a multi-kernel convolutional layer that adaptively partitions the parameters according to the corresponding task type and elements.

[0078] Based on the above embodiments, this application provides an autonomous driving multi-task perception device based on an adaptive hybrid expert model, see below. Figure 4 As shown, the autonomous driving multi-task perception device 200 based on an adaptive hybrid expert model provided in this application embodiment includes at least:

[0079] Acquisition unit 201 is used to acquire raw data collected by multiple sensors of the vehicle;

[0080] The processing unit 202 is used to process the raw data using a pre-trained multi-task perception model to obtain the perception results of N tasks.

[0081] The multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each convolutional unit at the same level of each task branch is connected to a gating network. The gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level convolutional unit.

[0082] It should be noted that the principle of the autonomous driving multi-task perception device 200 based on adaptive hybrid expert model provided in this application to solve the technical problem is similar to the method provided in this application. Therefore, the implementation of the autonomous driving multi-task perception device 200 based on adaptive hybrid expert model provided in this application can refer to the implementation of the method provided in this application, and the repeated parts will not be described again.

[0083] Based on the above embodiments, this application also provides an electronic device, see below. Figure 5 As shown, the electronic device 300 provided in this application embodiment includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and capable of running on the processor 301. When the processor 301 executes the computer program, it implements the autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in this application embodiment.

[0084] The electronic device 300 provided in this application embodiment may further include a bus 303 connecting different components (including processor 301 and memory 302). The bus 303 represents one or more types of bus structures, including memory bus, peripheral bus, local area bus, etc.

[0085] The memory 302 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 3021 and / or cache memory 3022, and may further include read-only memory (ROM) 3023.

[0086] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0087] Electronic device 300 can also communicate with one or more external devices 304 (e.g., keyboard, remote control, etc.), and with one or more devices that enable a user to interact with electronic device 300 (e.g., mobile phone, computer, etc.), and / or with any device that enables electronic device 300 to communicate with one or more other electronic devices 300 (e.g., router, modem, etc.). This communication can be performed through input / output (I / O) interface 305. Furthermore, electronic device 300 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 306. Figure 5 As shown, network adapter 306 communicates with other modules of electronic device 300 via bus 303. It should be understood that, although... Figure 5 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, Redundant Arrays of Independent Disks (RAID) subsystems, tape drives, and data backup storage subsystems.

[0088] It should be noted that, Figure 5 The electronic device 300 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0089] This application also provides a computer-readable storage medium storing computer instructions. When executed by a processor, these instructions implement the autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in this application. Specifically, the executable program can be built into or installed in an electronic device 300, so that the electronic device 300 can implement the autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in this application by executing the built-in or installed executable program.

[0090] The autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in this application embodiment can also be implemented as a program product. The program product includes program code. When the program product can run on the electronic device 300, the program code is used to make the electronic device 300 execute the autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in this application embodiment.

[0091] The program product provided in this application embodiment can be any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: electrical connections with one or more wires, portable disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0092] The program product provided in this application embodiment can be a CD-ROM and include program code, and can also run on a computing device. However, the program product provided in this application embodiment is not limited thereto. In this application embodiment, the readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0093] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0094] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application, and should all be covered within the scope of the claims of this application.

Claims

1. A multi-task perception method for autonomous driving based on adaptive hybrid experts, characterized in that, include: Acquire raw data collected by multiple sensors on the vehicle; The raw data is processed using a pre-trained multi-task perception model to obtain perception results for N tasks. The multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each level convolutional unit of each task branch is connected to a gating network. The gating network is used to control whether the output of each level convolutional unit is used as the input of the next level convolutional unit. In a task branch, if the gating network controls the m-th convolutional unit not to output a result, then the output of the (m-1)-th convolutional unit is used as the input of the (m+1)-th convolutional unit; where, when m=1, the input of the second convolutional unit is the raw data collected by multiple sensors. The parameters of the convolutional layers of the same level convolutional units in the N task branches are divided into shared parameters and specific parameters. The shared parameters of the convolutional layers of the same level convolutional units in the N task branches are all the same. The method further includes: a step of training a multi-task perception model, specifically including: Establish a training sample set, which includes: multiple sets of simultaneous spatiotemporal data samples from multiple sensors and real perception results of N tasks; The raw data collected by multiple sensors is processed using a multi-task perception model to obtain the predicted perception results for N tasks. By using the actual perception results and predicted perception results for each task, the loss function for each task is obtained; Determine the total loss function based on the loss functions of N tasks; Update the parameters of the convolutional layers of all convolutional units in each task branch and the parameters of each gated network according to the total loss function; Based on the loss functions of N tasks, determine the total loss function, including: Total loss function for: in, Let be the loss function for the k-th task. This represents the weight of the k-th task.

2. The method according to claim 1, characterized in that, The gating network is a multilayer sensor.

3. The method according to claim 1, characterized in that, Each convolutional unit includes at least one convolutional layer.

4. An autonomous driving multi-task perception device based on adaptive hybrid experts, characterized in that, include: The acquisition unit is used to acquire raw data collected by multiple sensors of the vehicle. The processing unit is used to process the raw data using a pre-trained multi-task perception model to obtain the perception results of N tasks. The multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and each convolutional unit at the same level of each task branch is connected to a gating network. The gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level convolutional unit. In a task branch, if the gating network controls the m-th convolutional unit not to output a result, then the output of the (m-1)-th convolutional unit is used as the input of the (m+1)-th convolutional unit; where, when m=1, the input of the second convolutional unit is the raw data collected by multiple sensors. The parameters of the convolutional layers of the same level convolutional units in the N task branches are divided into shared parameters and specific parameters. The shared parameters of the convolutional layers of the same level convolutional units in the N task branches are all the same. The device further includes: a training unit, specifically used for: Establish a training sample set, which includes: multiple sets of simultaneous spatiotemporal data samples from multiple sensors and real perception results of N tasks; The raw data collected by multiple sensors is processed using a multi-task perception model to obtain the predicted perception results for N tasks. By using the actual perception results and predicted perception results for each task, the loss function for each task is obtained; Determine the total loss function based on the loss functions of N tasks; Update the parameters of the convolutional layers of all convolutional units in each task branch and the parameters of each gated network according to the total loss function; Based on the loss functions of N tasks, determine the total loss function, including: Total loss function for: in, Let be the loss function for the k-th task. This represents the weight of the k-th task.

5. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-3.

Images