Automatic driving multi-task sensing method based on self-adaptive hybrid expert model

Abstract

The invention provides an automatic driving multi-task sensing method based on self-adaptive hybrid experts. The method comprises the following steps: acquiring original data acquired by a plurality of sensors of a vehicle; processing the original data by using a pre-trained multi-task perception model to obtain perception results of the N tasks; wherein the multi-task perception model comprises N task branches, each task branch comprises a backbone network and a detection head which are connected in sequence, each backbone network comprises M convolution units which are connected in sequence, and the convolution units of the same level of each task branch are connected with a gating network, and the gating network is used for controlling whether the output of each convolution unit at the same level is used as the input of the connected next-level convolution unit. The understanding and accuracy of the panoramic driving perception system on the surrounding environment are improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving multi-task perception method based on an adaptive hybrid expert model. Background Art

[0002] With the rapid development of driverless technology, panoramic driving perception systems are widely used in autonomous driving to identify and interpret surrounding environment information. Traditional panoramic driving perception systems usually deploy dedicated perception models for each task to complete different perception tasks. However, as the perception requirements increase, the total model size grows linearly with the number of perception tasks, which exceeds the deployment capacity of autonomous driving edge devices. Multi-task learning technology reduces the width and size of the model by widely sharing components within the model. While reducing the number of parameters, it also increases the accuracy and robustness of the perception model, providing an effective solution for the deployment of panoramic driving perception systems. However, traditional multi-task learning methods have the following problems:

[0003] First, existing methods are limited by specific domain configurations, and the input data of various tasks remain the same, which cannot effectively cope with the perception characteristics of autonomous driving vehicles using multiple visual cameras for multi-visual input.

[0004] Second, the existing methods use a static model structure. When faced with scenarios with large differences in perception tasks, they cannot flexibly adapt to the complexity of different tasks and changes in data characteristics, resulting in a decrease in the accuracy of a single task.

[0005] Third, there is parameter competition and training conflict between different perception tasks. When managing multiple tasks in different dataset domains, domain transfer will aggravate the conflict and further affect the performance, leading to degraded model performance and difficulty in training. Summary of the invention

[0006] In view of this, the present application proposes an autonomous driving multi-task perception method based on an adaptive hybrid expert model to solve the above technical problems.

[0007] In a first aspect, an embodiment of the present application provides an autonomous driving multi-task perception method based on adaptive hybrid experts, comprising:

[0008] Obtain the raw data collected by multiple sensors of the vehicle;

[0009] Use the pre-trained multi-task perception model to process the raw data and obtain the perception results of N tasks;

[0010] Among them, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

[0011] Furthermore, the gating network is a multi-layer perceptron.

[0012] Furthermore, in a task branch, when the gated network controls the mth convolution unit not to output a result, the output result of the m-1th convolution unit is used as the input of the m+1th convolution unit; when m=1, the input of the second convolution unit is the raw data collected by multiple sensors.

[0013] Furthermore, each of the convolution units includes at least one convolution layer.

[0014] Furthermore, the method further includes: a step of training the multi-task perception model, specifically including:

[0015] Establishing a training sample set, the training sample set includes: multiple groups of simultaneous and spatial data samples from multiple sensors and real perception results of N tasks;

[0016] The multi-task perception model is used to process the raw data collected by multiple sensors to obtain the predicted perception results of N tasks;

[0017] Using the real perception results and predicted perception results of each task, we get the loss function of each task.

[0018] Based on the loss functions of N tasks, determine the total loss function;

[0019] According to the total loss function, the parameters of the convolutional layers of all convolutional units of each task branch and the parameters of each gating network are updated.

[0020] Furthermore, based on the loss functions of the N tasks, the total loss function is determined, including:

[0021] The total loss function L total for:

[0022]

[0023] in, is the loss function of the kth task, λ k is the weight of the kth task.

[0024] Furthermore, the parameters of the convolutional layers of the same-level convolutional units of the N task branches are divided into shared parameters and specific parameters, wherein the shared parameters of the convolutional layers of the same-level convolutional units of the N task branches are the same.

[0025] In a second aspect, an embodiment of the present application provides an autonomous driving multi-task perception device based on an adaptive hybrid expert, comprising:

[0026] An acquisition unit, used to acquire raw data collected by multiple sensors of the vehicle;

[0027] A processing unit, used to process the raw data using a pre-trained multi-task perception model to obtain perception results of N tasks;

[0028] Among them, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method of the embodiment of the present application is implemented.

[0031] This application improves the understanding and accuracy of the surrounding environment of the panoramic driving perception system. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 A flowchart of an autonomous driving multi-task perception method based on an adaptive hybrid expert model provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a multi-task perception model provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of parameter division of a convolutional layer provided in an embodiment of the present application;

[0036] Figure 4 A functional structure diagram of an autonomous driving multi-task perception device based on an adaptive hybrid expert model provided in an embodiment of the present application;

[0037] Figure 5 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0039] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0040] First, the design concept of the embodiments of the present application is briefly introduced.

[0041] The defects of current multi-task learning methods are: there is parameter competition and training conflict between different perception tasks. When managing multiple tasks in different dataset domains, domain shift will aggravate the conflict and further affect performance, resulting in degraded model performance and difficulty in training.

[0042] To this end, the present application provides a multi-task visual perception method based on an adaptive hybrid expert framework, the method comprising: constructing a multi-task learning network framework with multi-input and multi-output. A hybrid expert layer is introduced into the backbone network, and the basic module in ResNet is replaced by a sparsely activated hybrid expert layer. An adaptive expert activation technology (AEAT) is introduced to adaptively adjust the number of experts involved in each task to ensure that the model structure meets the needs of multiple tasks. Gradient clipping technology is introduced to average the gradients between different tasks to control the maximum norm of the gradient. A joint optimization strategy is adopted to combine the training losses of multiple perception tasks for joint optimization, thereby improving the overall performance and efficiency of the model.

[0043] First, the mixed expert layer is defined as a set of N experts f1,f2,K,f NAnd a network structure consisting of a router R (or gated network), where the router R is set as a top-K gate. The specific process is expressed as:

[0044]

[0045] Among them, G represents the learnable network in the router, which uses a multi-layer perceptron MLP; softmax(·) combined with TopK(·,K) sets all elements in the vector to zero, except for the element with the maximum K value. By using the router to select the number of experts, the hybrid expert layer achieves a dynamic balance between large network capacity and high efficiency, effectively improving the perception performance of the multi-task model.

[0046] Secondly, this application iterates based on the perceived results of the model on the data set, searching for the number of experts when the model is in the optimal result, thereby automatically finding and determining the number of experts in each model, and improving the overall performance and training efficiency of the model.

[0047] This application includes two aspects of work: on the one hand, it introduces a mixed expert layer to effectively separate the parameter space of different tasks and reduce training conflicts. On the other hand, it introduces adaptive expert activation technology, which can adaptively adjust the number of experts involved in each task to prevent overfitting of simple tasks and underfitting of complex tasks.

[0048] Compared with the prior art, the advantages of this application are:

[0049] 1. Build a multi-task learning network framework with multiple inputs and multiple outputs, which can adaptively process multiple visual inputs and effectively utilize information from different perspectives, thereby improving the understanding and accuracy of the panoramic driving perception system for the surrounding environment;

[0050] 2. Through the adaptive hybrid expert framework and adaptive expert activation technology, the model structure and parameters can be dynamically adjusted, resources can be adaptively allocated and model complexity can be adjusted according to the needs of different tasks, thereby effectively improving the performance of each task and avoiding the problem of accuracy degradation;

[0051] 3. By introducing the mixed expert layer and gradient clipping technology, the parameter space of different tasks can be effectively separated to reduce training conflicts.

[0052] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0053] The present application provides an autonomous driving multi-task perception method based on an adaptive hybrid expert model, comprising the following steps:

[0054] Step 101: Obtain data collected by multiple sensors of the vehicle at the current moment and the target task;

[0055] The target task is one or more, and the target task may be: lane segmentation, semantic segmentation, target detection, etc. The multiple sensors include: RGB camera, millimeter wave radar, 4D millimeter wave radar, etc.

[0056] Step 102: Use a pre-trained multi-task perception model to process the original data to obtain perception results of N tasks; wherein the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

[0057] like Figure 2 As shown in the figure, the multi-task perception model includes three task branches, and the backbone network is ResNet50, which includes five convolutional units: stage1, stage2, stage3, stage4 and stage5.

[0058] In some embodiments, the gated network is a multi-layer perceptron. In a task branch, when the gated network controls the mth convolution unit not to output a result, the output result of the m-1th convolution unit is used as the input of the m+1th convolution unit; wherein, when m=1, the input of the second convolution unit is the raw data collected by multiple sensors.

[0059] It can be seen that the gating network is used to control whether the results of the corresponding multiple convolution units are output. For example, the result of the first convolution unit of the first task branch is not output, while the result of the first convolution unit of the second task branch is output. N , the number of experts is selected through the router, that is, the gating network is used to control which outputs and which do not output among the N convolutional units.

[0060] In some embodiments, each of the convolution units includes at least one convolution layer. Figure 2 As shown, each convolution unit of ResNet50 includes a convolution layer CONV.

[0061] In some embodiments, the method further includes: a step of training the multi-task perception model, specifically including:

[0062] Establishing a training sample set, the training sample set includes: multiple groups of simultaneous and spatial data samples from multiple sensors and real perception results of N tasks;

[0063] The multi-task perception model is used to process the raw data collected by multiple sensors to obtain the predicted perception results of N tasks;

[0064] Using the real perception results and predicted perception results of each task, we get the loss function of each task.

[0065] Based on the loss functions of N tasks, determine the total loss function;

[0066] According to the total loss function, the parameters of the convolutional layers of all convolutional units of each task branch and the parameters of each gating network are updated.

[0067] Furthermore, based on the loss functions of the N tasks, the total loss function is determined, including:

[0068] The total loss function L total for:

[0069]

[0070] in, is the loss function of the kth task, λ k is the weight of the kth task. By adjusting the weight λ k , the relative importance of each task can be controlled, so that model training can flexibly adapt to different task requirements and data set characteristics.

[0071] This application adopts a joint optimization strategy to combine the training losses of multiple perception tasks for joint optimization, which improves the overall performance and efficiency of the model.

[0072] For gradient estimation during training, the score s of the kth task is k,j The gradient formula is as follows:

[0073]

[0074] Furthermore, to indicate that the gradient of the function Θ(·) is learnable, a direct gradient estimator is employed which modifies the backpropagation and uses:

[0075]

[0076] This enables the model to effectively achieve high robustness and accuracy in a variety of different tasks.

[0077] In some embodiments, the parameters of the convolutional layers of the same-level convolutional units of the N task branches are divided into shared parameters and specific parameters, wherein the shared parameters of the convolutional layers of the same-level convolutional units of the N task branches are the same. Figure 3As shown in the figure, the convolution layer is a multi-core convolution layer, and the parameters are adaptively divided according to the corresponding task type and elements.

[0078] Based on the above embodiments, the present application embodiment provides an autonomous driving multi-task perception device based on an adaptive hybrid expert model. Figure 4 As shown, the autonomous driving multi-task perception device 200 based on the adaptive hybrid expert model provided in the embodiment of the present application at least includes:

[0079] An acquisition unit 201 is used to acquire raw data collected by multiple sensors of the vehicle;

[0080] A processing unit 202 is used to process the raw data using a pre-trained multi-task perception model to obtain perception results of N tasks;

[0081] Among them, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

[0082] It should be noted that the principle of solving the technical problem of the multi-task perception device 200 for autonomous driving based on an adaptive hybrid expert model provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the multi-task perception device 200 for autonomous driving based on an adaptive hybrid expert model provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0083] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 5 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the autonomous driving multi-task perception method based on the adaptive hybrid expert model provided in the embodiment of the present application is implemented.

[0084] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0085] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0086] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0087] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 5 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0088] It should be noted that Figure 5 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0089] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the autonomous driving multi-task perception method based on the adaptive hybrid expert model provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the autonomous driving multi-task perception method based on the adaptive hybrid expert model provided in the embodiment of the present application by executing the built-in or installed executable program.

[0090] The multi-task perception method for autonomous driving based on an adaptive hybrid expert model provided in an embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the multi-task perception method for autonomous driving based on an adaptive hybrid expert model provided in an embodiment of the present application.

[0091] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0092] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0093] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0094] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. An adaptive hybrid expert-based multi-task perception method for autonomous driving, characterized in that: include: Obtain the raw data collected by multiple sensors of the vehicle; Use the pre-trained multi-task perception model to process the raw data and obtain the perception results of N tasks; Among them, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

2. The method according to claim 1, characterized in that The gated network is a multi-layer perceptron.

3. The method according to claim 2, characterized in that In a task branch, when the gated network controls the mth convolution unit not to output a result, the output result of the m-1th convolution unit is used as the input of the m+1th convolution unit; when m=1, the input of the second convolution unit is the raw data collected by multiple sensors.

4. The method according to claim 1, characterized in that: Each of the convolution units includes at least one convolution layer.

5. The method according to claim 1, characterized in that The method further includes: a step of training the multi-task perception model, specifically including: Establishing a training sample set, the training sample set includes: multiple groups of simultaneous and spatial data samples from multiple sensors and real perception results of N tasks; The multi-task perception model is used to process the raw data collected by multiple sensors to obtain the predicted perception results of N tasks; Using the real perception results and predicted perception results of each task, we get the loss function of each task. Based on the loss functions of N tasks, determine the total loss function; According to the total loss function, the parameters of the convolutional layers of all convolutional units of each task branch and the parameters of each gating network are updated.

6. The method according to claim 5, characterized in that Based on the loss functions of N tasks, determine the total loss function, including: The total loss function L total for: in, is the loss function of the kth task, λ k is the weight of the kth task.

7. The method according to claim 5, characterized in that The parameters of the convolutional layers of the same-level convolutional units of the N task branches are divided into shared parameters and specific parameters, wherein the shared parameters of the convolutional layers of the same-level convolutional units of the N task branches are the same.

8. An autonomous driving multi-task perception device based on adaptive hybrid experts, characterized in that: include: An acquisition unit, used to acquire raw data collected by multiple sensors of the vehicle; A processing unit, used to process the raw data using a pre-trained multi-task perception model to obtain perception results of N tasks; Among them, the multi-task perception model includes N task branches, each task branch includes a backbone network and a detection head connected in sequence, each backbone network includes M convolutional units connected in sequence, and the convolutional units of the same level of each task branch are connected to a gating network, and the gating network is used to control whether the output of each convolutional unit at the same level is used as the input of the next level of convolutional unit connected.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images