Neural network, automatic driving method and device

Abstract

The present application discloses a neural network, an automatic driving method and device, wherein the neural network includes an expert hybrid MOE network and N parallel head networks. The MOE network is used to extract and fuse features from input data to obtain N types of feature data; the input data includes vehicle data, obstacle data, road topology data and navigation data, the N types of feature data correspond one-to-one to the N parallel head networks, and each of the N parallel head networks corresponds to an automatic driving task; each head network performs a corresponding automatic driving task based on a corresponding input feature data; the automatic driving tasks include prediction tasks, decision-making tasks and planning tasks. Through the present application, each automatic driving task can maintain a certain degree of independence while maintaining a certain degree of interaction, which is more in line with actual driving needs.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a neural network, an autonomous driving method and a device. Background Art

[0002] As the most indispensable module of the autonomous driving system, the decision-making and planning module of the autonomous driving is responsible for making decisions and planning for the next behavior and action of the vehicle based on the historical and current status. For example, if there is a car illegally parked in front of you, it will decide whether to change lanes, when to change lanes, and how to change lanes. The prediction module of the autonomous driving plays a vital role in interactive scenarios. Accurately predicting the next behavior of other vehicles can enable the vehicle to plan in advance and avoid collision risks.

[0003] The prediction, decision-making and planning problems in autonomous driving have always been a challenging topic. Their complexity stems from the diversity of traffic environments and the uncertainty of the movement of traffic participants. Existing rule-based decision-making and planning methods rely on the features and experience abstracted by algorithm designers. They have limited modeling capabilities for complex scenarios and are difficult to generalize to all scenarios. Therefore, inhuman decision-making and planning results often appear. With the development of deep learning, learning and imitating the performance of human drivers through neural networks has become a major trend. By collecting massive amounts of human driver driving data and environmental data, the network can be trained to predict the future movement of obstacles, make decisions and plan the behavior and trajectory of the vehicle based on historical and current vehicle status, obstacle status and environmental information.

[0004] However, existing neural network-based autonomous driving solutions do not consider the mutual influence and needs of prediction, decision-making and planning, so the output results of the solutions cannot meet actual driving needs. Summary of the invention

[0005] The embodiments of the present application provide a neural network, an autonomous driving method and a device, which can enable each autonomous driving task to maintain a certain degree of independence while maintaining a certain degree of interaction, which is more in line with actual driving needs.

[0006] In a first aspect, the present application provides a neural network for autonomous driving, comprising a mixture of experts (MOE) network and N parallel head networks, where N is a positive integer; wherein the MOE network is used to perform feature extraction and feature fusion on input data to obtain N types of feature data; wherein the input data comprises vehicle data, obstacle data, road topology data and navigation data, and the N types of feature data correspond one-to-one to the N parallel head networks, and each of the N parallel head networks corresponds to an autonomous driving task; each head network performs the corresponding autonomous driving task based on the corresponding input feature data; wherein the autonomous driving task comprises prediction tasks, decision-making tasks and planning tasks.

[0007] From the technical effect point of view, this application uses the MOE network to uniformly encode the data required for each autonomous driving task (i.e., feature extraction), so that each autonomous driving task maintains a certain degree of interactivity (i.e., there is no specific upstream and downstream relationship between the head networks used to perform different autonomous driving tasks. Therefore, during the training process, the output results of each head network will be used to update the entire neural network, so that the head networks can influence each other); at the same time, since the head networks corresponding to each autonomous driving task have independent outputs, the head networks corresponding to each autonomous driving task can also be optimized independently (that is, each network maintains a certain degree of independence), which is more in line with actual driving needs. In addition, the use of the MOE network for unified encoding can significantly increase the number of neural network model parameters while keeping the amount of calculations basically unchanged during the reasoning process, effectively improving the generalization ability of the model in multiple scenarios; at the same time, compared with the independent encoding of a single task, the unified encoding of multiple tasks can effectively save computing power consumption and shorten latency.

[0008] In a feasible implementation, the MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; wherein the self-attention network is used to extract features from the input data, and the mutual attention network is used to perform feature fusion on the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

[0009] In a feasible implementation, the multiple expert networks include a first expert network, which is an expert network that performs feature extraction and feature fusion on the input data during the reasoning process, and the first expert network includes a first self-attention network, a first mutual attention network, a second self-attention network, and a second mutual attention network; the first self-attention network is used to extract features from the vehicle data and the obstacle data to obtain a first query Query matrix, a first key value Key matrix, and a first value Value matrix; the second self-attention network is used to extract features from the road topology data and the navigation data to obtain a second query Query matrix, a second key value Key matrix, and a second value Value matrix; the first mutual attention network is used to perform feature fusion on the first Query matrix, the second Key matrix, and the second Value matrix; the second mutual attention network is used to perform feature fusion on the second Query matrix, the first Key matrix, and the first Value matrix; the N types of feature data are generated through the output results of the first mutual attention network and the output results of the second mutual attention network.

[0010] From the perspective of technical effects, the present application adopts a cascade structure of self-attention and mutual attention in each expert network, which can make the feature data output by the MOE network for performing various autonomous driving tasks contain more information, thereby improving the accuracy of the corresponding output results of each autonomous driving task.

[0011] In a feasible implementation manner, the output results corresponding to the prediction task include the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle; the output results corresponding to the decision task include the self-vehicle decision, and the self-vehicle decision includes the self-vehicle preemption decision, the self-vehicle avoidance decision, and ignoring the obstacle; the output results corresponding to the planning task include the planned road of the self-vehicle, the planned trajectory of the self-vehicle, and the planned trajectory distribution of the self-vehicle.

[0012] In a second aspect, the present application provides a neural network training method, wherein the neural network includes an expert hybrid MOE network and N parallel head networks, and the MOE network includes M expert networks, where M and N are positive integers; the method includes: in each training process, performing feature extraction and feature fusion on each training data in a batch of training data through each expert network in the M expert networks to obtain N types of feature data; weighting the feature data of the same type output by the M expert networks respectively to obtain weighted N types of feature data; wherein the weighted N types of feature data correspond one-to-one to the N parallel head networks, and each of the N parallel head networks corresponds to an automatic driving The automatic driving task includes a prediction task, a decision-making task and a planning task; the weighted N types of feature data are respectively input into the corresponding head network to obtain H types of output results corresponding to each training data, the H types of output results include K types of output results, and K is a positive integer less than or equal to H; the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained from the one batch of training data are accumulated to obtain K accumulated loss values ​​corresponding to the K types of output results; the K accumulated loss values ​​are weighted to obtain a loss value corresponding to each training process; and the parameters in the neural network are updated using the one loss value.

[0013] Among them, the above K types of output results correspond to at least one autonomous driving task.

[0014] From the technical effect point of view, this application can select K types of output results related to the scene from H types of output results during the training process for a specific application scenario, and then update the parameters in at least one head network corresponding to the K types of output results to achieve single-task training or multi-task joint training. At the same time, by selecting part or all of the output results from each type of output results in K categories for training, it is possible to select corresponding data for each autonomous driving task for training, thereby ensuring the generalization of each task and effectively improving the accuracy of the output results of each autonomous driving task.

[0015] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0016] From a technical perspective, by adding a label that represents the autonomous driving task to each piece of training data, it is possible to determine whether to use the data to train the corresponding autonomous driving task. The training data can be used selectively based on specific application scenarios, thereby meeting the needs of the scenario, improving the accuracy of the neural network output results, and reducing the computing power consumption of the training process.

[0017] In a feasible implementation, the N parallel head networks include a first head network, the first head network outputs D-type output results, the H-type output results include the D-type output results, and D is a positive integer; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and using the first loss value to update the parameters in the first head network.

[0018] The first head network is any one of the N parallel head networks.

[0019] From a technical perspective, during each training process, after the parameters of the entire neural network are updated, the head network is adjusted by calculating the loss value corresponding to the output result of each head network, thereby achieving independent update and optimization of each head network, thereby achieving targeted optimization of each autonomous driving task and improving the accuracy of the output results corresponding to each autonomous driving task.

[0020] In a feasible implementation, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and using the second loss value to update the parameters in the L head networks.

[0021] Among them, the above-mentioned L head networks are also the head networks corresponding to the first autonomous driving task in the N head networks.

[0022] From the technical effect point of view, during each training process, after the parameters of the entire neural network are updated, the output results of the head network corresponding to each autonomous driving task are calculated to optimize one or more head networks corresponding to the autonomous driving task, so that one or more head networks corresponding to each autonomous driving task can be independently updated and optimized, thereby achieving targeted optimization of the corresponding autonomous driving tasks and improving the accuracy of the output results corresponding to each autonomous driving task.

[0023] In a feasible implementation, the multiple categories of output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the self-vehicle decision, the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle; the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task, the self-vehicle decision corresponds to the decision task, and the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle correspond to the planning task.

[0024] In a third aspect, the present application provides an autonomous driving method, comprising: obtaining input data, the input data comprising vehicle data, obstacle data, road topology data and navigation data; performing feature extraction and feature fusion on the input data using an expert hybrid MOE network in a neural network to obtain N types of feature data, the neural network also comprising N parallel head networks, where N is a positive integer; inputting each type of feature data into a corresponding head network of the N parallel head networks to perform an autonomous driving task corresponding to the head network; and controlling the vehicle based on the output result of each autonomous driving task.

[0025] In a feasible implementation, each training process of the neural network is as follows: each expert network in the MOE network is used to perform feature extraction and feature fusion on each training data in a batch of training data to obtain the N types of feature data; the neural network is also used to weight the feature data of the same type output by the MOE network respectively to obtain the weighted N types of feature data; and the weighted N types of feature data are respectively input into the corresponding head network to obtain H types of output results corresponding to each training data, the H types of output results include K types of output results, K is a positive integer less than or equal to H; the updating of the neural network parameters is completed according to a loss value corresponding to each training process, the loss value is obtained by weighting using K accumulated loss values, the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results in each type of output results in the K types of output results obtained from the batch of training data.

[0026] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0027] In a feasible implementation, the N parallel head networks include a first head network, which outputs D-type output results, and the multiple types of output results include the D-type output results, where D is a positive integer; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and using the first loss value to update the parameters in the first head network or adjust the structure of the first head network.

[0028] In a feasible implementation, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and using the second loss value to update the parameters in the L head networks.

[0029] In a feasible implementation, the MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; wherein the self-attention network is used to extract features from the input data, and the mutual attention network is used to perform feature fusion on the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

[0030] In a feasible implementation, the autonomous driving task includes a prediction task, a decision task and a planning task; the output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the self-vehicle decision, and the output result of the planning task includes the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle.

[0031] In a fourth aspect, the present application provides a neural network training device, wherein the neural network includes an expert hybrid MOE network and N parallel head networks, the MOE network includes M expert networks, M and N are positive integers; the device includes: an encoding unit, which is used to perform feature extraction and feature fusion on each training data in a batch of training data through each expert network in the M expert networks during each training process to obtain N types of feature data; and weight the feature data of the same type output by the M expert networks to obtain weighted N types of feature data; wherein the weighted N types of feature data correspond one-to-one to the N parallel head networks, and each of the N parallel head networks corresponds to an autonomous driving task, The autonomous driving task includes a prediction task, a decision task and a planning task; a decoding unit is used to input the weighted N types of feature data into the corresponding head network respectively to obtain H types of output results corresponding to each training data, and the H types of output results include K types of output results, K is a positive integer less than or equal to H; an updating unit is used to accumulate the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained from the batch of training data to obtain K accumulated loss values ​​corresponding to the K types of output results respectively; and weight the K accumulated loss values ​​to obtain a loss value corresponding to each training process; and use the one loss value to update the parameters in the neural network.

[0032] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0033] In a feasible implementation manner, the N parallel head networks include a first head network, the first head network outputs D-type output results, the H-type output results include the D-type output results, and D is a positive integer; after using the one loss value to update the parameters in the neural network, the encoding unit and the decoding unit are also used to: process G training data in a training set through the updated neural network to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; the updating unit is also used to accumulate the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; and weight the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and use the first loss value to update the parameters in the first head network.

[0034] In a feasible implementation, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after the parameters in the neural network are updated using the one loss value, the encoding unit and the decoding unit are further used to: process G training data in a training set by the updated neural network to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; the updating unit is further used to accumulate the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; and weight the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and use the second loss value to update the parameters in the L head networks.

[0035] In a feasible implementation, the H-type output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the self-vehicle decision, the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle; the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task, the self-vehicle decision corresponds to the decision task, and the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle correspond to the planning task.

[0036] In a fifth aspect, the present application provides an autonomous driving device, comprising: an acquisition unit, used to acquire input data, wherein the input data includes vehicle data, obstacle data, road topology data and navigation data; a processing unit, used to perform feature extraction and feature fusion on the input data using an expert hybrid MOE network in a neural network to obtain N types of feature data, wherein the neural network also includes N parallel head networks, where N is a positive integer; and inputting each type of feature data into a corresponding head network of the N parallel head networks to perform an autonomous driving task corresponding to the head network; a control unit, used to control the vehicle based on the output result of each autonomous driving task.

[0037] In a feasible implementation manner, during each training process of the neural network, the processing unit is specifically used to: perform feature extraction and feature fusion on each training data in a batch of training data through each expert network in the MOE network to obtain the N types of feature data; and respectively weight the feature data of the same type output by the MOE network through the neural network to obtain the weighted N types of feature data; and respectively input the weighted N types of feature data into the corresponding head network to obtain H types of output results corresponding to each training data, the H types of output results include K types of output results, K is a positive integer less than or equal to H; wherein, the updating of the neural network parameters is completed according to a loss value corresponding to each training process, the loss value is obtained by weighting using K accumulated loss values, the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results in each type of output results in the K types of output results obtained for the batch of training data.

[0038] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0039] In a feasible implementation manner, the N parallel head networks include a first head network, which outputs D types of output results, and the multiple types of output results include the D types of output results, where D is a positive integer; after using the one loss value to update the parameters in the neural network, the processing unit is also used to: use the updated neural network to process G training data in a training set to obtain the D types of output results output by the first head network, where the training set includes the one batch of training data, and G is a positive integer; accumulate the loss values ​​of part or all of the output results in each type of output results in the D types of output results to obtain D accumulated loss values ​​corresponding to the D types of output results respectively; weight the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and use the first loss value to update the parameters in the first head network.

[0040] In a feasible implementation, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after using the one loss value to update the parameters in the neural network, the processing unit is also used to: use the updated neural network to process G training data in the training set to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; accumulate the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; weight the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and use the second loss value to update the parameters in the L head networks.

[0041] In a feasible implementation, the MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; wherein the self-attention network is used to extract features from the input data, and the mutual attention network is used to perform feature fusion on the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

[0042] In a feasible implementation, the autonomous driving task includes a prediction task, a decision task and a planning task; the output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the self-vehicle decision, and the output result of the planning task includes the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle.

[0043] In the sixth aspect, an embodiment of the present application provides a chip system, which includes at least one processor, a memory and an interface circuit, wherein the memory, the interface circuit and the at least one processor are interconnected through lines, and instructions are stored in the at least one memory; when the instructions are executed by the processor, any method described in the aforementioned second aspect is implemented.

[0044] In the seventh aspect, an embodiment of the present application provides a chip system, which includes at least one processor, a memory and an interface circuit, wherein the memory, the interface circuit and the at least one processor are interconnected through lines, and instructions are stored in the at least one memory; when the instructions are executed by the processor, any method described in the third aspect above is implemented.

[0045] In an eighth aspect, an embodiment of the present application provides a server, wherein the server comprises the chip system described in the sixth or seventh aspect.

[0046] In a ninth aspect, an embodiment of the present application provides a vehicle, comprising the chip system described in the seventh aspect.

[0047] In the tenth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the method described in any one of the second aspect and / or the third aspect above is implemented.

[0048] In the eleventh aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, the method described in any one of the second aspect and / or the third aspect above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The following is an introduction to the drawings used in the embodiments of the present application.

[0050] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a process of a neural network performing an autonomous driving task provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a flow chart of a method for executing an autonomous driving process using a neural network provided in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of a neural network for autonomous driving provided in this application;

[0054] Figure 5 A flowchart of a neural network training method provided in an embodiment of the present application;

[0055] Figure 6a-6b A schematic diagram of the training process of a neural network provided in an embodiment of the present application;

[0056] Figure 7 A schematic diagram of a process flow of an autonomous driving method provided in an embodiment of the present application;

[0057] Figure 8 A schematic diagram of a chip hardware structure provided in an embodiment of the present application;

[0058] Fig. 9 A neural network training device provided in an embodiment of the present application;

[0059] Fig.10 An automatic driving device provided in an embodiment of the present application;

[0060] Fig.11 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0062] The terms "first", "second", "third" and "fourth" in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. Mentioning "embodiment" in this article means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase at various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0063] The following are the professional terms involved in this application:

[0064] (1) Prediction task: In autonomous driving, the prediction task is used to predict the movement of an obstacle in the future. In this application, the output results corresponding to the prediction task include the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle.

[0065] (2) Decision-making tasks: In autonomous driving, decision-making tasks are used to output ego-vehicle decisions. In this application, ego-vehicle decisions include ego-vehicle avoidance decisions, ego-vehicle preemption decisions, and obstacle ignoring decisions when an obstacle is detected near the ego-vehicle.

[0066] (3) Planning task: In autonomous driving, the planning task is used to output the driving route of the vehicle in the future. In this application, the output results corresponding to the planning task include the planned road of the vehicle, the planned trajectory of the vehicle, and the planned trajectory distribution of the vehicle.

[0067] (4) Mixture of Experts (MOE) network: It includes multiple expert networks and gate networks. The internal structure of each expert network can be the same or different. During the training process, each expert network has different focuses when extracting features. The gate network is used to weight the features extracted by each expert network. During the reasoning process, the gate network is usually used to decide which expert network to activate, and only the activated expert network is used for feature extraction.

[0068] (5) Self-Attention Network and Cross-Attention Network: The input of the Self-Attention network comes from the same sequence; the input of the Cross-Attention network comes from different sequences (such as text, sound, image, etc.), but these different sequences need to have the same dimension.

[0069] (6) Own vehicle data: including the positioning of the own vehicle, which is characterized by the position and orientation of the own vehicle.

[0070] (7) Obstacle data: including the position, direction, speed, length and width of each obstacle. Obstacles include other vehicles, pedestrians, and other moving or stationary objects or plants and animals within a certain range from the vehicle, which are not listed in this application.

[0071] (8) Road topology data: including road center lines, side lines, traffic lights, speed limit signs, etc., and their corresponding attribute data. The attribute data of center lines and side lines include whether they are dashed lines, solid lines, and their lengths; the attributes of speed limit signs include speed limit values; and the attributes of traffic lights include the current color of the traffic lights and the waiting time.

[0072] (9) Navigation data: Before outputting the control data of the ego vehicle, the navigation data of the current ego vehicle is obtained, including the road sections, lanes, and trajectories planned for the ego vehicle in the future.

[0073] See also Figure 1 , Figure 1 An application scenario schematic diagram provided for an embodiment of the present application is used to describe the scenario used to execute the autonomous driving method in the present application.

[0074] exist Figure 1 In the present application, the self-vehicle is a vehicle that executes the automatic driving method in the present application, and the self-vehicle communicates with the server through the network.

[0075] The server is deployed with the neural network for autonomous driving in this application. The server collects data transmitted by vehicles on the road and then generates training data for training the neural network. Each piece of training data includes vehicle data, navigation data, road topology data, obstacle data, etc. The server regularly trains the neural network for autonomous driving deployed thereon with the training data in the training set to update model parameters and / or adjust the model structure.

[0076] When the vehicle is driving, it will collect data about the environment within a certain range from the vehicle through the sensors and cameras it carries. Figure 1The data of other vehicles, lane lines, pedestrians, traffic lights, etc. shown in the figure are used to generate corresponding vehicle data, obstacle data, road topology data, etc. Among them, the neural network for autonomous driving may be deployed or not deployed on the autonomous vehicle.

[0077] When the neural network is deployed on the self-driving vehicle, the server will periodically send the updated parameters of the neural network to the self-driving vehicle. During the autonomous driving process, the self-driving vehicle can input the above-generated self-driving data, obstacle data, road topology data, and navigation data into the neural network deployed on it, use the neural network to perform corresponding autonomous driving tasks, and generate self-driving vehicle control data based on the output results of the autonomous driving tasks to complete the control of the self-driving vehicle.

[0078] When the neural network is not deployed on the vehicle, the vehicle will send the above-mentioned collected data and its navigation data to the server through the network when performing autonomous driving. The server generates the input data required for the neural network to perform the autonomous driving task (i.e., vehicle data, obstacle data, road topology data, and navigation data), and inputs the input data into the neural network. The neural network outputs the vehicle control data and sends this control data to the vehicle through the network. The vehicle controls the vehicle based on the received control data.

[0079] Optionally, the input data of the neural network is vectorized data. Specifically, for the vehicle data and obstacle data, the vectorized form is characterized by the position, orientation, size, etc. of the vehicle and obstacles at different times in the past (e.g., the past 2s from the time of collection). For road topology data, the vectorized form is characterized by the attributes of the center line, side line, traffic light, speed limit sign, etc. on a section of road (e.g., 10-20m near the vehicle) at different times in the past. The vectorized form of navigation data is represented by the driving roads, driving lanes, and driving trajectories of the vehicle in the past.

[0080] See also Figure 2 , Figure 2 A schematic diagram of the process of a neural network performing an autonomous driving task provided in an embodiment of the present application is used to summarize the implementation process of the autonomous driving method in the present application. Figure 2 The neural network in the middle is a neural network used for autonomous driving, which includes a backbone network and N parallel head networks.

[0081] like Figure 2 As shown in the figure, the vehicle uses the sensors and cameras ( Figure 2The neural network is used to collect data (not shown) to obtain corresponding perception data, and then pre-process the perception data, vehicle data, navigation data, etc. (i.e., vectorization processing) to obtain the input data of the neural network (i.e., vectorized vehicle data, obstacle data, road topology data, and navigation data).

[0082] The backbone network is used to extract and fuse features of the vectorized input data, and then input the obtained feature data into the corresponding head networks respectively. Each head network is used to perform the corresponding autonomous driving task, and each head network outputs at least one type of output result.

[0083] The autonomous driving tasks in this application include prediction tasks, decision tasks and planning tasks. Each head network corresponds to an autonomous driving task, and the autonomous driving tasks corresponding to different head networks are the same or different, that is, each autonomous driving task corresponds to at least one head network.

[0084] The output results corresponding to each autonomous driving task are post-processed. The post-processing converts the predicted trajectory of obstacles, the ego vehicle decision and the planned trajectory output by the head network into the final control data of the ego vehicle. Specifically, the post-processing includes checking the validity of the predicted trajectory of obstacles and the planned trajectory of the ego vehicle, as well as smoothing and stabilizing the trajectory.

[0085] The vehicle controls the running of the vehicle based on the above control data.

[0086] The following will use Figure 3 and Figure 4 To describe the structure of the neural network used for autonomous driving in this application and the execution process of the neural network in executing the autonomous driving method of this application. Figure 3 A flowchart of a method for executing an autonomous driving process using a neural network provided in an embodiment of the present application. Figure 4 A schematic diagram of the structure of a neural network for autonomous driving provided in this application.

[0087] like Figure 4 As shown, the neural network includes a backbone network and N parallel head networks. The backbone network includes an expert mixture MOE network. N is a positive integer.

[0088] The corresponding execution process of the neural network in the autonomous driving method includes step S310 and step S320.

[0089] Step S310: The MOE network is used to perform feature extraction and feature fusion on the input data to obtain N types of feature data; wherein the input data includes vehicle data, obstacle data, road topology data and navigation data, and the N types of feature data correspond one-to-one to N parallel head networks, and each of the N parallel head networks corresponds to an autonomous driving task.

[0090] Among them, the above input data is vectorized data.

[0091] Among them, the MOE network includes multiple expert networks ( Figure 4 Each expert network includes a self-attention network, a mutual attention network, and a multilayer perceptron (MLP).

[0092] Among them, the self-attention network is used to extract features from the input data, and the mutual attention network is used to fuse the features extracted by the self-attention network. The fused results are then input into a multi-layer perception network for processing to obtain N types of feature data output by each expert network based on the input data (or a piece of training data).

[0093] The above-mentioned multiple expert networks include a first expert network, and the first expert network is any one of the multiple expert networks. Figure 4 An example of the internal structure of an expert network is given (i.e. Figure 4 The expert network 1) is used to specifically describe the process of the first expert network executing the autonomous driving method of the present application.

[0094] Optionally, the first expert network includes a first self-attention network, a first mutual attention network, a second self-attention network, and a second mutual attention network.

[0095] Among them, Figure 4 As shown, the first self-attention network is used to extract features from the vehicle data and obstacle data in an input data, and obtains the first query matrix, the first key matrix and the first value matrix (i.e. Figure 4 The second self-attention network is used to extract features of the road topology data and navigation data in an input data, and obtains the second query matrix, the second key matrix, and the second value matrix (i.e. Figure 4 Q, K, and V output by the second self-attention network in .

[0096] like Figure 4 As shown, the first mutual attention network is used to perform feature fusion on the first Query matrix, the second Key matrix and the second Value matrix; the second mutual attention network is used to perform feature fusion on the second Query matrix, the first Key matrix and the first Value matrix. The N types of feature data are generated through the output results of the first mutual attention network and the output results of the second mutual attention network.

[0097] Optionally, the N types of feature data output by the first expert network are generated by inputting the output results of the first mutual attention network and the second mutual attention network into a structure such as a multilayer perceptron.

[0098] Optionally, the N types of feature data include prediction features, decision features, road features, and vehicle features. Each type of feature data will be subsequently input into a corresponding head network to perform a corresponding autonomous driving task.

[0099] It should be understood that the above-mentioned internal structure of the expert network is only a feasible example given in the present application and does not constitute a limitation. The internal structure of each expert network can be the same or different, and different expert networks have different focuses when extracting features from input data.

[0100] The following describes the role of the gate network in the MOE network:

[0101] During the training process: each training data is input separately Figure 4 In the M expert networks, N types of feature data are obtained from each expert network. At this time, the gate network will weight the same type of feature data in the N types of feature data output by the M expert networks to obtain the weighted N types of feature data.

[0102] Among them, for multiple feature data of the same type, the weight corresponding to each feature data when the multiple feature data are weighted is related to the expert network that outputs the feature data.

[0103] During the reasoning process, the gate network activates one of the M expert networks based on the weight of each expert network to perform the reasoning process, and then inputs the N types of feature data output by the expert network into the corresponding head network to perform the corresponding autonomous driving task. For example, the gate network can activate the expert network with the largest weight among the M expert networks to perform the reasoning process.

[0104] Step S320: Each head network performs a corresponding autonomous driving task based on a corresponding input feature data; wherein the autonomous driving task includes a prediction task, a decision task and a planning task.

[0105] Among them, the prediction features in the above N types of feature data are used to perform prediction tasks, the decision features are used to perform decision tasks, and the road features and vehicle features are used to perform planning tasks.

[0106] Among them, each head network corresponds to an autonomous driving task, and each autonomous driving task corresponds to at least one head network, that is, each autonomous driving task is executed by at least one head network.

[0107] Each of the N head networks outputs at least one type of output result.

[0108] In a feasible implementation, the prediction task is performed by one head network, the decision task is performed by one head network, and the planning task is performed by two head networks. The output features of the head network that performs the prediction task are prediction features, the input features of the head network that performs the decision task are decision features, and the input features of the two head networks that perform the planning task are vehicle features and road features, respectively.

[0109] Optionally, the output result corresponding to the prediction task includes a predicted trajectory of the obstacle and a distribution of the predicted trajectory of the obstacle.

[0110] Optionally, the output result corresponding to the decision task includes the self-vehicle decision, and the self-vehicle decision includes the self-vehicle preemption decision, the self-vehicle avoidance decision, and ignoring the obstacle.

[0111] Optionally, the output results corresponding to the planning task include the planned road of the vehicle, the planned trajectory of the vehicle, and the planned trajectory distribution of the vehicle.

[0112] The predicted trajectory of the obstacle is the movement trajectory of the obstacle in the future. The predicted trajectory distribution of the obstacle is the probability value of the trajectory points on the movement trajectory of the obstacle in the future falling on or around different position points.

[0113] Among them, the ego vehicle decision is used to provide the ego vehicle with corresponding decision-making behaviors when it is predicted that an obstacle will be encountered in the future, including preemption decision, avoidance decision and ignoring the obstacle.

[0114] The planned road is the planning result of the output road scale, and the planned trajectory is the planning result of the output trajectory scale. The planned trajectory distribution is the probability value of the trajectory points on the planned trajectory of the vehicle in the future falling on or around different position points.

[0115] See also Figure 5 , Figure 5 A flowchart of a neural network training method provided in an embodiment of the present application is used to describe the neural network used for autonomous driving in the present application (i.e., the aforementioned Figure 4 The training method of neural network.

[0116] like Figure 5 As shown, the training method includes step S510, step S520, step S530, step S540, and step S550. Figure 5 The training method shown is described by taking a training process as an example. Each training process includes two parts: single-task training / multi-task joint training, and single-task optimization.

[0117] Step S510: In each training process, each expert network in the M expert networks performs feature extraction and feature fusion on each piece of training data in a batch of training data to obtain N types of feature data.

[0118] A batch of training data used in each training is located in a training set, and the batch of training data includes multiple training data. Each training data in the training set is the data in the aforementioned vectorized form.

[0119] Specifically, each piece of training data in the batch of training data is input into the M expert networks, and each of the M expert networks outputs N types of feature data based on each piece of training data. For each piece of training data, the N types of feature data output by each expert network are output into the gate network, and the gate network performs the weighted processing in step S520.

[0120] Step S520: weighting the same type of feature data output by the M expert networks respectively to obtain N weighted feature data; wherein the N weighted feature data correspond one-to-one to the N parallel head networks, each of the N parallel head networks corresponds to an autonomous driving task, and the autonomous driving tasks include prediction tasks, decision-making tasks and planning tasks.

[0121] Specifically, for each piece of training data in a batch of training data, each expert network will output N types of feature data, the number of which is N. For each piece of training data, M expert networks will also output N types of feature data, the number of each type of feature data in the N types of feature data is M. Then, the M pieces of each type of feature data are weighted to obtain the weighted N types of feature data.

[0122] Among them, for the M pieces of feature data in each type of feature data, the weight corresponding to each piece of feature data when weighted is related to the expert network that outputs the piece of feature data.

[0123] Among them, each weighted feature data corresponds to a head network, each head network corresponds to an autonomous driving task, and each autonomous driving task corresponds to at least one head network, that is, there is at least one head network for performing each autonomous driving task.

[0124] After obtaining the weighted N types of feature data, each type of weighted feature data will be input into a corresponding head network to perform the autonomous driving task corresponding to the head network.

[0125] Step S530: Input the weighted N types of feature data into the corresponding head network respectively to obtain H types of output results corresponding to each training data, and the H types of output results include K types of output results, where K is a positive integer less than or equal to H.

[0126] For each piece of training data, N head networks output a total of H types of output results, and each head network outputs at least one type of output result.

[0127] Optionally, the K-type output results in the above H-type output results correspond to at least one autonomous driving task (or at least one head network corresponding to the K-type output results). The head network used to output the K-type output results will perform parameter updates in the subsequent single-task training / multi-task joint training process (i.e., the single-task training / multi-task joint training process described below), while other head networks will not perform parameter updates in this process.

[0128] Step S540: Accumulate the loss values ​​of part or all of the output results in each category of the K categories of output results obtained from the one batch of training data to obtain K accumulated loss values ​​corresponding to the K categories of output results.

[0129] Specifically, for each piece of training data, the N head networks output H types of output results (the number is also H, that is, the number of output results of each type is 1). Assuming that the batch of training data contains F pieces of training data, H types of output results will also be obtained based on the batch of training data, and the number of H types of output results is H*F, where the number of output results of each type is F.

[0130] The above K categories of output results are selected from the H categories of output results obtained from the batch of training data. The number of the K categories of output results is K*F. The loss values ​​of part or all of the F output results in each category of the K categories of output results are accumulated to obtain the accumulated loss value corresponding to each category of output results, and a total of K accumulated loss values ​​are obtained.

[0131] Among them, the above-mentioned H-type output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the decision of the ego vehicle, the planned road of the ego vehicle, the planned trajectory of the ego vehicle and the planned trajectory distribution of the ego vehicle.

[0132] Among them, the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task. The decision of the ego vehicle corresponds to the decision task. The planned path of the ego vehicle, the planned trajectory of the ego vehicle and the planned trajectory distribution of the ego vehicle correspond to the planning task.

[0133] The following is a possible way to select some or all of the output results from each type of output results and then accumulate their loss values:

[0134] The description is performed with the first type of output results among the K types of output results as an object, and the first type of output results is any type of output results among the K types of output results.

[0135] Since this selection method is based on the task labels of the training data, the meaning of the task labels carried by the training data is first introduced, as shown in Table 1. Among them, each training data in the training set may or may not carry a task label, and further, each training data carrying a task label may carry one or more task labels.

[0136] As shown in Table 1, if the training data carries a planning task label, the loss value of the output result corresponding to the planning task in the H-type output results obtained based on the training data will be used for the above-mentioned cumulative calculation. Optionally, the planning task label can be marked on the ego vehicle data in the training data.

[0137] Similarly, if the training data carries a decision or prediction task label, the loss value of the output result corresponding to the decision or prediction task in the H-type output results obtained based on the training data will be used for the above cumulative calculation. Optionally, the decision task label and the prediction task label can be marked on the obstacle data in the training data.

[0138] Similarly, if the training data does not carry a task label (ie, none), the loss value of the H-type output result obtained based on the training data will not be used for the above cumulative calculation.

[0139] Table 1. Example of task labels for training data

[0140] A training data Task Tags meaning Self-driving car data planning This data will be used to calculate the loss value of the planning task Obstacle data Decision making / forecasting This data will be used to calculate the loss value of the decision or prediction task none This data will not be used to calculate the loss value of planning / decision-making and prediction tasks

[0141] Taking the aforementioned batch of training data containing F training data as an example, the K-type output results obtained from a batch of training data include F first-type output results. If the F first-type output results all correspond to the first autonomous driving task in the above-mentioned autonomous driving task, that is, the F first-type output results are all obtained by the head network output corresponding to the first autonomous driving task. If the F first-type output results include E first-type output results, and the training data used to calculate the E first-type output results all carry labels representing the first autonomous driving task, then the accumulated loss value corresponding to the first-type output result is obtained by accumulating the E loss values ​​corresponding to the E first-type output results, and E is a positive integer less than or equal to F.

[0142] After calculating and obtaining K accumulated loss values ​​corresponding to K types of output results, the following step S550 is executed.

[0143] Step S550: weighting the K accumulated loss values ​​to obtain a loss value corresponding to each training process; and using the loss value to update the parameters in the neural network.

[0144] Among them, the weight corresponding to each type of output result in the K types of output results is related to the autonomous driving task corresponding to the type of output result.

[0145] After obtaining a loss value corresponding to each training process, the parameters in the neural network (backbone network + head network for outputting the K-type output results) are updated using the loss value.

[0146] From the technical effect point of view, the present application can target specific application scenarios, thereby selecting K-type output results from H-type output results during the training process (the K-type output results correspond to at least one autonomous driving task, that is, corresponding to a specific application scenario), and then updating the parameters in at least one head network corresponding to the K-type output results to achieve single-task training or multi-task joint training (for example, for weak interaction scenarios such as vulnerable road users (Vulnerable Road Users, VRU) occupying the lane in the same direction ahead and the vehicle avoiding obstacles and overtaking, joint training of planning tasks and prediction tasks is performed; for strong interaction scenarios such as the vehicle changing lanes and vehicles coming from behind the target lane, joint training of planning tasks, decision-making tasks and prediction tasks is performed).

[0147] At the same time, by selecting part or all of the output results of each category in K categories for training, it is possible to select corresponding data for training for each autonomous driving task, thereby ensuring the generalization of each task (for example, data on driving on open roads has no value for decision-making and prediction tasks, but is valuable for training planning tasks).

[0148] After updating the parameters of the above neural network, single-task optimization can also be performed on the head network or the head network corresponding to a single autonomous driving task to achieve independent adjustment of each autonomous driving task. This optimization process can be specifically divided into the following two methods:

[0149] (I) Optimize a single head network independently

[0150] The following describes the optimization process of the first head network among the N parallel head networks as an object, where the first head network is any one of the N parallel head networks.

[0151] The first head network outputs D-type output results, and the above-mentioned H-type output results include D-type output results, where D is a positive integer. The G training data in the training set are respectively input into the updated neural network to obtain D-type output results output by the first head network. The number of output results contained in each category of the D-type output results is G, that is, a total of D*G output results are obtained, where G is a positive integer.

[0152] The G pieces of training data may be completely identical, completely identical or partially identical to the training data contained in the aforementioned batch of training data.

[0153] For the D types of output results obtained from the G training data, the loss values ​​of part or all of the output results in each type of output result are accumulated to obtain D accumulated loss values ​​corresponding to the D types of output results.

[0154] The process of selecting part or all of the output results from the G output results included in each type of output results can refer to the process of selecting using the training data task labels in the aforementioned embodiment, which will not be repeated here.

[0155] After obtaining D accumulated loss values, the D accumulated loss values ​​are weighted to obtain a first loss value corresponding to the first network in each training process; and then the first loss value is used to update the parameters in the first network.

[0156] Among them, D accumulated loss values ​​correspond to D types of output results, and the weight corresponding to each accumulated loss value is related to the autonomous driving task category corresponding to the output result of calculating the accumulated loss value.

[0157] Among them, the above-mentioned updating of the parameters in the first head network can be carried out in any one of the following two ways: (1) only updating the parameters in the first head network, and not adjusting the structure of the first head network; (2) adjusting the structure of the first head network, and simultaneously updating the parameters in the adjusted first head network.

[0158] That is, the structure of the updated first network is the same as or different from the structure of the first network in the aforementioned updated neural network (that is, the neural network obtained after single-task training or multi-task joint training) (that is, after the above-mentioned update of the parameters of the first network, the structure of the first network changes or remains unchanged compared to before the update).

[0159] (II) Optimizing the head network corresponding to a single autonomous driving task

[0160] The following describes a process of optimizing the L head networks corresponding to the first autonomous driving task in the N parallel head networks, wherein the first autonomous driving task is any one of a plurality of autonomous driving tasks.

[0161] The above L head networks output P types of output results. In the above embodiment, the H type of output results include the P type of output results. L and P are positive integers greater than or equal to 2.

[0162] The G training data in the training set are respectively input into the updated neural network to obtain P types of output results output by the L head networks. The number of output results contained in each category of the P types of output results is G, that is, a total of P*G output results are obtained, and G is a positive integer.

[0163] The G pieces of training data may be completely identical, completely identical or partially identical to the training data contained in the aforementioned batch of training data.

[0164] The loss values ​​of part or all of the output results in each category of P categories of output results are accumulated to obtain P accumulated loss values ​​corresponding to the P categories of output results.

[0165] After obtaining P accumulated loss values, the P accumulated loss values ​​are weighted to obtain the second loss value corresponding to the L head networks in each training process.

[0166] The parameters in the L head networks are updated using the second loss value. The updating of the parameters in the L head networks can be performed in any of the following two ways: (1) only updating the parameters in the L head networks without adjusting the structure of the L head networks; (2) adjusting the structure of the L head networks and updating the parameters in the adjusted L head networks at the same time.

[0167] That is, the structure of each head network in the updated L head networks is the same as or different from the structure of the corresponding head network in the aforementioned updated neural network (that is, after the parameters of the L head networks are updated as described above, the structure of the L head networks changes or remains unchanged compared to before the update).

[0168] The relevant steps in the above process can be found in the corresponding description above and will not be repeated here.

[0169] See also Figure 6a-Figure 6b , Figure 6a-6b A schematic diagram of the training process of a neural network provided in an embodiment of the present application. Figure 6a Schematic diagram of the process of single-task training / multi-task joint training in each training process; Figure 6b Schematic diagram of the process of single-task optimization during each training process.

[0170] like Figure 6a As shown in the figure, in single-task training / multi-task joint training, a batch of training data in the training set is input into the neural network (the specific structure and execution process of the neural network can be found in the above Figure 2-Figure 5 In the backbone network of the embodiment described in the embodiment, the backbone network extracts features from each training data in the batch of training data to obtain weighted N types of feature data. Then each type of weighted N feature data is respectively input into a corresponding head network, and each head network will output at least one type of output result, and a total of H types of output results are obtained. Based on the specific application scenario, K types of output results are selected from the H types of output results, and the loss values ​​of part or all of the output results in each type of output results in the K types of output results are accumulated to obtain K accumulated loss values ​​corresponding to the K types of output results. Finally, the K accumulated loss values ​​are weighted to obtain a loss value corresponding to the single-task training / multi-task joint training process, and the one loss value is used for back propagation to update the parameters in the entire neural network.

[0171] Specifically, the specific process of the above single-task training / multi-task joint training can be found in the above Figure 2-Figure 5 The corresponding description in the embodiments will not be repeated here.

[0172] After completing the above single-task training / multi-task joint training, the updated neural network is optimized for a single task, that is, Figure 6b The neural network is completed Figure 6a The neural network obtained after single-task training / multi-task joint training.

[0173] like Figure 6b As shown, in the single-task optimization process, the training data in the training set (such as the number of G in the above embodiment) is input into the updated neural network to obtain the weighted N feature data corresponding to the G training data, and then the weighted N feature data are respectively input into a corresponding head network to obtain H types of output results. Finally, refer to the above Figure 5 The description in the embodiment performs single-task optimization on a single head network or on at least one head network corresponding to a single autonomous driving task to update the parameters in the corresponding head network.

[0174] Specifically, the specific process of the above single task optimization can be found in the above Figure 5 The corresponding description in the embodiments will not be repeated here.

[0175] See also Figure 7 , Figure 7 A flowchart of an automatic driving method provided in an embodiment of the present application is shown below. Figure 7 As shown, the method includes step S710, step S720, step S730 and step S740. Among them,

[0176] Step S710: Acquire input data, where the input data includes vehicle data, obstacle data, road topology data, and navigation data.

[0177] Specifically, when the trained neural network is deployed on the vehicle, the vehicle perceives the surrounding environment based on the cameras, sensors, lidar and other equipment carried thereon, and vectorizes the perceived data and the navigation data of the vehicle to obtain the input data.

[0178] When the above-mentioned neural network for autonomous driving is not deployed on the vehicle side but on a cloud server, the vehicle perceives the surrounding environment based on the cameras, sensors, lidar and other equipment installed on it, and can upload the perceived data and the navigation data of the vehicle to the server. The server vectorizes the received data to obtain the above-mentioned input data.

[0179] Step S720: using the expert mixture MOE network in the neural network to perform feature extraction and feature fusion on the input data to obtain N types of feature data, the neural network also includes N parallel head networks, N is a positive integer.

[0180] Step S730: Input each of the feature data into a corresponding head network among the N parallel head networks to perform an autonomous driving task corresponding to the head network.

[0181] Specifically, the specific execution process of the above-mentioned step S720 and step S730 can be found in the description of the above-mentioned embodiment, and will not be repeated here.

[0182] Step S740: Control the vehicle based on the output results of each of the autonomous driving tasks.

[0183] Specifically, control data of the vehicle is generated based on the output result of the neural network, and then the vehicle is controlled based on the control data.

[0184] Among them, when the trained neural network is deployed on the vehicle, the control data is generated by the vehicle; when the neural network is deployed on the server, the control information is generated by the server and sent to the vehicle.

[0185] Optionally, each training process of the above-mentioned neural network is as follows: each expert network in the MOE network is used to perform feature extraction and feature fusion on each training data in a batch of training data to obtain the N types of feature data; the neural network is also used to weight the feature data of the same type output by the MOE network respectively to obtain the weighted N types of feature data; and the weighted N types of feature data are respectively input into the corresponding head network to obtain H types of output results corresponding to each training data, and the H types of output results include K types of output results, K is a positive integer less than or equal to H; the updating of the neural network parameters is completed according to a loss value corresponding to each training process, and the loss value is obtained by weighting using K accumulated loss values, and the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results in each type of output results in the K types of output results obtained from the batch of training data.

[0186] Optionally, the K categories of output results include first category output results, and the K categories of output results obtained from the batch of training data include F first category output results, and the F first category output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first category output results include E first category output results, and the training data used to calculate the E first category output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first category output result is obtained by accumulating E loss values ​​corresponding to the E first category output results, respectively, and E is a positive integer less than or equal to F.

[0187] Optionally, the N parallel head networks include a first head network, which outputs D-type output results, and the multiple types of output results include the D-type output results, where D is a positive integer; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and using the first loss value to update the parameters in the first head network.

[0188] Optionally, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P type output results, the H type output results include the P type output results, L and P are positive integers greater than or equal to 2; after using the one loss value to update the parameters in the neural network, the method also includes: inputting G training data in the training set into the updated neural network respectively to obtain the P type output results output by the L head networks, the training set includes the one batch of training data, G is a positive integer; accumulating the loss values ​​of part or all of the output results in each type of output results in the P type output results to obtain P accumulated loss values ​​corresponding to the P type output results respectively; weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and using the second loss value to update the parameters in the L head networks.

[0189] Optionally, the MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; wherein the self-attention network is used to extract features from the input data, and the mutual attention network is used to perform feature fusion on the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

[0190] Optionally, the autonomous driving task includes a prediction task, a decision task and a planning task; the output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the ego vehicle decision, and the output result of the planning task includes the planned road of the ego vehicle, the planned trajectory of the ego vehicle and the planned trajectory distribution of the ego vehicle.

[0191] Specifically, the specific structure of the neural network used in the above-mentioned autonomous driving method and the corresponding training and reasoning process can be found in the description in the above-mentioned embodiments, which will not be repeated here.

[0192] See also Figure 8 , Figure 8 A schematic diagram of a chip hardware structure provided for an embodiment of the present application can be used to execute the training method and / or the autonomous driving method in the embodiment of the present application.

[0193] like Figure 8 As shown, the neural-networks processing unit (NPU) 80 is mounted on the host CPU (Host CPU) as a coprocessor, and the Host CPU assigns tasks to execute the neural network training method in the aforementioned embodiment or the related processes in the autonomous driving method in the aforementioned embodiment.

[0194] The core part of the NPU is the operation circuit 803, and the controller 804 controls the operation circuit 803 to extract data from the memory (weight memory or input memory) and perform operations.

[0195] In some implementations, the operation circuit 803 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 803 is a two-dimensional systolic array. The operation circuit 803 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 803 is a general-purpose matrix processor.

[0196] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 802 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 801 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 808accumulator.

[0197] The vector calculation unit 807 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 807 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0198] In some implementations, the vector calculation unit 807 can store the processed output vector to the unified memory 806. For example, the vector calculation unit 807 can apply a nonlinear function to the output of the operation circuit 803, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 807 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 803, such as for use in a subsequent layer in a neural network.

[0199] The unified memory 806 is used to store input data and output data.

[0200] The memory unit access controller 805 (Direct Memory Access Controller, DMAC) moves the input data in the external memory to the input memory 801 and / or the unified memory 806, stores the weight data in the external memory into the weight memory 802, and stores the data in the unified memory 806 into the external memory.

[0201] The bus interface unit (BIU) 810 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 809 through the bus.

[0202] An instruction fetch buffer 809 connected to the controller 804 is used to store instructions used by the controller 804 .

[0203] The controller 804 is used to call the instructions cached in the instruction fetch memory 809 to control the working process of the computing accelerator.

[0204] Generally, the unified memory 806, the input memory 801, the weight memory 802 and the instruction fetch memory 809 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated as DDR SDRAM), a high bandwidth memory (High Bandwidth Memory, HBM) or other readable and writable memory.

[0205] See also Fig. 9 , Fig. 9 A neural network training device provided in an embodiment of the present application is used to execute the neural network training method in the aforementioned embodiment. The neural network used for training may be the neural network in the aforementioned embodiment, that is, the neural network includes an expert mixed MOE network and N parallel head networks, and the MOE network includes M expert networks, where M and N are positive integers.

[0206] like Fig. 9 As shown, the device includes an encoding unit 910, a decoding unit 920 and an updating unit 930. Among them,

[0207] The encoding unit 910 is used to extract and fuse features of each training data in a batch of training data through each expert network in the M expert networks during each training process to obtain N types of feature data; and to weight the feature data of the same type output by the M expert networks to obtain weighted N types of feature data; wherein the weighted N types of feature data correspond to the N parallel head networks one by one, and each head network in the N parallel head networks corresponds to an autonomous driving task, and the autonomous driving tasks include prediction tasks, decision tasks, and planning tasks. The decoding unit 920 is used to input the weighted N types of feature data into the corresponding head networks to obtain H types of output results corresponding to each training data, and the H types of output results include K types of output results, where K is a positive integer less than or equal to H. The updating unit 930 is used to accumulate the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained by the one batch of training data to obtain K accumulated loss values ​​corresponding to the K types of output results respectively; and to weight the K accumulated loss values ​​to obtain a loss value corresponding to each training process; and to use the one loss value to update the parameters in the neural network.

[0208] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0209] In a feasible implementation manner, the N parallel head networks include a first head network, the first head network outputs D-type output results, the H-type output results include the D-type output results, and D is a positive integer; after the parameters in the neural network are updated using the one loss value, the encoding unit 910 and the decoding unit 920 are also used to: process G training data in a training set through the updated neural network to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; the updating unit is also used to accumulate the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; and weight the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and use the first loss value to update the parameters in the first head network.

[0210] In a feasible implementation manner, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after the parameters in the neural network are updated using the one loss value, the encoding unit 910 and the decoding unit 920 are also used to: process G training data in a training set by the updated neural network to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; the updating unit is also used to accumulate the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; and weight the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and use the second loss value to update the parameters in the L head networks.

[0211] In a feasible implementation, the H-type output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the self-vehicle decision, the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle; the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task, the self-vehicle decision corresponds to the decision task, and the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle correspond to the planning task.

[0212] Specifically, the specific execution process of each unit in the above-mentioned device can refer to the corresponding description of the above-mentioned embodiment, which will not be repeated here.

[0213] See also Fig.10 , Fig.10 An automatic driving device provided in an embodiment of the present application is used to execute the automatic driving method in the above embodiment. The device includes an acquisition unit 1010, a processing unit 1020 and a control unit 1030. Among them,

[0214] The acquisition unit 1010 is used to acquire input data, and the input data includes vehicle data, obstacle data, road topology data and navigation data. The processing unit 1020 is used to extract and fuse the input data using a mixture of experts MOE network in the neural network to obtain N types of feature data, and the neural network also includes N parallel head networks, N is a positive integer; and input each type of feature data into a corresponding head network in the N parallel head networks to perform an autonomous driving task corresponding to the head network. The control unit 1030 is used to control the vehicle based on the output results of each autonomous driving task.

[0215] In a feasible implementation manner, during each training process of the neural network, the processing unit 1020 is specifically used to: perform feature extraction and feature fusion on each training data in a batch of training data through each expert network in the MOE network to obtain the N types of feature data; and respectively weight the feature data of the same type output by the MOE network through the neural network to obtain the weighted N types of feature data; and respectively input the weighted N types of feature data into the corresponding head network to obtain H types of output results corresponding to each training data, the H types of output results include K types of output results, K is a positive integer less than or equal to H; wherein, the updating of the neural network parameters is completed according to a loss value corresponding to each training process, the loss value is obtained by weighting using K accumulated loss values, the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results in each type of output results in the K types of output results obtained for the batch of training data.

[0216] In a feasible implementation, the K types of output results include first type output results, the K types of output results obtained from the batch of training data include F first type output results, and the F first type output results all correspond to the first autonomous driving task in the autonomous driving tasks; the F first type output results include E first type output results, and the training data used to calculate the E first type output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first type output result is obtained by accumulating E loss values ​​corresponding to the E first type output results, respectively, and E is a positive integer less than or equal to F.

[0217] In a feasible implementation manner, the N parallel head networks include a first head network, and the first head network outputs D-type output results, and the multiple types of output results include the D-type output results, where D is a positive integer; after using the one loss value to update the parameters in the neural network, the processing unit 1020 is specifically used to: use the updated neural network to process G training data in a training set to obtain the D-type output results output by the first head network, and the training set includes the one batch of training data, and G is a positive integer; accumulate the loss values ​​of part or all of the output results in each type of output results in the D-type output results to obtain D accumulated loss values ​​corresponding to the D-type output results respectively; weight the D accumulated loss values ​​to obtain the first loss value corresponding to the first head network in each training process; and use the first loss value to update the parameters in the first head network.

[0218] In a feasible implementation, the N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, L and P are positive integers greater than or equal to 2; after using the one loss value to update the parameters in the neural network, the processing unit 1020 is also used to: use the updated neural network to process G training data in the training set to obtain the P-type output results output by the L head networks, the training set includes the one batch of training data, G is a positive integer; accumulate the loss values ​​of part or all of the output results in each type of output results in the P-type output results to obtain P accumulated loss values ​​corresponding to the P-type output results respectively; weight the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and use the second loss value to update the parameters in the L head networks.

[0219] In a feasible implementation, the MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; wherein the self-attention network is used to extract features from the input data, and the mutual attention network is used to perform feature fusion on the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

[0220] In a feasible implementation, the autonomous driving task includes a prediction task, a decision task and a planning task; the output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the self-vehicle decision, and the output result of the planning task includes the planned road of the self-vehicle, the planned trajectory of the self-vehicle and the planned trajectory distribution of the self-vehicle.

[0221] Specifically, the specific execution process of each unit in the above-mentioned automatic driving device can be found in the corresponding description in the aforementioned method embodiment, and will not be repeated here.

[0222] See also Fig.11 , Fig.11 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. Fig.11 As shown, the chip system includes at least one processor 1101, a memory 1102 and an interface circuit 1103; wherein at least one processor 1101, the memory 1102 and the interface circuit 1103 are connected via a bus 1104. wherein at least one processor 1101 may include Figure 8 The neural network processor shown.

[0223] The memory 1102 stores program instructions corresponding to the aforementioned neural network training method and / or the automatic driving method. The interface circuit 1103 is used to transmit input data. The memory 1102 implements the neural network training method and / or the automatic driving method in the aforementioned embodiment by executing the program instructions in the memory 1102.

[0224] Specifically, the specific execution process of the above chip system can refer to the corresponding description in the aforementioned method embodiment, which will not be repeated here.

[0225] The embodiment of the present application provides a server, which includes the aforementioned Fig.11 The chip system described in the embodiment.

[0226] The present application embodiment provides a vehicle, which includes the aforementioned Fig.11 The chip system described in the embodiment.

[0227] The embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the aforementioned Figure 3 and / or Figure 5 The method described in any one of the embodiments is implemented.

[0228] The embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, the aforementioned Figure 3 and / or Figure 5 The method described in any one of the embodiments is implemented.

[0229] In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited to the described order of actions, because according to this application, some steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0230] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of the above-mentioned units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0231] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0232] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A computer program product for implementing autonomous driving based on a deep learning architecture, the computer program product comprising a neural network, characterized in that: The neural network includes a mixture of experts MOE network and N parallel head networks, where N is a positive integer; wherein, The MOE network is used to extract and fuse features from input data to obtain N types of feature data; wherein the input data includes vehicle data, obstacle data, road topology data, and navigation data, and the N types of feature data correspond one-to-one to the N parallel head networks, and each of the N parallel head networks corresponds to an autonomous driving task; Each head network performs a corresponding autonomous driving task based on a corresponding input feature data; wherein the autonomous driving task includes a prediction task, a decision task and a planning task.

2. The computer program product according to claim 1, characterized in that The MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; Among them, the self-attention network is used to extract features from the input data, the mutual attention network is used to fuse the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

3. The computer program product according to claim 2, characterized in that The multiple expert networks include a first expert network, which is an expert network that performs feature extraction and feature fusion on the input data during the reasoning process, and the first expert network includes a first self-attention network, a first mutual attention network, a second self-attention network, and a second mutual attention network; The first self-attention network is used to perform feature extraction on the vehicle data and the obstacle data to obtain a first Query matrix, a first Key matrix and a first Value matrix; the second self-attention network is used to perform feature extraction on the road topology data and the navigation data to obtain a second Query matrix, a second Key matrix and a second Value matrix; The first mutual attention network is used to perform feature fusion on the first Query matrix, the second Key matrix and the second Value matrix; the second mutual attention network is used to perform feature fusion on the second Query matrix, the first Key matrix and the first Value matrix; the N types of feature data are generated through the output results of the first mutual attention network and the output results of the second mutual attention network.

4. The computer program product according to any one of claims 1 to 3, characterized in that The output result corresponding to the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle; The output result corresponding to the decision task includes self-vehicle decision, and the self-vehicle decision includes self-vehicle preemption decision, self-vehicle avoidance decision, and ignoring obstacles; The output results corresponding to the planning task include the planned road of the vehicle, the planned trajectory of the vehicle and the planned trajectory distribution of the vehicle.

5. A neural network training method, characterized in that: The neural network includes an expert mixture MOE network and N parallel head networks, the MOE network includes M expert networks, M and N are positive integers; the method includes: In each training process, each expert network in the M expert networks performs feature extraction and feature fusion on each piece of training data in a batch of training data to obtain N types of feature data; The feature data of the same type output by the M expert networks are weighted respectively to obtain N types of weighted feature data; wherein the N types of weighted feature data correspond one-to-one to the N parallel head networks, and each head network in the N parallel head networks corresponds to an autonomous driving task, and the autonomous driving tasks include prediction tasks, decision tasks, and planning tasks; Inputting the weighted N types of feature data into the corresponding head network respectively, to obtain H types of output results corresponding to each piece of training data, wherein the H types of output results include K types of output results, where K is a positive integer less than or equal to H; Accumulating the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained from the one batch of training data to obtain K accumulated loss values ​​corresponding to the K types of output results respectively; The K accumulated loss values ​​are weighted to obtain a loss value corresponding to each training process; and the parameters in the neural network are updated using the loss value.

6. The method according to claim 5, characterized in that The K types of output results include first type of output results, the K types of output results obtained from the one batch of training data include F first type of output results, and the F first type of output results all correspond to a first autonomous driving task in the autonomous driving tasks; The F first-category output results include E first-category output results, and the training data used to calculate the E first-category output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first-category output result is obtained by adding the E loss values ​​corresponding to the E first-category output results, respectively, and E is a positive integer less than or equal to F.

7. The method according to claim 5 or 6, characterized in that: The N parallel head networks include a first head network, the first head network outputs a D-type output result, the H-type output result includes the D-type output result, and D is a positive integer; After the parameters in the neural network are updated using the one loss value, the method further includes: Input G training data in the training set into the updated neural network respectively to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the D categories of output results to obtain D accumulated loss values ​​corresponding to the D categories of output results respectively; Weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; The first loss value is used to update parameters in the first head network.

8. The method according to claim 6, characterized in that The N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; After the parameters in the neural network are updated using the one loss value, the method further includes: Input G training data in the training set into the updated neural network respectively to obtain the P types of output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the P categories of output results to obtain P accumulated loss values ​​corresponding to the P categories of output results; Weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; The parameters in the L head networks are updated using the second loss value.

9. The method according to claim 8, characterized in that The H-type output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the ego vehicle decision, the planned path of the ego vehicle, the planned trajectory of the ego vehicle, and the planned trajectory distribution of the ego vehicle; The predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task, the ego vehicle decision corresponds to the decision task, and the planned road of the ego vehicle, the planned trajectory of the ego vehicle and the planned trajectory distribution of the ego vehicle correspond to the planning task.

10. An automatic driving method, characterized in that: The method comprises: Acquiring input data, wherein the input data includes vehicle data, obstacle data, road topology data, and navigation data; Using a mixture of experts MOE network in a neural network to perform feature extraction and feature fusion on the input data to obtain N types of feature data, the neural network also includes N parallel head networks, where N is a positive integer; Inputting each of the feature data into a corresponding head network among the N parallel head networks to perform an autonomous driving task corresponding to the head network; The vehicle is controlled based on the output results of each of the autonomous driving tasks.

11. The method according to claim 10, characterized in that Each training process of the neural network is as follows: each expert network in the MOE network is used to perform feature extraction and feature fusion on each piece of training data in a batch of training data to obtain the N types of feature data; the neural network is also used to weight the feature data of the same type output by the MOE network to obtain the weighted N types of feature data; and the weighted N types of feature data are respectively input into the corresponding head network to obtain H types of output results corresponding to each piece of training data, and the H types of output results include K types of output results, where K is a positive integer less than or equal to H; Among them, the updating of the neural network parameters is completed according to a loss value corresponding to each training process, and the loss value is obtained by weighting K accumulated loss values, and the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained from the batch of training data.

12. The method according to claim 11, characterized in that The K types of output results include first type of output results, the K types of output results obtained from one batch of training data include F first type of output results, and the F first type of output results all correspond to a first autonomous driving task in the autonomous driving tasks; The F first-category output results include E first-category output results, and the training data used to calculate the E first-category output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first-category output result is obtained by adding the E loss values ​​corresponding to the E first-category output results, respectively, and E is a positive integer less than or equal to F.

13. The method according to claim 12, characterized in that The N parallel head networks include a first head network, the first head network outputs D-type output results, the multiple types of output results include the D-type output results, and D is a positive integer; After the parameters in the neural network are updated using the one loss value, the method further includes: Input G training data in the training set into the updated neural network respectively to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the D categories of output results to obtain D accumulated loss values ​​corresponding to the D categories of output results respectively; Weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; The first loss value is used to update parameters in the first head network.

14. The method according to claim 12, characterized in that The N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; After the parameters in the neural network are updated using the one loss value, the method further includes: Input G training data in the training set into the updated neural network respectively to obtain the P types of output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the P categories of output results to obtain P accumulated loss values ​​corresponding to the P categories of output results; Weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; The parameters in the L head networks are updated using the second loss value.

15. The method according to claim 10, characterized in that The MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; Among them, the self-attention network is used to extract features from the input data, the mutual attention network is used to fuse the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

16. The method according to claim 10, characterized in that The autonomous driving task includes a prediction task, a decision task and a planning task; The output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the vehicle decision, and the output result of the planning task includes the planned road of the vehicle, the planned trajectory of the vehicle and the planned trajectory distribution of the vehicle.

17. A neural network training device, characterized in that: The neural network includes an expert mixture MOE network and N parallel head networks, the MOE network includes M expert networks, M and N are positive integers; the device includes: An encoding unit is used to extract and fuse features of each piece of training data in a batch of training data through each expert network in the M expert networks during each training process to obtain N types of feature data; and to weight the feature data of the same type output by the M expert networks to obtain weighted N types of feature data; wherein the weighted N types of feature data correspond one-to-one to the N parallel head networks, and each head network in the N parallel head networks corresponds to an autonomous driving task, and the autonomous driving tasks include prediction tasks, decision tasks, and planning tasks; A decoding unit, used to input the weighted N types of feature data into the corresponding head network respectively, to obtain H types of output results corresponding to each training data, wherein the H types of output results include K types of output results, where K is a positive integer less than or equal to H; An updating unit is used to accumulate the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained by the one batch of training data to obtain K accumulated loss values ​​corresponding to the K types of output results respectively; and to weight the K accumulated loss values ​​to obtain a loss value corresponding to each training process; and to use the one loss value to update the parameters in the neural network.

18. The device according to claim 17, characterized in that The K types of output results include first type of output results, the K types of output results obtained from the one batch of training data include F first type of output results, and the F first type of output results all correspond to a first autonomous driving task in the autonomous driving tasks; The F first-category output results include E first-category output results, and the training data used to calculate the E first-category output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first-category output result is obtained by adding the E loss values ​​corresponding to the E first-category output results, respectively, and E is a positive integer less than or equal to F.

19. The device according to claim 17 or 18, characterized in that The N parallel head networks include a first head network, the first head network outputs a D-type output result, the H-type output result includes the D-type output result, and D is a positive integer; after the parameters in the neural network are updated by using the one loss value, the encoding unit and the decoding unit are further used to: Processing G training data in a training set by the updated neural network to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; The updating unit is further used to accumulate the loss values ​​of part or all of the output results in each category of the D types of output results to obtain D accumulated loss values ​​corresponding to the D types of output results respectively; And weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; and using the first loss value to update the parameters in the first head network.

20. The device according to claim 18, characterized in that The N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after the parameters in the neural network are updated by using the one loss value, the encoding unit and the decoding unit are further used to: Processing G training data in a training set by the updated neural network to obtain the P types of output results output by the L head networks, wherein the training set includes the one batch of training data, and G is a positive integer; The updating unit is further used to accumulate the loss values ​​of part or all of the output results in each category of the P categories of output results to obtain P accumulated loss values ​​corresponding to the P categories of output results respectively; And weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; and using the second loss value to update the parameters in the L head networks.

21. The device according to claim 20, characterized in that The H-type output results include the predicted trajectory of the obstacle, the predicted trajectory distribution of the obstacle, the ego vehicle decision, the planned path of the ego vehicle, the planned trajectory of the ego vehicle, and the planned trajectory distribution of the ego vehicle; The predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle correspond to the prediction task, the ego vehicle decision corresponds to the decision task, and the planned road of the ego vehicle, the planned trajectory of the ego vehicle and the planned trajectory distribution of the ego vehicle correspond to the planning task.

22. An automatic driving device, characterized in that: The device comprises: An acquisition unit, used to acquire input data, wherein the input data includes vehicle data, obstacle data, road topology data and navigation data; A processing unit, configured to perform feature extraction and feature fusion on the input data using a mixture of experts MOE network in a neural network to obtain N types of feature data, wherein the neural network further includes N parallel head networks, where N is a positive integer; and input each type of the feature data into a corresponding head network among the N parallel head networks to perform an autonomous driving task corresponding to the head network; A control unit is used to control the vehicle based on the output results of each of the autonomous driving tasks.

23. The device according to claim 22, characterized in that During each training process of the neural network, the processing unit is specifically used to: Each expert network in the MOE network performs feature extraction and feature fusion on each piece of training data in a batch of training data to obtain the N types of feature data; and the feature data of the same type output by the MOE network are weighted by the neural network to obtain the weighted N types of feature data; and the weighted N types of feature data are respectively input into the corresponding head network to obtain H types of output results corresponding to each piece of training data, and the H types of output results include K types of output results, where K is a positive integer less than or equal to H; Among them, the updating of the neural network parameters is completed according to a loss value corresponding to each training process, and the loss value is obtained by weighting K accumulated loss values, and the K accumulated loss values ​​correspond to the K types of output results one by one, and the K accumulated loss values ​​are obtained by accumulating the loss values ​​of part or all of the output results of each type of output results in the K types of output results obtained from the batch of training data.

24. The device according to claim 23, characterized in that The K types of output results include first type of output results, the K types of output results obtained from one batch of training data include F first type of output results, and the F first type of output results all correspond to a first autonomous driving task in the autonomous driving tasks; The F first-category output results include E first-category output results, and the training data used to calculate the E first-category output results all carry labels representing the first autonomous driving task, and the accumulated loss value corresponding to the first-category output result is obtained by adding the E loss values ​​corresponding to the E first-category output results, respectively, and E is a positive integer less than or equal to F.

25. The device according to claim 24, characterized in that The N parallel head networks include a first head network, the first head network outputs D-type output results, the multiple types of output results include the D-type output results, and D is a positive integer; after the parameters in the neural network are updated by using the one loss value, the processing unit is further used to: Using the updated neural network to process G training data in a training set to obtain the D-type output results output by the first head network, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the D categories of output results to obtain D accumulated loss values ​​corresponding to the D categories of output results respectively; Weighting the D accumulated loss values ​​to obtain a first loss value corresponding to the first head network in each training process; The first loss value is used to update parameters in the first head network.

26. The device according to claim 24, characterized in that The N parallel head networks include L head networks, the L head networks correspond to the first autonomous driving task, the L head networks output P-type output results, the H-type output results include the P-type output results, and L and P are positive integers greater than or equal to 2; after the parameters in the neural network are updated by using the one loss value, the processing unit is further used to: Using the updated neural network to process G training data in a training set to obtain the P types of output results output by the L head networks, the training set includes the one batch of training data, and G is a positive integer; Accumulate the loss values ​​of part or all of the output results in each category of the P categories of output results to obtain P accumulated loss values ​​corresponding to the P categories of output results; Weighting the P accumulated loss values ​​to obtain a second loss value corresponding to the L head networks in each training process; The parameters in the L head networks are updated using the second loss value.

27. The device according to claim 26, characterized in that The MOE network includes multiple expert networks, each of the multiple expert networks includes a self-attention network and a mutual attention network; Among them, the self-attention network is used to extract features from the input data, the mutual attention network is used to fuse the features extracted by the self-attention network, and the N types of feature data are generated through the output results of the mutual attention network.

28. The device according to claim 27, characterized in that The autonomous driving task includes a prediction task, a decision task and a planning task; The output result of the prediction task includes the predicted trajectory of the obstacle and the predicted trajectory distribution of the obstacle, the output result of the decision task includes the vehicle decision, and the output result of the planning task includes the planned road of the vehicle, the planned trajectory of the vehicle and the planned trajectory distribution of the vehicle.

29. A chip system, characterized in that: The chip system includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected by lines. Instructions are stored in the at least one memory. When the instructions are executed by the processor, any method described in claims 5-9 is implemented.

30. A chip system, characterized in that: The chip system includes at least one processor, a memory and an interface circuit. The memory, the interface circuit and the at least one processor are interconnected by lines. Instructions are stored in the at least one memory. When the instructions are executed by the processor, any method described in claims 10-16 is implemented.

31. A server, characterized in that: The server comprises the chip system described in claim 29 or claim 30.

32. A vehicle, characterized in that: The vehicle comprises the chip system of claim 30.

33. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method described in any one of claims 5 to 16 is implemented.

34. A computer program product, characterized in that The computer program product comprises instructions, and when the instructions are executed, the method according to any one of claims 5 to 16 is implemented.

Images