A multi-task data processing method and a multi-task model training method

Abstract

The embodiment of the present invention discloses a multi-task data processing method and a multi-task model training method, which obtains at least one type of sensor data, inputs each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, obtains the initial feature information corresponding to each sensor data, inputs each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtains at least one task processing result. The architecture of the multi-task model of this embodiment can perform multi-tasks suitable for single sensor data, and can also perform multi-tasks suitable for the fusion of multiple sensor data, and decouples each network module in the multi-task model, improves the scalability and compatibility of the multi-task model, and reduces code redundancy.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a multi-task data processing method and a multi-task model training method. Background Art

[0002] With the development of technology, multi-task learning technology is being used more and more widely. For example, in the perception task in the field of autonomous driving, the vehicle will rely on different sensors to predict the dynamic and static objects around it. Among them, each type of sensor may correspond to at least one model. For example, when the perception is difficult, there is a lot of additional input information, or the accuracy requirement is high, multiple models may be used to process the data collected by the same sensor to achieve better results. However, model splitting will result in larger memory usage and longer system running time, so a certain trade-off needs to be made between accuracy and speed.

[0003] Moreover, in the case where each of the above sensors has an independent model, it is obvious that the independent model and independent training method can only learn relevant content from a single sensor data for prediction, but in fact, different sensor data have certain complementarity for some tasks. Therefore, in some technologies, a multi-sensor fusion model is used to improve the task execution results based on the complementarity of data. But similarly, some tasks only require single sensor data, and the use of a multi-sensor fusion model may cause certain interference. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a multi-task data processing method and a multi-task model training method, so that the architecture of the multi-task model of this embodiment can execute multi-tasks suitable for single sensor data, and can also execute multi-tasks suitable for the fusion of multiple sensor data, and decouple the network modules in the multi-task model, thereby improving the scalability and compatibility of the multi-task model and reducing code redundancy.

[0005] In a first aspect, an embodiment of the present invention provides a multi-task data processing method, the method comprising:

[0006] acquiring at least one type of sensor data;

[0007] Inputting each of the sensor data into the corresponding sensor backbone network in the pre-trained multi-task model to obtain initial feature information corresponding to each of the sensor data;

[0008] Each of the initial feature information is input into a corresponding sensor head network and / or a fusion branch network in the multi-task model for processing to obtain at least one task processing result.

[0009] In a second aspect, an embodiment of the present invention provides a method for training a multi-task model, the method comprising:

[0010] Performing data sampling on at least one data set based on a predetermined data sampling architecture to obtain a plurality of training data batches, wherein the training data batches include at least one training sample, and the training sample includes sensor data, a data label, and a target head network identifier;

[0011] Based on the target head network identifier, the corresponding training samples are input into the multi-task model for processing to train the multi-task model.

[0012] In a third aspect, an embodiment of the present invention provides a vehicle, the vehicle comprising:

[0013] Vehicle body;

[0014] at least one type of sensor configured to collect sensor data, the sensor data including environmental data surrounding the vehicle;

[0015] The controller is configured to execute the method described in the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention.

[0016] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention is implemented.

[0018] In a sixth aspect, an embodiment of the present invention provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect of the embodiment of the present invention and / or the second aspect of the embodiment of the present invention.

[0019] The embodiment of the present invention obtains at least one type of sensor data, inputs each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, obtains the initial feature information corresponding to each sensor data, inputs each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtains at least one task processing result. The architecture of the multi-task model of this embodiment can perform multi-tasks suitable for single sensor data, and can also perform multi-tasks suitable for the fusion of multiple sensor data, and decouples each network module in the multi-task model, improves the scalability and compatibility of the multi-task model, and reduces code redundancy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0021] Figure 1 It is a schematic diagram of a multi-task model architecture of related technologies;

[0022] Figure 2 is a schematic diagram of another multi-task model architecture of the related technology;

[0023] Figure 3 is a schematic diagram of a multi-task model architecture according to an embodiment of the present invention;

[0024] Figure 4 is a flow chart of a multi-task data processing method according to an embodiment of the present invention;

[0025] Figure 5 is a flowchart of a training method for a multi-task model according to an embodiment of the present invention;

[0026] Figure 6 It is a schematic diagram of the data sampling process in the related art;

[0027] Figure 7 is a flow chart of a data sampling method according to an embodiment of the present invention;

[0028] Figure 8 is a schematic diagram comparing the data sampling process of an embodiment of the present invention with the general data sampling process of the related art;

[0029] Fig. 9 is a schematic diagram of a framework description of a data set category class in an embodiment of the present invention;

[0030] Fig.10 is a schematic diagram of a framework description of a sampler class according to an embodiment of the present invention;

[0031] Fig.11 is a flow chart of a multi-task model parameter adjustment method according to an embodiment of the present invention;

[0032] Fig.12 is a flow chart of another multi-task model parameter adjustment method according to an embodiment of the present invention;

[0033] Fig.13 is another schematic diagram of a multi-task model architecture diagram according to an embodiment of the present invention;

[0034] Fig.14 is a schematic diagram of a multi-task data processing device according to an embodiment of the present invention;

[0035] Fig.15 is a schematic diagram of a training device for a multi-task model according to an embodiment of the present invention;

[0036] Fig.16 is a schematic diagram of a vehicle according to an embodiment of the present invention;

[0037] Fig.17 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0039] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0040] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0041] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0042] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0043] In the following description of the embodiments of the present invention, the use of multiple sensors for perception tasks in the field of autonomous driving is mainly taken as an example. It should be understood that the embodiments of the present invention are not limited to this. The multi-task data processing method of the embodiments of the present invention can be applied to any multi-task processing that uses various types of sensor data for perception. The application in various other scenarios (such as various types of picking and delivery robots, data processing of non-autonomous driving processes, etc.) is not described one by one here.

[0044] Furthermore, the sensors of this embodiment may include camera sensors, radar sensors (lidar, millimeter wave radar, etc.), and other environmental perception sensors, such as temperature and humidity sensors, inertial measurement sensors (velocity acceleration sensors, gyroscopes, etc.), light sensors, etc. This embodiment does not limit the type of sensors.

[0045] Different sensors have certain advantages and disadvantages for many types of tasks, and they are also complementary to each other in certain tasks. Taking the most common lidar sensors and camera sensors as examples, the data collected by the lidar is a point cloud file, including multiple point cloud points. The information of each point cloud point is [x, y, z, intensity], where x, y, z are the coordinates of the corresponding point cloud point (for example, the coordinates in the Cartesian coordinate system centered on the vehicle), and intensity is the reflection intensity of the lidar. The value is related to the material, surface roughness, incident angle, distance, etc. of the scanned object. The advantage of lidar is accurate positioning, but because the information of lidar is sparse and the target texture information cannot be obtained, and the farther the object is, the sparser the corresponding point cloud points are, so the accuracy may not be high in specific target detection tasks. Camera sensors are different. They can obtain detailed object texture information and at a higher frequency, but their positioning accuracy is lower than that of lidar. Therefore, in some tasks, the complementarity of LiDAR and camera can obtain more accurate results, but in tasks where each sensor has outstanding advantages and other sensors have disadvantages in this regard, if fusion data is used, it may have adverse interference on the task results. In this regard, related technologies usually use independent models to handle data processing solutions with different advantages, which causes code redundancy and requires more computing resources.

[0046] Figure 1 It is a schematic diagram of a multi-task model architecture of related technologies. Figure 1 As shown, Figure 1 The multi-task model architecture in is a multi-task model architecture corresponding to a single type of sensor a, and the architecture includes a sensor backbone network 10 corresponding to the sensor a and head networks head11-head1n corresponding to multiple tasks, where n is an integer greater than or equal to 1. The value of n corresponds to the number of tasks.

[0047] Among them, the sensor backbone network 10 performs feature processing on the input sensor a data to obtain initial feature information t, and inputs the initial feature information t into the head networks head11-head1n corresponding to each task for processing to obtain the output result of the corresponding task.

[0048] Figure 2 Schematic diagram of the multi-task model architecture of an embodiment of the present invention. Figure 2 As shown, Figure 2 The multi-task model architecture in is a multi-task model architecture of multi-sensor fusion, which includes a sensor backbone network 201 corresponding to sensor a, a sensor backbone network 202 corresponding to sensor b, a fusion network 200, and head networks head21-head2m corresponding to multiple tasks, where m is an integer greater than or equal to 1. The value of m corresponds to the number of tasks.

[0049] Among them, the sensor backbone network 201 performs feature processing on the input sensor a data to obtain initial feature information ta, the sensor backbone network 202 performs feature processing on the input sensor b data to obtain initial feature information tb, and the fusion network 200 performs feature fusion on the initial feature information ta and the initial feature information tb, and inputs the fused features into the head networks head21-head1m corresponding to each task for processing to obtain the output results of the corresponding tasks.

[0050] It can be seen that in the multi-task model of the related art, the multi-task model of a single type of sensor can only process the task of applying the sensor data, and the multi-task model of multi-sensor fusion can only process the task of applying multiple sensors. If the multi-task model of a single type of sensor and the multi-task model of multi-sensor fusion are deployed in the system, there will be code redundancy (for example, Figure 1 and Figure 2 The models in the above diagram all require the backbone network corresponding to the deployed sensor a), and the training process also needs to be trained separately, which leads to the problem that the training process is complicated and requires large computing resources.

[0051] Therefore, the embodiments of the present invention provide a multi-task data processing method, device, vehicle and electronic device, so that through the architecture of the multi-task model of this embodiment, multi-tasks suitable for single sensor data can be executed, and multi-tasks suitable for the fusion of multiple sensor data can be executed, and the network modules in the multi-task model are decoupled, thereby improving the scalability and compatibility of the multi-task model and reducing code redundancy.

[0052] Figure 3Schematic diagram of a multi-task model architecture of an embodiment of the present invention. The multi-task model architecture of the embodiment of the present invention includes a sensor backbone network, a sensor head network and a fusion branch network corresponding to each type of sensor.

[0053] like Figure 3 As shown, in the multi-task model architecture of this embodiment, the sensor backbone network includes a sensor backbone network 201 corresponding to sensor a1, a sensor backbone network 202 corresponding to sensor a2, ..., and a sensor backbone network 20x corresponding to sensor ax, where x is an integer greater than or equal to 1.

[0054] The sensor head network includes a sensor head network a1_head corresponding to sensor a1 and head networks a1_head1, a1_head2 and a1_head11 for multiple tasks corresponding to sensor a1, a sensor head network a2_head corresponding to sensor a2 and head networks a2_head1 and a2_head2 for multiple tasks corresponding to sensor a2, ..., a sensor head network ax_head corresponding to sensor ax and head networks ax_head1, ax_head2 and ax_head21 for multiple tasks corresponding to sensor ax.

[0055] The fusion branch network includes a fusion network 200 and a fusion head network fusion_head and corresponding multi-task head networks f_head1 and f_head2.

[0056] It should be understood that the tree structure of the head network of the above multi-task is only exemplary, and the structure of the head network of the multi-task in a specific application scenario can be designed and adjusted according to the relationship between the tasks, and this embodiment is not limited to this.

[0057] Furthermore, in this embodiment, the sensor a1 data, the sensor a2 data, and the sensor ax data are input into the multi-task model. The sensor backbone network 201 performs feature processing on the sensor a1 data to obtain an initial feature map ta1. The sensor backbone network 202 performs feature processing on the sensor a2 data to obtain an initial feature map ta2. The sensor backbone network 20x performs feature processing on the sensor ax data to obtain an initial feature map tax.

[0058] Further, the sensor head network a1_head processes the initial feature map ta1, and transmits the processing results to the subsequent task head networks a1_head1 and a1_head2. The task head network a1_head1 transmits the task processing results to the subsequent task head network a1_head11 to obtain the corresponding task processing results. For example, the sensor head network a1_head can be used to determine the corresponding original output results, and has two branch task head networks a1_head1 and a1_head2. Among them, a1_head1 can be used to perform target detection tasks, and the task head network a1_head2 can be used to perform semantic segmentation tasks. The task head network a1_head1 further has a branch task head network a1_head11, which can be used to detect specific target objects, such as pedestrian detection, etc., and this embodiment is not limited to this.

[0059] Similarly, the sensor head network a2_head processes the initial feature map ta2 and transmits the processing results to the subsequent task head networks a2_head1 and a2_head2 to obtain the corresponding task processing results. The sensor head network ax_head processes the initial feature map tax and transmits the processing results to the subsequent task head networks ax_head1 and ax_head2. The task head network ax_head2 transmits the task processing results to the subsequent task head network ax_head21 to obtain the corresponding task processing results.

[0060] Furthermore, the fusion network 200 performs feature fusion processing on the received initial feature map ta1-tax to obtain a fused feature T. The fusion head network fusion_head processes the fused feature T and transmits the processing result to the subsequent task head networks f_head1 and f_head2 for processing to obtain the corresponding task processing result.

[0061] In another possible implementation, the input data of this embodiment may also carry a target head network identifier, so that only the corresponding network module can be executed according to the target head network to further reduce the amount of calculation. For example, if only sensor a1 data is input in the multi-task model, and the target head network identifier carried by the sensor a1 data is a1_head2, then only the sensor backbone network 201, the sensor head network a1_head and a1_head2 need to be started and executed to obtain the required task processing results, which obviously improves the data processing efficiency and saves computing resources.

[0062] Therefore, in the multi-task model architecture of the embodiment of the present invention, it is possible to implement related tasks based on the advantages of single-type sensor data, and it is also possible to implement related tasks based on the data fusion advantages of complementary multi-sensor data, and to decouple the network modules in the multi-task model, thereby improving the scalability and compatibility of the multi-task model. At the same time, the fusion branch network and each sensor branch network share each sensor backbone network, reducing code redundancy and computing resources, and further reducing system latency.

[0063] Figure 4 is a flowchart of a multi-task data processing method according to an embodiment of the present invention. The multi-task data processing method according to this embodiment can be based on Figure 3 The multi-task model architecture shown in Figure 4 As shown, the multi-task data processing method of this embodiment includes the following steps:

[0064] Step S110, obtaining at least one type of sensor data. Taking the field of autonomous driving as an example, the vehicle is equipped with various sensors for sensing environmental information, such as camera sensors, radar sensors, temperature and humidity sensors, light sensors, inertial measurement sensors, etc. This embodiment can obtain sensor data collected by various sensors to identify environmental information through a multi-task model, and then control the vehicle to perform corresponding operations, such as turning, avoiding, parking, etc.

[0065] Step S120 , inputting each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model to obtain initial feature information corresponding to each sensor data.

[0066] In this embodiment, based on the tasks corresponding to each type of sensor data, each type of sensor has a corresponding sensor backbone network. For example, a camera sensor has a corresponding backbone network for processing images. It should be understood that the backbone network of a camera sensor can be any network structure for extracting image features, such as a backbone network built on the basis of network structures such as CNN, VGG, and ResNet, etc., and this embodiment does not limit this. A radar sensor has a corresponding backbone network for processing point cloud data. It should be understood that the backbone network of a radar sensor can be a task network structure for extracting point cloud features, such as a backbone network built on the basis of network structures such as pointnet and DGCNN, etc., and this embodiment does not limit this.

[0067] Furthermore, the sensor backbone network is used to extract features from the received sensor data to obtain initial feature information. For example, the sensor backbone network corresponding to the camera sensor extracts features from the received image to obtain initial feature information of the image, and the sensor backbone network corresponding to the radar sensor extracts features from the received point cloud data to obtain initial feature information of the point cloud.

[0068] Step S130, input each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing to obtain at least one task processing result. Optionally, the fusion branch network includes a fusion network and a fusion head network. The fusion network is used to perform feature fusion on multiple initial feature information corresponding to multiple sensors to obtain fused feature information, and the fusion head network is used to process the fused feature information to obtain the task processing result of the corresponding at least one task.

[0069] In this embodiment, each initial feature information can be input into the corresponding sensor head network for processing to obtain at least one task processing result of each type of independent sensor task. At the same time, each initial feature information can be input into the fusion branch network for feature fusion to obtain at least one task processing result of the multi-sensor fusion task.

[0070] Taking sensors including camera sensors and radar sensors as an example, the acquired initial feature information of the image is input into the head network corresponding to the camera sensor for processing to obtain the task processing result of the corresponding at least one task, the acquired initial feature information of the point cloud is input into the head network corresponding to the radar sensor for processing to obtain the task processing result of the corresponding at least one task, the initial feature information of the image and the initial feature information of the point cloud are input into the fusion network in the fusion branch network for processing to obtain the fusion feature information, the fusion feature information is input into the fusion head network in the fusion branch network for processing to obtain the task processing result of the corresponding at least one task.

[0071] Optionally, the fusion network can convert the two-dimensional image initial feature information into three-dimensional image feature information (for example, by using inverse perspective mapping (IPM) and other methods), and perform feature mapping fusion on the converted three-dimensional image feature information and the three-dimensional point cloud initial feature information to obtain feature fusion information, or convert the three-dimensional point cloud initial feature information into two-dimensional point cloud feature information, and perform feature mapping fusion on the two-dimensional point cloud feature information and the two-dimensional image initial feature information to obtain fused feature information. Further optionally, the image initial feature information and the point cloud initial feature information can be fused in a bird's eye view (BEV) space.

[0072] It should be understood that the feature information fusion method of this embodiment can adopt neural network, clustering algorithm, template method and the like. This embodiment does not limit the method of fusing feature information of various types of sensor data, and it only needs to realize the fusion of feature information of different sensors.

[0073] Furthermore, the various initial feature information received by the fusion network should be time-synchronized in advance to improve the accuracy of the task processing results. That is, the sensor data collected by various sensors are generated at the same time, representing the environmental information in the same environment at the same time.

[0074] In an optional implementation, various types of head networks (sensor head network, fusion head network) can be lightweight networks, which may include one or more of a fully connected layer, a convolutional layer, a normalization layer, a Dropout layer, a regressor, a classifier, NMS (Non-Maximum Suppression), a decoder (deconvolution layer or upsampling layer), RNN, and LSTM, which can be determined based on the task to be performed. For example, if the task is an image classification task, the head network may include a fully connected layer, a convolutional layer, and / or a normalization layer. If the task is a target detection task, the head network may include a regressor, a classifier, or NMS. If it is a semantic segmentation task, the head network may include an upsampling layer (Upsampling) and a classifier (Classifier), etc.

[0075] It should be understood that the selection of the network structure of the head network corresponding to different tasks in this embodiment can be adjusted according to factors such as task requirements (such as target detection, semantic segmentation, etc.), processing data characteristics (such as image data, point cloud data density, distribution and noise, etc.) and performance requirements (model calculation efficiency, model size, loss function type, etc.) to better adapt to the task processing process, which is conducive to further improving the task processing efficiency.

[0076] In an optional implementation, each network model of the multi-task model of this embodiment is in a startup state during the task processing process. That is, this embodiment inputs each sensor data into the corresponding sensor backbone network to obtain the corresponding initial feature information, and inputs each initial feature information into the corresponding sensor head network for processing to obtain at least one task processing result of each type of sensor independent task, and at the same time inputs each initial feature information into the fusion branch network for feature fusion, and obtains at least one task processing result of the multi-sensor fusion task.

[0077] In another optional implementation, the present embodiment can also start the network module in the multi-task model according to the current task target by setting the current task target. Optionally, the corresponding current task target can be determined by adding the corresponding target header network identifier to the input sensor data. For example, Figure 3As shown, if the input sensor data includes the a1_head identifier, it indicates that the current target task includes the processing task of the single sensor a1, and if the input sensor data includes the fusion_head identifier, it indicates that the current target task includes the fusion processing task. Furthermore, the target head network identifier can also be the head network identifier corresponding to the leaf node, such as a1_head11, etc., and this embodiment does not limit this.

[0078] Furthermore, in response to the current task target including a fusion processing task, this embodiment inputs multiple initial feature information required for the fusion processing task into the feature fusion network of the fusion branch network for feature fusion, obtains fusion feature information, and inputs the fusion feature information into the fusion head network of the fusion branch network for processing to obtain the corresponding task processing result. In response to the current task target including at least one single sensor processing task, this embodiment inputs the initial feature information corresponding to each single sensor processing task into the corresponding sensor head network for processing to obtain the corresponding task processing result.

[0079] Therefore, this embodiment can execute related network modules according to the task target of the current processing task, further saving computing resources and improving task processing efficiency.

[0080] This embodiment obtains at least one type of sensor data, inputs each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, obtains the initial feature information corresponding to each sensor data, inputs each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtains at least one task processing result. The architecture of the multi-task model of this embodiment can perform multi-tasks suitable for single sensor data, and can also perform multi-tasks suitable for the fusion of multiple sensor data, and decouples each network module in the multi-task model, improves the scalability and compatibility of the multi-task model, and reduces code redundancy.

[0081] Figure 5 is a flow chart of a training method for a multi-task model according to an embodiment of the present invention. In an optional implementation, Figure 5 As shown, the training method of the multi-task model of this embodiment includes the following steps:

[0082] Step S210, sampling at least one data set based on a predetermined data sampling architecture to obtain multiple training data batches, wherein the training data batches include at least one type of training samples, and the training samples include sensor data, data labels, and target head network identifiers.

[0083] Optionally, the data set may be a known data set for training in a relevant scenario, or may be a data set formed by relevant scenario data collected before training, and this embodiment does not limit this. For example, in the field of autonomous driving, the image data set may include the KITTI data set, the SYNTHIA data set, the Cityscapes data set, and / or an image data set in an autonomously collected traffic scene. The point cloud data set may include the Waymo open data set, the nuScenes data set, the KITTI data set, the Argoverse data set, and / or a point cloud data set in an autonomously collected traffic scene.

[0084] Among them, the KITTI dataset is currently the world's largest algorithm evaluation dataset for autonomous driving scenarios. The dataset contains image data collected in urban, rural and highway scenes, with up to 15 vehicles and 30 pedestrians in each image, as well as various degrees of occlusion and truncation.

[0085] The SYNTHIA dataset is a collection of photorealistic frames rendered from a virtual city with precise pixel-level semantic annotations for 13 classes, including miscellaneous, sky, buildings, roads, sidewalks, fences, vegetation, poles, cars, signs, pedestrians, cyclists, lane markings, etc.

[0086] The Cityscapes dataset contains 5,000 finely annotated video frames collected from 50 cities, as well as a large number of hand-crafted labeled 3D models that can be used to train autonomous driving systems to detect and track 3D targets.

[0087] Since the input of the multi-task model of this embodiment may include multiple sensor data, it is also necessary to obtain training samples from the data sets corresponding to different sensors during training. When the network model requires multiple data sets, if the same sampling strategy is used, it may cause problems of data category imbalance. Therefore, for different data sets, different data sampling methods can be selected based on their characteristics to solve problems such as data category imbalance.

[0088] Figure 6 is a schematic diagram of the data sampling process in the related art. In the related art, Figure 6As shown, each dataset usually has a completely independent process to output sampled data. For example, for dataset dataset1, in this thread, data sampling is performed through the corresponding collector sampler1 and data loader dataloader1 to obtain training data batch databatch1. For dataset dataset2, in this thread, data sampling is performed through the corresponding collector sampler2 and data loader dataloader2 to obtain training data batch databatch2. For dataset dataset3, in this thread, data sampling is performed through the corresponding collector sampler3 and data loader dataloader3 to obtain training data batch databatch3. As the number of datasets increases and the number of datasets increases, the number of data loaders dataloaders increases, and the number of threads for data reading increases. Ultimately, the model training speed may be slow due to too much thread data.

[0089] Therefore, the embodiment of the present invention designs a data sampling architecture to perform data sampling on various data sets. It can adopt specified sampling strategies for different data sets without changing the data sampling architecture of a single data set, thereby achieving high cohesion and low coupling of system code.

[0090] In an optional implementation, the data sampling architecture of this embodiment is defined with a data set category class and a sampler class. The data set category class includes a data set list and a data determination function. The sampler class includes a data set category class, a sampler list, and a relationship between a data set in the data set list and a sampler in the sampler list.

[0091] Figure 7 FIG. 1 is a flow chart of a data sampling method according to an embodiment of the present invention. Figure 7 As shown, by executing the data sampling architecture of the embodiment of the invention, the data sampling method of the embodiment of the invention includes the following steps:

[0092] Step S211, determining the target data set and the target sampler corresponding to the target data set according to the corresponding sampling instruction. Optionally, the sampling instruction may represent a data sampling start command, or may be determined based on a corresponding sampling strategy, which is not limited in this embodiment.

[0093] Step S212: sampling the target data set based on the target sampler to obtain corresponding training data batches.

[0094] Step S213, updating the meta-attributes of the training data batch, wherein the meta-attributes include a target header network identification field.

[0095] The data sampling process of the embodiment of the present invention is described in detail below based on an exemplary data sampling architecture.

[0096] Figure 8 FIG. 2 is a schematic diagram comparing the data sampling process of the embodiment of the present invention with the general data sampling process of the related art. Figure 8 As shown, the general data sampling process 81 of the related art is a data sampling process of a data sampling architecture of a single data set, wherein data sampling is performed on the data set dataset through a sampler sampler and a data loader dataloader to obtain a training data batch databatch.

[0097] In the data sampling process 82 of this embodiment, the corresponding data sampling architecture defines two classes: a data set category class MSMTDataset and a sampler class MSMTSampler, so as to implement the use of specified sampling strategies for different data sets.

[0098] Furthermore, the dataset category class MSMTDataset includes a dataset list and a data determination function. The data determination function can determine data sampling from a corresponding position in a corresponding dataset in the dataset list by inputting an index parameter. Optionally, the index parameter can include a corresponding dataset identifier and a data index number in the dataset.

[0099] Fig. 9 Schematic diagram of the framework description of the data set category class in the embodiment of the present invention. This embodiment takes the data set list including 3 data sets as an example, and it should be understood that the data sets in the data set list can be expanded according to data requirements.

[0100] like Fig. 9 As shown, the dataset category class MSMTDataset includes a dataset list dataset_list. Each element in the dataset list dataset_list is an independent dataset, for example, datasets sub_dataset1, sub_dataset2, and sub_dataset3. The dataset category class MSMTDataset also defines a data determination function __getitem__(idx). Among them, idx is a list file with a length of 2, representing the index parameter. Among them, the first element in the index parameter idx is sub_dataset_idx, which can represent the dataset identifier, that is, determine which dataset to take the data from, and the second element data_sample_idx can represent the data index number in the corresponding dataset, that is, determine from which position in the dataset to read the data. Therefore, this embodiment can realize the function of selecting multiple datasets by defining the dataset category class MSMTDataset.

[0101] Further optionally, the meta attribute meta of the output result result after the data set category class MSMTDataset of this embodiment is executed includes a target head network identification field data_to_head. The target head network identification field data_to_head can be a list attribute, and each element in the corresponding list is a head network identification, and the head network represented by it can be a parent node in the head network tree structure, or a leaf node. Figure 3 Taking the tree-structured header network shown as an example, the target header network identification field data_to_head may include one or more of the parent nodes a1_head, a2_head, ax_head, fusion_head, the child nodes a1_head1, ax_head2, and the leaf nodes a1_head11, a1_head2, a2_head1, a2_head2, ax_head1, ax_head21, f_head1, and f_head2.

[0102] The output result result after the execution of the dataset category class MSMTDataset of this embodiment carries the head network corresponding to the sampled training data batch, that is, its corresponding model output result, so that the parameters of the corresponding model network module can be adjusted through the model output result and the data labels of the training samples in the training data batch. Further, this can enable the dataset carrying the corresponding data label information to train the network modules corresponding to some tasks in the model in a targeted manner, avoiding the network modules corresponding to the tasks that are not suitable for using the dataset from being affected by the related training process. For example, if the dataset dataset1 carries the data label of the target object o1, but does not carry the data label of the target object o2, it is obvious that the dataset dataset1 is suitable for training the network module corresponding to the task of detecting the target object o1, but not suitable for training the network module corresponding to the task of detecting the target object o2. Therefore, this embodiment can further improve the efficiency of multi-task model training and the accuracy of the multi-task model after training.

[0103] Further optionally, the sampler class MSMTSampler of this embodiment is used to perform sampling operations on the data set category class MSMTDataset. The sampler class MSMTSampler includes the data set category class MSMTDataset, a sampler list, and the relationship between the data sets in the data set list and the samplers in the sampler list.

[0104] Fig.10 Schematic diagram of the framework description of the sampler class in the embodiment of the present invention. Fig.10As shown, the dataset category class MSMTDataset is an attribute in the sampler class MSMTSampler. In the sampler class MSMTSampler, an independent sampler is set for each dataset in the dataset list in the dataset category class MSMTDataset, so as to achieve data sampling of different datasets using an adaptive sampler, thereby maintaining the balance of data categories.

[0105] Furthermore, if Fig.10 As shown, taking the dataset list including three datasets sub_dataset1, sub_dataset2 and sub_dataset3 as an example, the sampler list sampler_list has corresponding samplers sub_sampler1, sub_sampler2 and sub_sampler3. Each sampler can be an iterator, and the returned result represents the subscript of the data (representing its position in the corresponding dataset, that is, the data index number).

[0106] Further optionally, this embodiment can adopt a sampling aggregation strategy (Aggregator Policy) so that each time a data subscript can be selected from a certain data set, combined into a list (sub_dataset_idx, data_sampler_idx) and put into a queue, which is also the input parameter of the data determination function __getitem__(idx) of the data set category class MSMTDataset. Thus, the required training data batches can be sampled from the corresponding data set.

[0107] In an optional implementation, the sampling strategies of the samplers in the sampler list of this embodiment may include weighted_sampler, uniform_sampler, and uniform_sampler (one batch). Among them, the sampling strategy weighted_sampler specifically selects a sub_sampler_i (sampler i) index for each training data batch (batch) of data according to probability and stores it. The sampling strategy uniform_sampler specifically selects a sub_sampler_i (sampler i) index for each training data batch (batch) of data according to the probability of uniform sampling and stores it. The sampling strategy uniform_sampler (one batch) specifically selects a sub_sampler_i (sampler i) index for each training data batch (batch) of data according to the same probability and stores it.

[0108] Under different sampling strategies, the data source (i.e., data set) corresponding to each training data batch may be different, and the data iteration and sampling frequency may also be controlled by the corresponding sampling strategy based on the actual application situation. It should be understood that various sampling strategies are determined based on the corresponding data set, that is, whichever sampling strategy is suitable for the selected data set, then the sampling strategy can be added to the sampler list, and this embodiment does not limit the specific sampling strategy.

[0109] Furthermore, in this embodiment, since the sampling aggregation policy (Aggregator Policy) module is an independent module, it has strong scalability and can adjust or add new sampling policies based on the sampling requirements of specific application scenarios.

[0110] It should be understood that Fig. 9 and Fig.10 The architecture description of the dataset category class MSMTDataset and the sampler class MSMTSampler is merely exemplary and can be adjusted based on the needs of actual applications. This embodiment is not limited to this.

[0111] This embodiment uses the newly defined dataset category class MSMTDataset and sampler class MSMTSampler to enable the training samples required for multi-task model training to read any amount of data, and each dataset can specify a separate sampling strategy to maintain the balance of sample data categories. At the same time, the entire data sampling structure process has not changed, achieving high cohesion and low coupling of the system code.

[0112] Step S220: Based on the target head network identifier, the corresponding training sample is input into the multi-task model for processing to train the multi-task model.

[0113] In an optional implementation, this embodiment can train the network module that the corresponding task passes through based on the target header network identifier. Figure 3 As shown, assuming that the target head network identifier of the training sample includes a1_head2, the training sample is input into the multi-task model for processing, the output of the target head network a1_head2 is obtained, and the corresponding loss is calculated based on the difference between the output of the target head network a1_head2 and the data label of the training sample, and the parameters of the sensor backbone network 201, the sensor head network a1_head and a1_head2 and other network modules are adjusted based on the loss. In this way, the multi-task model can be trained in a targeted manner based on the advantages of each data set for each type of task, further improving the training efficiency of the multi-task model and the task processing accuracy of the trained multi-task model.

[0114] In an optional implementation, this embodiment first trains each sensor branch network (ie, each sensor trunk network and sensor head network) in the multi-task model, and then trains the fusion branch network based on the trained sensor trunk network and training samples.

[0115] Fig.11 FIG. 1 is a flow chart of a multi-task model parameter adjustment method according to an embodiment of the present invention. Fig.11 As shown, the multi-task model parameter adjustment method of the embodiment of the present invention includes the following steps:

[0116] Step S221A: training the corresponding sensor branch network based on the training samples of the target head network identified as the sensor head network, wherein the sensor branch network includes the sensor trunk network and the sensor head network.

[0117] In an optional implementation, the training data batches whose target head network identifier is the sensor head network can be first screened out, and then the corresponding sensor branch network can be trained based on the training samples in these training data batches. In another optional implementation, the parameters of the network module passed by the corresponding task can also be adjusted by identifying the target head network identifier in the training sample through a multi-task model, and the output result of the head network corresponding to the target head network identifier and the difference in the data label of the corresponding training sample, that is, the result does not affect the parameters of the network model passed by the task corresponding to the non-target head network.

[0118] Further optionally, this embodiment can adjust the parameters of each network module in the sensor branch network based on a predetermined loss function. It should be understood that this embodiment does not limit the type of loss function, which can characterize the difference between the output result of the head network and the data label of the training sample.

[0119] Step S222A, maintaining the parameters of the trained sensor backbone network, and training the fusion branch network based on the training samples of the target head network identified as the fusion head network.

[0120] That is to say, in this embodiment, after the training of each sensor branch network is completed, the parameters of each network module in the sensor branch network are maintained, and the trained sensor backbone network is used to train the fusion branch network based on the training samples identified as the fusion head network by the target head network.

[0121] That is to say, the corresponding training samples are input into the corresponding sensor backbone network in the multi-task model for processing to obtain the initial feature information of the samples, and the initial feature information of each sample is input into the fusion branch network for processing to obtain the output of the corresponding target head network. The parameters of the fusion branch network are adjusted based on the difference between the data of the target head network and the data labels of the corresponding training samples to train the fusion branch network in the multi-task model.

[0122] Therefore, this embodiment improves the model training efficiency by first training a separate sensor branch network in the multi-task model, maintaining the parameters of the sensor branch network, and then training the fusion branch network in the multi-task model.

[0123] Fig.12 FIG. 1 is a flow chart of another method for adjusting multi-task model parameters according to an embodiment of the present invention. Fig.11 As shown, the multi-task model parameter adjustment method of the embodiment of the present invention includes the following steps:

[0124] Step S221B, based on the training samples of the target head network identified as the sensor head network, the corresponding sensor branch network is trained. The sensor branch network includes the sensor trunk network and the sensor head network. It should be understood that the training method of the sensor branch network in this step is similar to the above step S221A, and will not be repeated here.

[0125] Step S222B, in response to the completion of training of each sensor branch network, train the fused branch network based on the training samples of the target head network identified as the fused head network. It should be understood that the training method of the fused branch network in this step is similar to the above step S222A, and will not be repeated here.

[0126] Step S223B, fine-tune the parameters of each sensor backbone network based on the difference between the sample task processing results of the fused branch network and the corresponding data labels. In this embodiment, during the fused branch network training process, each sensor backbone network is fine-tuned based on the output of the fused branch network, further improving the accuracy of the trained multi-task model.

[0127] In an optional implementation, this embodiment sets the loss of feature points corresponding to missing data labels in the training sample to a predetermined value. Optionally, the predetermined value is 0. Since a data label in the training sample is missing, the distance between the output result of the head network and the true result cannot be determined. Therefore, if the corresponding loss is fed back to the network, it may cause a certain impact, which may further extend the training process or affect the accuracy of the multi-task model after training. Therefore, this embodiment sets the loss of feature points corresponding to missing data labels in the training sample to a predetermined value to avoid the impact caused by the missing data label.

[0128] The above embodiments are described in detail below by taking the multi-task model of the fusion of camera sensor data (i.e., image data) and radar sensor (i.e., point cloud data) as an example. It should be understood that this embodiment does not limit the type and quantity of sensors, which can be determined based on actual applications and the specific architecture of the multi-task model can be adjusted based on the determined sensors, that is, the sensor backbone network and the corresponding sensor head network can be added or adjusted to achieve different tasks.

[0129] Fig.13 FIG. 1 is another schematic diagram of a multi-task model architecture according to an embodiment of the present invention. Fig.13 As shown, the multi-task model of this embodiment includes a radar branch network. The radar branch network includes a radar trunk network and a tree head network structure corresponding to the radar trunk network. The tree head network structure corresponding to the radar trunk network includes head networks lidar_head, lidar_det_head, lidar_seg_head, pedestrian / cyclist, vehicle, and large vehicle. Among them, the radar trunk network is used to extract features from the received radar point cloud data to obtain the initial feature information of the point cloud, and the head network lidar_head is used to process the received initial feature information of the point cloud to obtain the corresponding original processing results (such as the results of each object detection box, or the semantic segmentation results, etc.), which may include two branches, namely the head network lidar_det_head and lidar_seg_head. The head network lidar_det_head is used to output the 3D target detection branch of the radar, which can be split into multiple sub-branches based on different target objects, such as pedestrian / cyclist, vehicle, large vehicle, etc., and the specific sub-branches can be determined based on the object type of the required target detection. The head network lidar_seg_head is used to output the semantic segmentation results of the radar point cloud, which includes the semantic categories corresponding to each point cloud point.

[0130] Furthermore, the multi-task model of this embodiment includes a camera branch network. The camera branch network includes a camera trunk network and a tree head network structure corresponding to the camera trunk network. The tree head network structure corresponding to the camera trunk network includes head networks camera_head, object_det_head, semantic_det_head, animal_det_head, and traffic_light_panel_det_head. Among them, the camera trunk network is used to extract features from the received image data to obtain the initial feature information of the image, and the head network camera_head is used to process the received initial feature information of the image to obtain the corresponding original processing result, which may include multiple branches, namely, the head networks object_det_head, semantic_det_head, animal_det_head, and traffic_light_panel_det_head. Among them, the object_det_head branch of the head network is used to detect the corresponding 2D target detection branch, the semantic_det_head branch of the head network is used to detect the 2D road semantic segmentation result, the animal_det_head branch of the head network is a separate animal detection branch, and the traffic_light_panel_det_head branch of the head network is used to detect the traffic light panel.

[0131] Furthermore, the multi-task model of this embodiment includes a fusion branch network. The fusion branch network includes a fusion network and a corresponding tree head network structure. Among them, the tree head network structure corresponding to the fusion network includes a head network fusion_head and lidar_det_head(cone, barrier). Among them, the fusion network is used to fuse the received point cloud initial feature information and image initial feature information, and input the fusion feature to the head network fusion_head and lidar_det_head(cone, barrier) to detect various obstacles in the road.

[0132] Optionally, the fusion network can convert the two-dimensional image initial feature information into three-dimensional image feature information (for example, by using inverse perspective mapping (IPM) and other methods), and perform feature mapping fusion on the converted three-dimensional image feature information and the three-dimensional point cloud initial feature information to obtain feature fusion information, or convert the three-dimensional point cloud initial feature information into two-dimensional point cloud feature information, and perform feature mapping fusion on the two-dimensional point cloud feature information and the two-dimensional image initial feature information to obtain fused feature information. Further optionally, the image initial feature information and the point cloud initial feature information can be fused in a bird's eye view (BEV) space.

[0133] It should be understood that the feature information fusion method of this embodiment can adopt neural network, clustering algorithm, template method and the like. This embodiment does not limit the method of fusing feature information of various types of sensor data, and it only needs to realize the fusion of feature information of different sensors.

[0134] Furthermore, the various initial feature information received by the fusion network should be time-synchronized in advance to improve the accuracy of the task processing results. That is, the sensor data collected by various sensors are generated at the same time, representing the environmental information in the same environment at the same time.

[0135] In the training process of the multi-task model corresponding to the radar and camera sensors of this embodiment, each data set is sampled based on the above data sampling method to obtain a training data batch data_batch. The training data batch data_batch includes multiple training samples data1, data2, etc., wherein the data types of the training samples in the same training data batch data_batch can be the same or different.

[0136] Based on the data sampling method of this embodiment, the meta-attribute of each training sample data includes a data_to_head field, that is, a target head network identification field. Therefore, this embodiment can feed back the loss corresponding to the training sample to the corresponding network module based on the data_to_head field corresponding to each training sample (determined based on the output of the training sample in the corresponding target head network and the data label) to adjust the parameters of the network module corresponding to the training sample.

[0137] For example, assuming that the data_to_head field of the training sample data1 includes lidar_seg_head and largevehicle, the parameters of the radar backbone network, the head network lidar_head, lidar_seg_head and large vehicle are adjusted based on the output of the head network lidar_seg_head and large vehicle and the training sample data1. At the same time, if other network modules (such as lidar_det_head, pedestrian / cyclist, vehicle, etc.) are initialized, but it is obvious that they have no corresponding training data, the relevant losses can be fed back as a predetermined value (such as 0), that is, no parameter adjustment is made to these network modules. Optionally, this embodiment can define a zero_flag field and feed back zero_flag based on the data_to_head field. If zero_flag is true, the corresponding loss is 0. It should be understood that this embodiment does not limit the method of how to implement the feedback of the loss of the predetermined value.

[0138] Furthermore, the sensor branch network of this embodiment can implement a single call during the training process. For example, when training the radar branch network, the outputs of the camera branch network and the fusion branch network (such as the outputs of the head networks camera_head and fusion_head) are not initialized, so that when the radar branch network is trained, the parameters of the camera branch network and the fusion branch network remain unchanged. In other optional implementations, the outputs of the camera branch network and the fusion branch network can also be initialized, but the related loss of the feedback is made 0 to achieve parameter retention of the camera branch network and the fusion branch network, and this embodiment does not limit this.

[0139] Therefore, this embodiment can put the perception tasks of the radar and camera of autonomous driving into a multi-task model, so that any perception task using radar and camera can be added to the multi-task model for training and use, thereby improving the scalability of the multi-task model.

[0140] At the same time, the network modules of the multi-task model of this embodiment are decoupled, which improves the convenience of optimizing the training strategies and adjusting the parameters of each network module during the training process.

[0141] In summary, this embodiment can effectively avoid the impact of missing labeled data on network parameter adjustment, and can adopt a suitable sampling strategy for each data set, and can expand the number of data sets. At the same time, this embodiment can achieve the decoupling between the network modules of the multi-task model, improve the convenience of optimizing the training strategies and adjusting the parameters of each network module during the training process. Furthermore, the multi-task model of this embodiment sets the head network corresponding to different network branches (such as the above-mentioned lidar_head, camera_head, fusion_head), which improves the convenience of further data fusion experiments, and can support the training and task processing of a single sensor.

[0142] This embodiment obtains at least one type of sensor data, inputs each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, obtains the initial feature information corresponding to each sensor data, inputs each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtains at least one task processing result. The architecture of the multi-task model of this embodiment can perform multi-tasks suitable for single sensor data, and can also perform multi-tasks suitable for the fusion of multiple sensor data, and decouples each network module in the multi-task model, improves the scalability and compatibility of the multi-task model, and reduces code redundancy.

[0143] Fig.14 Schematic diagram of a multi-task data processing device according to an embodiment of the present invention. Figure 1 As shown, the multi-task data processing device 14 according to the embodiment of the present invention includes a data acquisition unit 141 , a first processing unit 142 and a second processing unit 143 .

[0144] The data acquisition unit 141 is configured to acquire at least one type of sensor data. The first processing unit 142 is configured to input each of the sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, and obtain the initial feature information corresponding to each of the sensor data. The second processing unit 143 is configured to input each of the initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtain at least one task processing result.

[0145] In an optional implementation, the second processing unit 143 is further configured to respond to the current task objective including a fusion processing task, input the multiple initial feature information required for the fusion processing task into the feature fusion network of the fusion branch network for feature fusion, obtain fusion feature information, input the fusion feature information into the fusion head network of the fusion branch network for processing, and obtain the corresponding task processing result.

[0146] In an optional implementation, the second processing unit 143 is further configured to input the initial feature information corresponding to each single sensor processing task into the corresponding sensor head network for processing in response to the current task target including at least one single sensor processing task, so as to obtain the corresponding task processing result.

[0147] Optionally, the at least one type of sensor data includes image data and / or radar point cloud data.

[0148] In an optional implementation, the multi-task data processing device 14 includes a model training unit. The model training unit includes a data sampling subunit and a training subunit.

[0149] The data sampling subunit is configured to perform data sampling on at least one data set based on a predetermined data sampling architecture to obtain multiple training data batches, wherein the training data batches include at least one training sample, and the training sample includes sensor data, a data label, and a target head network identifier. The training subunit is configured to input the corresponding training sample into the multi-task model for processing based on the target head network identifier to train the multi-task model.

[0150] In an optional implementation, the data sampling architecture is defined with a data set category class and a sampler class;

[0151] The data set category class includes a data set list and a data determination function, and the sampler class includes the data set category class, a sampler list, and a relationship between data sets in the data set list and samplers in the sampler list.

[0152] In an optional implementation, the data sampling subunit is further configured to execute:

[0153] Execute the data sampling framework, and determine a target data set and a target sampler corresponding to the target data set according to a corresponding sampling instruction;

[0154] Performing data sampling on the target data set based on the target sampler to obtain corresponding training data batches;

[0155] The meta-attributes of the training data batch are updated, the meta-attributes comprising a target header network identification field.

[0156] In an optional implementation, the training subunit is further configured to execute:

[0157] Based on the training samples of the target head network identified as the sensor head network, the corresponding sensor branch network is trained, wherein the sensor branch network includes a sensor trunk network and a sensor head network;

[0158] The parameters of the trained sensor backbone network are maintained, and the fusion branch network is trained based on the training samples of the target head network identified as the fusion head network.

[0159] In an optional implementation, the training subunit is further configured to execute:

[0160] Based on the training samples of the target head network identified as the sensor head network, the corresponding sensor branch network is trained, wherein the sensor branch network includes a sensor trunk network and a sensor head network;

[0161] In response to the completion of the training of each of the sensor branch networks, training the fusion branch network based on the training samples of the target head network identified as the fusion head network;

[0162] The parameters of each sensor backbone network are fine-tuned based on the difference between the sample task processing results of the fusion branch network and the corresponding data labels.

[0163] Optionally, the loss of feature points corresponding to missing data labels in the training samples is a predetermined value.

[0164] The embodiment of the present invention obtains at least one type of sensor data, inputs each sensor data into the corresponding sensor backbone network in the pre-trained multi-task model, obtains the initial feature information corresponding to each sensor data, inputs each initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing, and obtains at least one task processing result. The architecture of the multi-task model of this embodiment can perform multi-tasks suitable for single sensor data, and can also perform multi-tasks suitable for the fusion of multiple sensor data, and decouples each network module in the multi-task model, improves the scalability and compatibility of the multi-task model, and reduces code redundancy.

[0165] Fig.15 Schematic diagram of a training device for a multi-task model according to an embodiment of the present invention. Fig.15 As shown, the training device 15 of the multi-task model includes a data sampling unit 151 and a training unit 152.

[0166] The sampling unit 151 is configured to perform data sampling on at least one data set based on a predetermined data sampling architecture to obtain multiple training data batches, wherein the training data batches include at least one training sample, and the training sample includes sensor data, data labels, and a target head network identifier. The training unit 152 is configured to input the corresponding training sample into the multi-task model for processing based on the target head network identifier to train the multi-task model.

[0167] In an optional implementation, the data sampling architecture is defined with a data set category class and a sampler class; the data set category class includes a data set list and a data determination function, and the sampler class includes the data set category class, the sampler list, and the relationship between the data sets in the data set list and the samplers in the sampler list;

[0168] The data sampling subunit is further configured to perform:

[0169] Execute the data sampling framework, and determine a target data set and a target sampler corresponding to the target data set according to a corresponding sampling instruction;

[0170] Performing data sampling on the target data set based on the target sampler to obtain corresponding training data batches;

[0171] The meta-attributes of the training data batch are updated, the meta-attributes comprising a target header network identification field.

[0172] This embodiment obtains multiple training data batches by sampling at least one data set based on a predetermined data sampling architecture, wherein the training data batches include at least one training sample, and the training sample includes sensor data, data labels, and target head network identifiers. Based on the target head network identifier, the corresponding training samples are input into a multi-task model for processing to train the multi-task model to obtain a trained multi-task model, thereby improving the efficiency of model training and the task processing accuracy of the trained multi-task model.

[0173] Fig.16 Schematic diagram of a vehicle according to an embodiment of the present invention. Fig.16 As shown, the vehicle 16 of this embodiment includes a vehicle body 161, at least one type of sensor 162 and a controller 163. Among them, at least one type of sensor 162 is configured to collect sensor data. Among them, the sensor data includes environmental data around the vehicle. The controller 163 can be an independent microprocessor or a collection of one or more microprocessors to execute the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices, thereby realizing the automatic driving function.

[0174] Fig.17 Schematic diagram of an electronic device according to an embodiment of the present invention. Fig.17As shown, the server 17 is a general data processing device, which includes a general computer hardware structure, which at least includes a processor 171 and a memory 172. The processor 171 and the memory 172 are connected via a bus 173. The memory 172 is suitable for storing instructions or programs executable by the processor 171. The processor 171 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 171 executes the instructions stored in the memory 172, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 173 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 174 and the display device and the input / output (I / O) device 175. The input / output (I / O) device 175 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 175 is connected to the system via an input / output (I / O) controller 176.

[0175] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0177] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0178] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0179] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0180] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0181] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-task data processing method, characterized in that: The method comprises: Acquire multiple different types of sensor data; Inputting each of the sensor data into a corresponding sensor backbone network in a pre-trained multi-task model to obtain initial feature information corresponding to each of the sensor data, wherein different types of sensor data correspond to different sensor backbone networks; Inputting each of the initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing according to the current task goal, and obtaining at least one task processing result; Among them, the fusion branch network includes a fusion network and a fusion head network. The fusion network is used to perform feature fusion on multiple initial feature information corresponding to multiple sensors to obtain fused feature information. The fusion head network is used to process the fused feature information to obtain the task processing result of at least one corresponding task.

2. The method according to claim 1, characterized in that: Inputting each of the initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing to obtain at least one task processing result comprises: In response to the current task target including a fusion processing task, a plurality of the initial feature information required for the fusion processing task is input into the feature fusion network of the fusion branch network for feature fusion to obtain fusion feature information; The fused feature information is input into the fusion head network of the fusion branch network for processing to obtain the corresponding task processing result.

3. The method according to claim 1 or 2, characterized in that: Inputting each of the initial feature information into the corresponding sensor head network and / or fusion branch network in the multi-task model for processing to obtain at least one task processing result comprises: In response to the current task target including at least one single sensor processing task, initial feature information corresponding to each single sensor processing task is input into the corresponding sensor head network for processing to obtain a corresponding task processing result.

4. The method according to claim 1, characterized in that The sensor data includes at least image data or radar point cloud data.

5. The method according to claim 1, characterized in that The multi-task model is trained by the following steps: Performing data sampling on at least one data set based on a predetermined data sampling architecture to obtain a plurality of training data batches, wherein the training data batches include at least one training sample, and the training sample includes sensor data, a data label, and a target head network identifier; Based on the target head network identifier, the corresponding training samples are input into the multi-task model for processing to train the multi-task model.

6. The method according to claim 5, characterized in that The data sampling architecture is defined with a data set category class and a sampler class; The data set category class includes a data set list and a data determination function, and the sampler class includes the data set category class, a sampler list, and a relationship between data sets in the data set list and samplers in the sampler list.

7. The method according to claim 6, characterized in that The step of sampling at least one data set based on a predetermined data sampling architecture to obtain a plurality of training data batches comprises: Execute the data sampling framework, and determine a target data set and a target sampler corresponding to the target data set according to a corresponding sampling instruction; Performing data sampling on the target data set based on the target sampler to obtain corresponding training data batches; The meta-attributes of the training data batch are updated, the meta-attributes comprising a target header network identification field.

8. The method according to claim 5, characterized in that The step of inputting corresponding training samples into the multi-task model for processing based on the target head network identifier to train the multi-task model includes: Based on the training samples of the target head network identified as the sensor head network, the corresponding sensor branch network is trained, wherein the sensor branch network includes a sensor trunk network and a sensor head network; The parameters of the trained sensor backbone network are maintained, and the fusion branch network is trained based on the training samples of the target head network identified as the fusion head network.

9. The method according to claim 5, characterized in that The step of inputting corresponding training samples into the multi-task model for processing based on the target head network identifier to train the multi-task model includes: Based on the training samples of the target head network identified as the sensor head network, the corresponding sensor branch network is trained, wherein the sensor branch network includes a sensor trunk network and a sensor head network; In response to the completion of the training of each of the sensor branch networks, training the fusion branch network based on the training samples of the target head network identified as the fusion head network; The parameters of each sensor backbone network are fine-tuned based on the difference between the sample task processing results of the fusion branch network and the corresponding data labels.

10. The method according to any one of claims 5 to 9, characterized in that: The loss of the feature points corresponding to the missing data labels in the training samples is a predetermined value.

11. A multi-task model training method, characterized in that: The method comprises: Performing data sampling on at least one data set based on a predetermined data sampling architecture to obtain a plurality of training data batches, wherein the training data batches include at least one training sample, and the training sample includes sensor data, a data label, and a target head network identifier; Based on the target head network identifier, inputting the corresponding training sample into the multi-task model for processing to train the multi-task model; The multi-task model includes a plurality of sensor backbone networks, a plurality of head networks and at least one fusion branch network, wherein the plurality of sensor backbone networks correspond to multiple types of sensors, the sensor backbone network is used to obtain initial feature information corresponding to each sensor data, the head network takes the output of the corresponding sensor backbone network as input, and the fusion branch network; Among them, the fusion branch network includes a fusion network and a fusion head network. The fusion network is used to perform feature fusion on multiple initial feature information corresponding to multiple sensors to obtain fused feature information. The fusion head network is used to process the fused feature information to obtain the task processing result of at least one corresponding task.

12. The method according to claim 11, characterized in that The data sampling architecture is defined with a data set category class and a sampler class; the data set category class includes a data set list and a data determination function, and the sampler class includes the data set category class, the sampler list, and the relationship between the data sets in the data set list and the samplers in the sampler list; The step of sampling at least one data set based on a predetermined data sampling architecture to obtain a plurality of training data batches comprises: Execute the data sampling framework, and determine a target data set and a target sampler corresponding to the target data set according to a corresponding sampling instruction; Performing data sampling on the target data set based on the target sampler to obtain corresponding training data batches; The meta-attributes of the training data batch are updated, the meta-attributes comprising a target header network identification field.

13. A vehicle, characterized in that: The vehicle comprises: Vehicle body; at least one type of sensor configured to collect sensor data, the sensor data including environmental data surrounding the vehicle; A controller configured to execute the method according to any one of claims 1 to 12.

14. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

Images