Training, prediction method of multi-task prediction model and storage medium

Abstract

The application discloses a kind of multi-task prediction model training, prediction method and storage medium, the method comprises: obtaining sample point cloud data;Sample point cloud data is input into preset model, and sample point cloud data is carried out point cloud feature extraction based on the point cloud feature extraction network of preset model, and sample point cloud feature is obtained;Multiple sample semantic features are predicted based on the object attribute prediction network of preset model Object attribute prediction, and sample predicted object attribute is obtained;Multiple sample semantic features are processed based on the clustering network of preset model Clustering, and sample predicted center point and sample predicted offset are obtained;Then, according to the prediction result and label, preset model is trained, and multi-task prediction model is obtained.The model of the application can reduce the calculation complexity in multi-task prediction process, improve the efficiency and real-time performance of multi-task prediction.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a training and prediction method and storage medium for a multi-task prediction model. Background Technology

[0002] Deep learning-based 3D object detection is gradually becoming a mainstream method for autonomous driving. However, in the general obstacle detection domain, traditional rules have been the primary approach, with very few based on lidar deep learning. Similarly, real-time performance remains a significant obstacle to the implementation of lidar-based general object detection. Although the rapid development of AI chips in recent years has greatly improved the real-time performance of deep learning model deployment, it still cannot solve many real-time problems. General object detection based on traditional rules has high requirements for computational resources and computational complexity. Furthermore, fusing 3D object detection results with classification labels consumes additional computational resources and increases deployment difficulty, thus causing inference delays throughout the autonomous driving process.

[0003] Traditional solutions typically employ CPU computation. On NVIDIA platforms, this can be switched to CUDA computation to accelerate the results. However, on other platforms, such AI computing power generally does not support this type of accelerated computation. Therefore, even on NVIDIA platforms, the problem cannot be fundamentally solved. Summary of the Invention

[0004] This invention provides a training and prediction method and storage medium for a multi-task prediction model, which can reduce the computational complexity in the multi-task prediction process and improve the efficiency and real-time performance of multi-task prediction.

[0005] On one hand, the present invention provides a training method for a multi-task prediction model, the method comprising:

[0006] Acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute tags, sample center point tags, and sample offset tags; the sample object attributes include at least one of the sample object's heatmap, size information, and category information.

[0007] The sample point cloud data is input into a preset model, and the point cloud feature extraction network of the preset model is used to extract point cloud features from the sample point cloud data to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object.

[0008] Based on the preset model, the object attribute prediction network performs object attribute prediction on the semantic features of the multiple samples to obtain the predicted object attributes of the samples.

[0009] The clustering network based on the preset model performs clustering processing on the semantic features of the multiple samples to obtain the sample prediction center point and the sample prediction offset.

[0010] The preset model is trained based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

[0011] Optionally, the clustering network includes a clustering sub-network, a centroid prediction network, and an offset prediction network. The clustering network based on the preset model performs clustering processing on the semantic features of the multiple samples to obtain the predicted centroids and predicted offsets of the samples, including:

[0012] Based on the clustering subnetwork, the semantic features of the multiple samples are clustered to obtain the sample clustering results;

[0013] Based on the centroid prediction network, the sample clustering results are processed to predict centroids of the samples.

[0014] Based on the offset prediction network, the sample clustering results are processed to predict the offset, thereby obtaining the predicted offset of the sample.

[0015] Optionally, training the preset model to obtain a multi-task prediction model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction centroid, the sample centroid label, the sample prediction offset, and the sample offset label includes:

[0016] The target loss information is determined based on the first difference between the sample predicted object attribute and the sample object attribute label, the second difference between the sample predicted centroid and the sample centroid label, and the third difference between the sample predicted offset and the sample offset label.

[0017] The model parameters of the preset model are adjusted according to the target loss information until the training termination condition is met, and the preset model at the end of training is determined as the multi-task prediction model.

[0018] Optionally, determining the target loss information based on the first difference between the predicted object attribute and the attribute label of the sample object, the second difference between the predicted center point and the label of the sample center point, and the third difference between the predicted offset and the label of the sample offset includes:

[0019] First loss information is determined based on the first difference between the predicted object attribute and the attribute label of the sample object;

[0020] The second loss information is determined based on the second difference between the predicted center point of the sample and the label of the center point of the sample;

[0021] The third loss information is determined based on the third difference between the predicted offset of the sample and the offset label of the sample;

[0022] The target loss information is determined based on the first loss information, the second loss information, and the third loss information.

[0023] Optionally, the point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features, including:

[0024] The point cloud feature extraction network based on the preset model extracts the vertical axis data and point cloud intensity of the sample point cloud data.

[0025] Based on the vertical axis data and point cloud intensity of the sample point cloud data, a sample pseudo-image and a sample column are generated.

[0026] Point cloud features are extracted based on the sample pseudo-image and sample pillars to obtain the sample point cloud features.

[0027] Optionally, the preset model further includes a semantic segmentation network. After the point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain the sample point cloud features, the method further includes:

[0028] Based on the semantic segmentation network, the sample point cloud features are semantically segmented to obtain multiple sample semantic features.

[0029] Optionally, acquiring the sample point cloud data includes:

[0030] Obtain initial point cloud data for the sample, wherein the initial point cloud data for the sample is labeled with the attribute labels of the sample object and the label of the sample segmentation result;

[0031] Based on the initial point cloud data of the sample and the label of the sample segmentation result, generate sample center point label and sample offset label;

[0032] The initial point cloud data of the sample is labeled with the attribute labels of the sample object, the label of the sample center point, and the label of the sample offset to obtain the sample point cloud data.

[0033] On the other hand, a prediction method for a multi-task prediction model is also provided, the method comprising:

[0034] Acquire the point cloud data to be tested;

[0035] The point cloud data to be tested is input into the multi-task prediction model for multi-task prediction processing to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained using the above method.

[0036] Optionally, the step of inputting the point cloud data to be measured into the multi-task prediction model for multi-task prediction processing to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset includes:

[0037] The point cloud data to be tested is input into the multi-task prediction model, and the point cloud feature extraction network of the multi-task prediction model is used to extract point cloud features from the point cloud data to be tested to obtain target point cloud features; the target point cloud features are used to extract multiple target semantic features.

[0038] Based on the multi-task prediction model, the object attribute prediction network performs object attribute prediction on the multiple target semantic features to obtain the target predicted object attributes.

[0039] The clustering network based on the multi-task prediction model performs clustering processing on the multiple target semantic features to obtain the target prediction center point and the target prediction offset.

[0040] On the other hand, a training device for a multi-task prediction model is provided, the device comprising:

[0041] The sample point cloud acquisition module is used to acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute tags, sample center point tags, and sample offset tags; the sample object attributes include at least one of the sample object's heatmap, size information, and category information.

[0042] The sample feature determination module is used to input the sample point cloud data into a preset model, and extract point cloud features from the sample point cloud data based on the point cloud feature extraction network of the preset model to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object.

[0043] The sample attribute determination module is used to predict the object attributes of the multiple sample semantic features based on the object attribute prediction network of the preset model, and obtain the sample predicted object attributes.

[0044] The sample offset determination module is used to perform clustering processing on the semantic features of the multiple samples based on the clustering network of the preset model to obtain the sample prediction center point and the sample prediction offset.

[0045] The model determination module is used to train the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

[0046] Optionally, the clustering network includes a clustering sub-network, a centroid prediction network, and an offset prediction network, and the sample offset determination module includes:

[0047] The clustering result determination unit is used to perform clustering processing on the semantic features of the multiple samples based on the clustering sub-network to obtain the sample clustering result;

[0048] The center point determination unit is used to perform center point prediction processing on the sample clustering results based on the center point prediction network to obtain the predicted center points of the samples.

[0049] The offset determination unit is used to perform offset prediction processing on the sample clustering results based on the offset prediction network to obtain the sample predicted offset.

[0050] Optionally, the model determination module includes:

[0051] The target loss determination unit is used to determine target loss information based on a first difference between the sample prediction object attribute and the sample object attribute label, a second difference between the sample prediction center point and the sample center point label, and a third difference between the sample prediction offset and the sample offset label.

[0052] The model determination unit is used to adjust the model parameters of the preset model according to the target loss information until the training termination condition is met, and to determine the preset model at the end of training as the multi-task prediction model.

[0053] Optionally, the target loss determination unit includes:

[0054] The first determining subunit is used to determine first loss information based on the first difference between the sample predicted object attribute and the sample object attribute label;

[0055] The second determining subunit is used to determine the second loss information based on the second difference between the sample predicted center point and the sample center point label;

[0056] The third determining subunit is used to determine the third loss information based on the third difference between the sample predicted offset and the sample offset label;

[0057] The target determination subunit is used to determine the target loss information based on the first loss information, the second loss information, and the third loss information.

[0058] Optionally, the sample feature determination module includes:

[0059] The point cloud data extraction unit is used to extract the vertical axis data and point cloud intensity of the sample point cloud data based on the point cloud feature extraction network of the preset model.

[0060] The sample image generation unit is used to generate sample pseudo-images and sample pillars based on the vertical axis data and point cloud intensity of the sample point cloud data.

[0061] The point cloud determination unit is used to extract point cloud features based on the sample pseudo-image and the sample pillar to obtain the sample point cloud features.

[0062] Optionally, the preset model further includes a semantic segmentation network, and the device further includes:

[0063] The sample semantic feature determination module is used to perform semantic segmentation processing on the sample point cloud features based on the semantic segmentation network to obtain multiple sample semantic features.

[0064] Optionally, the sample point cloud acquisition module includes:

[0065] An initial point cloud acquisition unit is used to acquire initial point cloud data of a sample, wherein the initial point cloud data of the sample is labeled with the attribute labels of the sample object and the label of the sample segmentation result.

[0066] The label generation unit is used to generate sample center point labels and sample offset labels based on the initial point cloud data of the sample and the sample segmentation result labels;

[0067] The labeling unit is used to label the initial point cloud data of the sample with the attribute labels of the sample objects, the center point labels of the samples, and the offset labels of the samples, so as to obtain the sample point cloud data.

[0068] On the other hand, a prediction device for a multi-task prediction model is provided, the device comprising:

[0069] The test point cloud acquisition module is used to acquire the test point cloud data;

[0070] The multi-task prediction module is used to input the point cloud data to be measured into the multi-task prediction model for multi-task prediction processing, and to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained using the above method.

[0071] Optionally, the multi-task prediction module includes:

[0072] The target point cloud determination unit is used to input the point cloud data to be tested into the multi-task prediction model, and extract point cloud features from the point cloud data to be tested based on the point cloud feature extraction network of the multi-task prediction model to obtain target point cloud features; the target point cloud features are used to extract multiple target semantic features.

[0073] The target attribute prediction unit is used to predict the object attributes of the multiple target semantic features based on the object attribute prediction network of the multi-task prediction model, so as to obtain the target predicted object attributes.

[0074] The target offset prediction unit is used to perform clustering processing on the multiple target semantic features based on the clustering network of the multi-task prediction model to obtain the target prediction center point and the target prediction offset.

[0075] On the other hand, an electronic device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the training method or prediction method of the multi-task prediction model as described above.

[0076] On the other hand, a computer storage medium is provided, which stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the training method or prediction method of the multi-task prediction model as described above.

[0077] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a training method or prediction method to implement the multi-task prediction model as described above.

[0078] The training, prediction method, and storage medium for the multi-task prediction model provided by this invention have the following technical advantages:

[0079] This invention acquires sample point cloud data; the sample point cloud data is labeled with sample object attribute tags, sample center point tags, and sample offset tags; the sample object attributes include at least one of the sample object's heatmap, size information, and category information; the sample point cloud data is input into a preset model, and a point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object; an object attribute prediction network based on the preset model predicts object attributes from the multiple sample semantic features to obtain the sample predicted object attributes; a clustering network based on the preset model clusters the multiple sample semantic features to obtain the sample predicted center point and sample predicted offset; then, based on the sample predicted object attributes, sample object attribute tags, sample predicted center point, sample center point tag, sample predicted offset, and sample offset tag, the preset model is trained to obtain a multi-task prediction model. The model trained by this invention can reduce the computational complexity in the multi-task prediction process and improve the efficiency and real-time performance of multi-task prediction. Attached Figure Description

[0080] To more clearly illustrate the technical solutions and advantages in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0081] Figure 1 This is a schematic diagram of the network structure of a multi-task prediction model provided in the embodiments of this specification;

[0082] Figure 2 This is a flowchart illustrating a training method for a multi-task prediction model provided in the embodiments of this specification.

[0083] Figure 3 This is a schematic diagram of the structure of a point cloud feature extraction network provided in the embodiments of this specification;

[0084] Figure 4 This is a schematic diagram of the structure of a semantic segmentation network provided in the embodiments of this specification;

[0085] Figure 5 This is a schematic diagram of a Pillar conversion into a BEV view feature map provided in the embodiments of this specification;

[0086] Figure 6 This is a flowchart illustrating a method provided in this specification for clustering the semantic features of multiple samples using a clustering network based on the preset model to obtain the sample prediction center point and the sample prediction offset.

[0087] Figure 7 This is a schematic diagram of the structure of a clustering network provided in the embodiments of this specification;

[0088] Figure 8 This is a flowchart illustrating a method for training the preset model to obtain a multi-task prediction model, as provided in an embodiment of this specification.

[0089] Figure 9 This is a flowchart illustrating a method for determining target loss information based on a first difference between the predicted object attribute and the attribute label of the sample object, a second difference between the predicted center point of the sample and the label of the sample center point, and a third difference between the predicted offset of the sample and the label of the sample offset, provided in an embodiment of this specification.

[0090] Figure 10 This is a schematic diagram of the structure of a training device for a multi-task prediction model provided in the embodiments of this specification;

[0091] Figure 11 This is a schematic diagram of the structure of a server provided in the embodiments of this specification. Detailed Implementation

[0092] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0094] In this embodiment of the invention, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0095] like Figure 1 As shown, Figure 1 This diagram illustrates the network structure of the model in this embodiment, including a point cloud feature extraction network, a semantic segmentation network, a clustering network, a center point prediction network, an offset prediction network, and an object attribute prediction network. During model application, the point cloud data to be tested (e.g., a point cloud image) can be input into the model. The point cloud feature extraction network extracts target point cloud features, and the semantic segmentation network performs semantic segmentation on these features to obtain multiple target semantic features. These multiple target semantic features are then input into the clustering network and the object attribute prediction network. The object attribute prediction network predicts object attributes from these features to obtain the predicted object attributes. The clustering network extracts cluster features from these features to obtain target cluster features. The center point prediction network predicts center points from these cluster features to obtain the predicted center points. The offset prediction network predicts offsets from these cluster features to obtain the predicted offsets. A first display image can then be generated based on the predicted center points and the predicted offsets. A second display image is generated based on the predicted object attributes, displaying multiple predicted objects, which may include, but are not limited to, vehicles and obstacles.

[0096] The following describes a training method for a multi-task prediction model according to the present invention. Figure 2 This is a flowchart illustrating a training method for a multi-task prediction model provided in an embodiment of this specification. This specification provides the operational steps described in the embodiments or flowchart, but based on conventional or non-inventive methods, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0097] S201: Obtain sample point cloud data; the sample point cloud data is labeled with sample object attribute labels, sample center point labels, and sample offset labels; the sample object attributes include at least one of the sample object's heatmap, size information, and category information.

[0098] In the embodiments of this specification, sample point cloud data of a vehicle during its historical driving process can be acquired. This sample point cloud data may include a point cloud map or data obtained by processing a point cloud map. Sample object attributes include at least one of the following: a heatmap of the sample object, size information, and category information; wherein, size information may include information such as the height of the sample object.

[0099] In the embodiments of this specification, acquiring sample point cloud data includes:

[0100] Obtain initial point cloud data for the sample, wherein the initial point cloud data for the sample is labeled with the attribute labels of the sample object and the label of the sample segmentation result;

[0101] Based on the initial point cloud data of the sample and the label of the sample segmentation result, generate sample center point label and sample offset label;

[0102] The initial point cloud data of the sample is labeled with the attribute labels of the sample object, the label of the sample center point, and the label of the sample offset to obtain the sample point cloud data.

[0103] In the embodiments of this specification, sample object attribute labels and sample segmentation result labels can be labeled on the initial point cloud data of the sample first; then, the sample center point label and sample offset label can be obtained by parsing the sample segmentation result label, thereby labeling the initial point cloud data of the sample with the sample object attribute label, the sample center point label and the sample offset label, and obtaining the sample point cloud data, which reduces the workload of manual labeling and improves the labeling efficiency of the data.

[0104] S203: Input the sample point cloud data into a preset model, and extract point cloud features from the sample point cloud data based on the point cloud feature extraction network of the preset model to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object.

[0105] In this embodiment of the specification, the point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features, including:

[0106] The point cloud feature extraction network based on the preset model extracts the vertical axis data and point cloud intensity of the sample point cloud data.

[0107] Based on the vertical axis data and point cloud intensity of the sample point cloud data, a sample pseudo-image and a sample column are generated.

[0108] Point cloud features are extracted based on the sample pseudo-image and sample pillars to obtain the sample point cloud features.

[0109] In the embodiments of this specification, the point cloud detection range format is [x_min, y_min, z_min, x_max, y_max, z_max], and the true value is [0, 102.4, -3, -76.8, 76.8, 3]; the Pillar size format is [p_x, p_y, p_z], and the true value is [0.32, 0.32, 6]; therefore, the generated top view grid size is [480, 480]. Since the pseudo-image is generated using Pillar and the final feature output scale (HW) remains unchanged after convolution and other operations, the ground truth label size is also [480, 480].

[0110] Based on the design dimensions of the truth system, preprocessing requires obtaining pseudo-image information. Based on the z-axis and intensity of the input point cloud, a two-channel pseudo-image is first generated. Following the label data design of the truth system, pillar information is obtained. The pseudo-image generated based on the pillar has a size of [480, 480], calculated as follows:

[0111]

[0112] Where Xmax is the maximum value on the x-axis, Xmin is the minimum value on the x-axis, and y max The maximum value in the ordinate, y min px is the minimum value in the vertical axis; py is the horizontal axis of the column and py is the vertical axis of the column.

[0113] In the embodiments described in this specification, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a point cloud feature extraction network. The point cloud feature extraction network can be Pillar Feature Net, which includes four sub-networks: Pillarencoder, Pointnet, Featurescopression, and Scattermax. These sub-networks encode and expand the dimensions of the sample point cloud data [N, 4] in sequence, and finally obtain a multi-dimensional array with a scale of [N, 64, 480, 480].

[0114] In some embodiments, the preset model further includes a semantic segmentation network. After the point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain the sample point cloud features, the method further includes:

[0115] Based on the semantic segmentation network, the sample point cloud features are semantically segmented to obtain multiple sample semantic features.

[0116] In some embodiments, such as Figure 4 As shown, Figure 4 This is a schematic diagram of a semantic segmentation network structure. The semantic segmentation network can be BEV Unet, a 2D convolutional neural network, or a VGG (Visual Geometry Group) or FPN network structure. VGG, also known as VGGNet, was the runner-up in the 2014 ImageNet competition. It is a convolutional neural network implemented by stacking convolutional and pooling layers, and finally adding a fully connected layer. Its main feature is the use of smaller filters, i.e., convolutional kernels. Feature Pyramid Networks (FPN) are a deep learning network structure used for object detection and semantic segmentation. Its main function is to solve the multi-scale problem in object detection, especially when dealing with small objects. The core innovation of FPN lies in its pyramid-shaped feature hierarchy structure, which, through cross-layer connections and feature fusion, enables the network to extract and predict features at different scales.

[0117] In some embodiments, such as Figure 5 As shown, Figure 5 This is a schematic diagram of Pillar conversion into a BEV-view feature map. Figure 5 Figure a shows a Pillar column chart. Figure 5 Figure b is a top-view feature map of the BEV, converted from the Pillar column diagram.

[0118] S205: Based on the preset model, the object attribute prediction network performs object attribute prediction on the semantic features of the multiple samples to obtain the predicted object attributes of the samples.

[0119] In the embodiments of this specification, the object attribute prediction network of the multiple sample semantic features can be used to predict the object attributes of the sample; the sample object may include, but is not limited to, vehicles, obstacles, etc.; a sample point cloud data may include one or more categories of sample objects.

[0120] S207: Based on the clustering network of the preset model, the semantic features of the multiple samples are clustered to obtain the sample prediction center point and the sample prediction offset.

[0121] In the embodiments described in this specification, such as Figure 6As shown, the clustering network includes a clustering sub-network, a centroid prediction network, and an offset prediction network. The clustering network based on the preset model performs clustering processing on the semantic features of the multiple samples to obtain the predicted centroids and predicted offsets of the samples, including:

[0122] S2071: Based on the clustering subnetwork, perform clustering processing on the semantic features of the multiple samples to obtain sample clustering results;

[0123] S2073: Based on the centroid prediction network, perform centroid prediction processing on the sample clustering results to obtain the predicted centroids of the samples;

[0124] S2075: Based on the offset prediction network, perform offset prediction processing on the sample clustering results to obtain the sample predicted offset.

[0125] In the embodiments of this specification, to obtain the clustering results, feature results are extracted, and the center point heatmap and corresponding center point offset are obtained through one convolution and one MLP fully connected layer. The original 3D object detection OD head continues to use an anchor-free method to predict the heatmap, height, dimension, heading, and center point, and the loss values ​​are also the original L1 Loss and Gaussian Focal Loss. The clustering prediction head uses MSELoss for the center heatmap and L1 Loss for the offset.

[0126] In the embodiments described in this specification, such as Figure 7 As shown, Figure 7 This is a schematic diagram of a clustering network structure. The clustering network can be a Cluster Net, which includes convolutional layers (Conv) and fully connected layers (MLP).

[0127] S209: Train the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

[0128] In the embodiments of this specification, the preset model can be trained based on the first difference between the sample prediction object attribute and the sample object attribute label, the second difference between the sample prediction center point and the sample center point label, and the third difference between the sample prediction offset and the sample offset label to obtain a multi-task prediction model.

[0129] In the embodiments described in this specification, such as Figure 8 As shown, the step of training the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction centroid, the sample centroid label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model includes:

[0130] S2091: Determine the target loss information based on the first difference between the sample predicted object attribute and the sample object attribute label, the second difference between the sample predicted center point and the sample center point label, and the third difference between the sample predicted offset and the sample offset label;

[0131] In the embodiments described in this specification, such as Figure 9 As shown, determining the target loss information based on the first difference between the predicted object attribute and the attribute label of the sample object, the second difference between the predicted center point and the label of the sample center point, and the third difference between the predicted offset and the label of the sample offset includes:

[0132] S20911: Determine first loss information based on the first difference between the predicted object attribute of the sample and the attribute label of the sample object;

[0133] S20913: Determine the second loss information based on the second difference between the predicted center point of the sample and the label of the center point of the sample;

[0134] S20915: Determine third loss information based on the third difference between the predicted sample offset and the sample offset label;

[0135] S20917: Determine the target loss information based on the first loss information, the second loss information, and the third loss information.

[0136] In the embodiments of this specification, after determining the first loss information, the second loss information, and the third loss information, the sum of the first loss information, the second loss information, and the third loss information can be calculated to obtain the target loss information. Alternatively, the weights of the first loss information, the second loss information, and the third loss information can be determined respectively, and then a weighted sum of the first loss information, the second loss information, and the third loss information can be calculated to obtain the target loss information.

[0137] S2093: Adjust the model parameters of the preset model according to the target loss information until the training termination condition is met, and determine the preset model at the end of training as the multi-task prediction model.

[0138] In the embodiments of this specification, the training termination condition may include, but is not limited to, the target loss information being less than a preset threshold or the number of iterations reaching a preset number. The preset model at the end of training is determined as the multi-task prediction model.

[0139] In the embodiments described in this specification, for LiDAR3D object detection, it is impossible to solve the object detection problem in real-world autonomous driving scenarios by infinitely expanding the types of detected objects; a more general approach is necessary. Similarly, for the long-tail problem, infinitely collecting labeled corner cases is also impractical. Therefore, this invention combines traditional rule-based clustering with a deep learning model to solve the detection of general or unknown obstacles, while simultaneously eliminating the need for traditional rule-based clustering during deployment.

[0140] Most open-source point cloud segmentation algorithms are based on 3D convolution, which offers high accuracy but also increases computational complexity, creating a barrier to real-time performance. Furthermore, embedded deployments require NVIDIA platforms, as ordinary AI chips struggle to implement 3D or sparse convolution. Therefore, this invention employs the PointPillars algorithm, using pillars instead of voxels to reduce computational complexity, and utilizes BEV features for learning to acquire target features. Moreover, a dual-anchor-free approach significantly simplifies the computational complexity of subsequent target matching.

[0141] As can be seen from the technical solutions provided in the embodiments of this specification above, the embodiments of this specification acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute labels, sample center point labels, and sample offset labels; the sample object attributes include at least one of the sample object's heatmap, size information, and category information; the sample point cloud data is input into a preset model, and a point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object; an object attribute prediction network based on the preset model predicts object attributes from the multiple sample semantic features to obtain sample predicted object attributes; a clustering network based on the preset model clusters the multiple sample semantic features to obtain sample predicted center points and sample predicted offsets; then, based on the sample predicted object attributes, sample object attribute labels, sample predicted center points, sample center point labels, sample predicted offsets, and sample offset labels, the preset model is trained to obtain a multi-task prediction model. The model trained by this invention can reduce the computational complexity in the multi-task prediction process and improve the efficiency and real-time performance of multi-task prediction.

[0142] This specification also provides a prediction method for a multi-task prediction model, the method comprising:

[0143] Acquire the point cloud data to be tested;

[0144] The point cloud data to be tested is input into the multi-task prediction model for multi-task prediction processing to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained using the above method.

[0145] For example, the step of inputting the point cloud data to be measured into the multi-task prediction model for multi-task prediction processing to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset includes:

[0146] The point cloud data to be tested is input into the multi-task prediction model, and the point cloud feature extraction network of the multi-task prediction model is used to extract point cloud features from the point cloud data to be tested to obtain target point cloud features; the target point cloud features are used to extract multiple target semantic features.

[0147] Based on the multi-task prediction model, the object attribute prediction network performs object attribute prediction on the multiple target semantic features to obtain the target predicted object attributes.

[0148] The clustering network based on the multi-task prediction model performs clustering processing on the multiple target semantic features to obtain the target prediction center point and the target prediction offset.

[0149] The multi-task prediction model in this embodiment can be applied to the autonomous driving process of vehicles. By identifying obstacles around the vehicle, it can avoid obstacles and thus improve the safety of autonomous driving.

[0150] This specification also provides a training device for a multi-task prediction model, such as... Figure 10 As shown, the device includes:

[0151] The sample point cloud acquisition module 1010 is used to acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute labels, sample center point labels, and sample offset labels; the sample object attributes include at least one of the sample object's heatmap, size information, and category information.

[0152] The sample feature determination module 1020 is used to input the sample point cloud data into a preset model, and extract point cloud features from the sample point cloud data based on the point cloud feature extraction network of the preset model to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object.

[0153] The sample attribute determination module 1030 is used to predict the object attributes of the multiple sample semantic features based on the object attribute prediction network of the preset model, and obtain the sample predicted object attributes.

[0154] The sample offset determination module 1040 is used to perform clustering processing on the semantic features of the multiple samples based on the clustering network of the preset model to obtain the sample prediction center point and the sample prediction offset.

[0155] The model determination module 1050 is used to train the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

[0156] In some embodiments, the clustering network includes a clustering subnetwork, a centroid prediction network, and an offset prediction network, and the sample offset determination module includes:

[0157] The clustering result determination unit is used to perform clustering processing on the semantic features of the multiple samples based on the clustering sub-network to obtain the sample clustering result;

[0158] The center point determination unit is used to perform center point prediction processing on the sample clustering results based on the center point prediction network to obtain the predicted center points of the samples.

[0159] The offset determination unit is used to perform offset prediction processing on the sample clustering results based on the offset prediction network to obtain the sample predicted offset.

[0160] In some embodiments, the model determination module includes:

[0161] The target loss determination unit is used to determine target loss information based on a first difference between the sample prediction object attribute and the sample object attribute label, a second difference between the sample prediction center point and the sample center point label, and a third difference between the sample prediction offset and the sample offset label.

[0162] The model determination unit is used to adjust the model parameters of the preset model according to the target loss information until the training termination condition is met, and to determine the preset model at the end of training as the multi-task prediction model.

[0163] In some embodiments, the target loss determination unit includes:

[0164] The first determining subunit is used to determine first loss information based on the first difference between the sample predicted object attribute and the sample object attribute label;

[0165] The second determining subunit is used to determine the second loss information based on the second difference between the sample predicted center point and the sample center point label;

[0166] The third determining subunit is used to determine the third loss information based on the third difference between the sample predicted offset and the sample offset label;

[0167] The target determination subunit is used to determine the target loss information based on the first loss information, the second loss information, and the third loss information.

[0168] In some embodiments, the sample feature determination module includes:

[0169] The point cloud data extraction unit is used to extract the vertical axis data and point cloud intensity of the sample point cloud data based on the point cloud feature extraction network of the preset model.

[0170] The sample image generation unit is used to generate sample pseudo-images and sample pillars based on the vertical axis data and point cloud intensity of the sample point cloud data.

[0171] The point cloud determination unit is used to extract point cloud features based on the sample pseudo-image and the sample pillar to obtain the sample point cloud features.

[0172] In some embodiments, the preset model further includes a semantic segmentation network, and the apparatus further includes:

[0173] The sample semantic feature determination module is used to perform semantic segmentation processing on the sample point cloud features based on the semantic segmentation network to obtain multiple sample semantic features.

[0174] In some embodiments, the sample point cloud acquisition module includes:

[0175] An initial point cloud acquisition unit is used to acquire initial point cloud data of a sample, wherein the initial point cloud data of the sample is labeled with the attribute labels of the sample object and the label of the sample segmentation result.

[0176] The label generation unit is used to generate sample center point labels and sample offset labels based on the initial point cloud data of the sample and the sample segmentation result labels;

[0177] The labeling unit is used to label the initial point cloud data of the sample with the attribute labels of the sample objects, the center point labels of the samples, and the offset labels of the samples, so as to obtain the sample point cloud data.

[0178] This specification also provides a prediction apparatus for a multi-task prediction model, the apparatus comprising:

[0179] The test point cloud acquisition module is used to acquire the test point cloud data;

[0180] The multi-task prediction module is used to input the point cloud data to be measured into the multi-task prediction model for multi-task prediction processing, and to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained using the above method.

[0181] In some embodiments, the multi-task prediction module includes:

[0182] The target point cloud determination unit is used to input the point cloud data to be tested into the multi-task prediction model, and extract point cloud features from the point cloud data to be tested based on the point cloud feature extraction network of the multi-task prediction model to obtain target point cloud features; the target point cloud features are used to extract multiple target semantic features.

[0183] The target attribute prediction unit is used to predict the object attributes of the multiple target semantic features based on the object attribute prediction network of the multi-task prediction model, so as to obtain the target predicted object attributes.

[0184] The target offset prediction unit is used to perform clustering processing on the multiple target semantic features based on the clustering network of the multi-task prediction model to obtain the target prediction center point and the target prediction offset.

[0185] The apparatus and method embodiments described herein are based on the same inventive concept.

[0186] This specification provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the training method or prediction method of the multi-task prediction model provided in the above method embodiments.

[0187] Embodiments of the present invention also provide a computer storage medium, which can be disposed in a terminal to store at least one instruction or at least one program related to the training method of a multi-task prediction model in the method embodiment. The at least one instruction or at least one program is loaded and executed by the processor to implement the training method or prediction method of the multi-task prediction model provided in the above method embodiment.

[0188] Embodiments of the present invention also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a training method or prediction method for the multi-task prediction model provided in the above-described method embodiments.

[0189] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0190] The memory described in the embodiments of this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for the functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0191] The training method for the multi-task prediction model provided in the embodiments of this specification can be executed on a mobile terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 11 This is a hardware structure block diagram of a server for a training method of a multi-task prediction model provided in the embodiments of this specification. For example... Figure 11As shown, the server 1100 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1110 (CPUs 1110 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1130 for storing data, and one or more storage media 1120 (e.g., one or more mass storage devices) for storing application programs 1123 or data 1122. The memory 1130 and storage media 1120 may be temporary or persistent storage. The program stored in the storage media 1120 may include one or more modules, each module including a series of instruction operations on the server. Furthermore, the CPU 1110 may be configured to communicate with the storage media 1120 and execute the series of instruction operations stored in the storage media 1120 on the server 1100. Server 1100 may also include one or more power supplies 1160, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1140, and / or one or more operating systems 1121, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0192] The input / output interface 1140 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1100. In one example, the input / output interface 1140 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1140 may be a radio frequency (RF) module for wireless communication with the Internet.

[0193] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1100 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.

[0194] As can be seen from the embodiments of the training and prediction methods, apparatus, devices, or storage media of the multi-task prediction model provided by the present invention, the present invention acquires sample point cloud data; the sample point cloud data is labeled with sample object attribute labels, sample center point labels, and sample offset labels; the sample object attributes include at least one of the sample object's heatmap, size information, and category information; the sample point cloud data is input into a preset model, and the point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features; the sample point cloud features are used to extract multiple sample semantic features of the sample object; the object attribute prediction network based on the preset model predicts object attributes from the multiple sample semantic features to obtain sample predicted object attributes; the clustering network based on the preset model clusters the multiple sample semantic features to obtain sample predicted center points and sample predicted offsets; then, based on the sample predicted object attributes, sample object attribute labels, sample predicted center points, sample center point labels, sample predicted offsets, and sample offset labels, the preset model is trained to obtain a multi-task prediction model. The model trained by the present invention can reduce the computational complexity in the multi-task prediction process and improve the efficiency and real-time performance of multi-task prediction.

[0195] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0196] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0197] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer storage medium, such as a read-only memory, a disk, or an optical disk.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a multi-task prediction model, characterized in that, The method includes: Acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute tags, sample center point tags, and sample offset tags; the sample object attributes include at least one of the sample object's heatmap, size information, and category information. The sample point cloud data is input into a preset model, and the point cloud feature extraction network of the preset model is used to extract point cloud features from the sample point cloud data to obtain sample point cloud features; the preset model also includes a semantic segmentation network, and the sample point cloud features are used to perform semantic segmentation processing to obtain multiple sample semantic features. Based on the preset model, the object attribute prediction network performs object attribute prediction on the semantic features of the multiple samples to obtain the predicted object attributes of the samples. The clustering network based on the preset model performs clustering processing on the semantic features of the multiple samples to obtain the sample prediction center point and the sample prediction offset. The preset model is trained based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

2. The method according to claim 1, characterized in that, The clustering network includes a clustering sub-network, a centroid prediction network, and an offset prediction network. The clustering network based on the preset model performs clustering processing on the semantic features of the multiple samples to obtain the predicted centroids and predicted offsets of the samples, including: Based on the clustering subnetwork, the semantic features of the multiple samples are clustered to obtain the sample clustering results; Based on the centroid prediction network, the sample clustering results are processed to predict centroids of the samples. Based on the offset prediction network, the sample clustering results are processed to obtain the sample predicted offset.

3. The method according to claim 1, characterized in that, The step of training the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction centroid, the sample centroid label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model includes: The target loss information is determined based on the first difference between the sample predicted object attribute and the sample object attribute label, the second difference between the sample predicted centroid and the sample centroid label, and the third difference between the sample predicted offset and the sample offset label. The model parameters of the preset model are adjusted according to the target loss information until the training termination condition is met, and the preset model at the end of training is determined as the multi-task prediction model.

4. The method according to claim 3, characterized in that, The step of determining target loss information based on the first difference between the predicted object attribute and the attribute label of the sample object, the second difference between the predicted center point and the label of the sample center point, and the third difference between the predicted offset and the offset label of the sample includes: First loss information is determined based on the first difference between the predicted object attribute and the attribute label of the sample object; The second loss information is determined based on the second difference between the predicted center point of the sample and the label of the center point of the sample; The third loss information is determined based on the third difference between the predicted offset of the sample and the offset label of the sample; The target loss information is determined based on the first loss information, the second loss information, and the third loss information.

5. The method according to claim 1, characterized in that, The point cloud feature extraction network based on the preset model extracts point cloud features from the sample point cloud data to obtain sample point cloud features, including: The point cloud feature extraction network based on the preset model extracts the vertical axis data and point cloud intensity of the sample point cloud data. Based on the vertical axis data and point cloud intensity of the sample point cloud data, a sample pseudo-image and a sample column are generated; the vertical axis data is the z-axis data. Point cloud features are extracted based on the sample pseudo-image and sample pillars to obtain the sample point cloud features.

6. The method according to claim 1, characterized in that, The acquisition of sample point cloud data includes: Obtain initial point cloud data for the sample, wherein the initial point cloud data for the sample is labeled with the attribute labels of the sample object and the label of the sample segmentation result; Based on the initial point cloud data of the sample and the label of the sample segmentation result, generate sample center point label and sample offset label; The initial point cloud data of the sample is labeled with the attribute labels of the sample object, the label of the sample center point, and the label of the sample offset to obtain the sample point cloud data.

7. A prediction method for a multi-task prediction model, characterized in that, The method includes: acquiring point cloud data to be tested; The point cloud data to be tested is input into the multi-task prediction model for multi-task prediction processing to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained using the method described in any one of claims 1-6.

8. The method according to claim 7, characterized in that, The step of inputting the point cloud data to be measured into the multi-task prediction model for multi-task prediction processing to obtain at least one of the following: target prediction object attributes, target prediction center point, and target prediction offset, includes: The point cloud data to be tested is input into the multi-task prediction model, and the point cloud feature extraction network of the multi-task prediction model is used to extract point cloud features from the point cloud data to be tested to obtain target point cloud features; the target point cloud features are used to extract multiple target semantic features. Based on the multi-task prediction model, the object attribute prediction network performs object attribute prediction on the multiple target semantic features to obtain the target predicted object attributes. The clustering network based on the multi-task prediction model performs clustering processing on the multiple target semantic features to obtain the target prediction center point and the target prediction offset.

9. A training device for a multi-task prediction model, characterized in that, The device includes: The sample point cloud acquisition module is used to acquire sample point cloud data; the sample point cloud data is labeled with sample object attribute labels, sample center point labels, and sample offset labels; the sample object attributes include at least one of the sample object's heatmap, size information, and category information; The sample feature determination module is used to input the sample point cloud data into a preset model, and extract point cloud features from the sample point cloud data based on the point cloud feature extraction network of the preset model to obtain sample point cloud features; the preset model also includes a semantic segmentation network, and performs semantic segmentation processing on the sample point cloud features based on the semantic segmentation network to obtain multiple sample semantic features. The sample attribute determination module is used to predict the object attributes of the multiple sample semantic features based on the object attribute prediction network of the preset model, and obtain the sample predicted object attributes. The sample offset determination module is used to perform clustering processing on the semantic features of the multiple samples based on the clustering network of the preset model to obtain the sample prediction center point and the sample prediction offset. The model determination module is used to train the preset model based on the sample prediction object attributes, the sample object attribute labels, the sample prediction center point, the sample center point label, the sample prediction offset, and the sample offset label to obtain a multi-task prediction model.

10. A prediction device for a multi-task prediction model, characterized in that, The device includes: The test point cloud acquisition module is used to acquire the test point cloud data; The multi-task prediction module is used to input the point cloud data to be tested into the multi-task prediction model for multi-task prediction processing, and to obtain at least one of the target prediction object attributes, target prediction center point, and target prediction offset; the multi-task prediction model is trained by the method described in any one of claims 1-6.

11. An electronic device, characterized in that, The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the training method of the multi-task prediction model as described in any one of claims 1-6 or the prediction method of the multi-task prediction model as described in any one of claims 7-8.

12. A computer storage medium, characterized in that, The computer storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the training method of the multi-task prediction model as described in any one of claims 1-6 or the prediction method of the multi-task prediction model as described in any one of claims 7-8.

13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the training method of the multi-task prediction model according to any one of claims 1-6 or the prediction method of the multi-task prediction model according to any one of claims 7-8.

Images