Detection and segmentation multi-task training method and device, electronic equipment and storage medium

Abstract

The invention discloses a detection and segmentation multi-task training method and device, electronic equipment and a storage medium, and the method comprises the steps: training a backbone network and a segmentation head module according to first point cloud data, and enabling the backbone network and the segmentation head module to serve as a pre-training model, the first point cloud data at least comprises a result marked with point cloud detection and a result marked with point cloud segmentation; the backbone network and the segmentation head module are frozen, the backbone network adopts point cloud data feature parameters in the pre-training model and trains a detection head module according to second point cloud data, and the second point cloud data at least comprises a result marked with a detection frame; and training to obtain a detection and segmentation model according to the backbone network, the segmentation head module and the detection head module. According to the invention, two-stage multi-task model training is realized. The invention further provides a point cloud data processing method.

Description

Technical Field

[0001] The present application relates to the fields of autonomous driving technology and point cloud data processing technology, and in particular to a multi-task training method, device, electronic device, and storage medium for detection and segmentation. Background Art

[0002] Point cloud data segmentation and 3D object detection based on point cloud data are crucial modules in autonomous driving perception systems. Specifically, point cloud segmentation uses a segmentation algorithm to classify LiDAR point clouds at the point level, predicting the semantic information of each point. This helps autonomous vehicles better perceive their surroundings and achieve more fine-grained obstacle segmentation. Specifically, 3D object detection based on point cloud data learns the 3D box of the desired obstacle based on the point cloud information features, including but not limited to the length, width, height, center coordinates, and orientation angle.

[0003] Related technologies typically use separate detection and segmentation heads, each with its own output. This fails to consider the strong insight that the output of the segmentation task provides for the detection task, and also increases the cost of inference time. Summary of the Invention

[0004] The embodiments of the present application provide a multi-task training method, device, electronic device, and storage medium for detection and segmentation to implement two-stage multi-task model training, reduce inference time, and make full use of the output of the segmentation task to provide prompts for the detection task.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a multi-task training method for detection and segmentation, wherein the training method includes:

[0007] Training a backbone network and a segmentation head module as a pre-trained model based on first point cloud data, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results;

[0008] Freezing the backbone network and the segmentation head module, wherein the backbone network uses feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on second point cloud data, wherein the second point cloud data at least includes a result marked with a detection box;

[0009] According to the backbone network, the segmentation head module and the detection head module, a detection and segmentation model is trained.

[0010] Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0011] In some embodiments, further comprising:

[0012] The point cloud segmentation result in the segmentation head module is used as the fifth point cloud feature;

[0013] The fifth point cloud feature is pooled to obtain a third point cloud feature;

[0014] The first point cloud feature in the pre-trained model is combined with the third point cloud feature to obtain a fourth point cloud feature.

[0015] In some embodiments, the dimension of the third point cloud feature is [B, C3, H, W], and is the same as the size of the point cloud feature data in the pre-trained model;

[0016] The dimension of the fourth point cloud feature is [B, C1+C3, H, W].

[0017] In some embodiments, the training of the backbone network and the segmentation head module according to the first point cloud data as a pre-trained model includes:

[0018] The backbone network extracts the overall point cloud features and outputs the first point cloud features. The dimension of the first point cloud features is [B*C1*H, W].

[0019] Among them, B is the batch size of the processed data, C1 is the dimension of the feature, and H and W are the grid length and width of the plane feature.

[0020] In some embodiments, the segmentation head module is used to

[0021] The dimension of the second point cloud feature obtained by processing the first point cloud feature is [B, C2, N], where N represents the number of point clouds and C2 represents the feature dimension of each point cloud;

[0022] The dimension of the fifth point cloud feature output is [B, Class_num, N], where Class_num represents the number of categories for point cloud segmentation, and N represents the number of classified point clouds.

[0023] In some embodiments, the segmentation head module is frozen, and the backbone network uses the feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on the second point cloud data, including:

[0024] The detection and segmentation model outputs the geometric information dimension of each detection box as [B, C, M];

[0025] The detection and segmentation model also outputs category information of dimension [B, det_class, M],

[0026] Wherein, C represents the feature dimension of the detection box, M is the number of output detection boxes, and det_class represents the number of detection categories.

[0027] In a second aspect, an embodiment of the present application further provides a point cloud data processing method, wherein the processing method includes:

[0028] A point cloud detection and segmentation model is obtained by adopting the multi-task training method for detection and segmentation described in the first aspect;

[0029] According to the point cloud detection and segmentation model, the point cloud data is processed to obtain category information and size information results of the point cloud 3D detection box.

[0030] In a third aspect, an embodiment of the present application further provides a multi-task training device for detection and segmentation, wherein the device comprises:

[0031] A pre-training module, configured to train a backbone network and a segmentation head module based on first point cloud data as a pre-training model, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results;

[0032] a detection module, configured to freeze the backbone network and the segmentation head module, wherein the backbone network uses feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on second point cloud data, wherein the second point cloud data at least includes a result marked with a detection box;

[0033] A training module is used to train a detection and segmentation model based on the backbone network, the segmentation head module and the detection head module.

[0034] Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0035] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer-executable instructions, wherein the executable instructions, when executed, enable the processor to perform the above method.

[0036] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes the above method.

[0037] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the backbone network and the segmentation head module are trained according to the first point cloud data, and the segmentation head module is frozen as a pre-training model. The backbone network adopts the feature parameters of the point cloud data in the pre-training model and trains the detection head module according to the second point cloud data. This can better cope with the situation in which detection data is easier to obtain than segmentation data in the actual production environment, and in combination with the need of feature transfer, a two-stage segmentation and detection multi-task training framework is adopted according to the backbone network, the segmentation head module and the detection head module to train the detection and segmentation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 Schematic diagram of the framework of the multi-task training method for detection and segmentation in the embodiment of the present application;

[0040] Figure 2 Schematic diagram of the process of the multi-task training method for detection and segmentation in an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the structure of a multi-task training device for detection and segmentation in an embodiment of the present application;

[0042] Figure 4 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] Because point cloud segmentation and 3D object detection are both essential functions based on point cloud information, some solutions attempt to integrate both modules into a single framework. The basic approach within this framework is to uniformly process point cloud information to generate unified features, which are then connected to different prediction heads to generate different information outputs.

[0045] An obvious benefit of adopting such an integrated approach is that the feature extraction backbone networks of the two functions can be merged as much as possible to save overall inference time, which plays a very important role in autonomous driving systems that focus on real-time performance. However, the two independent prediction heads do not make good use of the complementary information between the two tasks. Although some improvements have attempted to fuse them at the feature level, there are no further more explicit constraints. For example, when point clouds identified as background in the segmentation task have a high confidence level (vegetation, buildings, ground), these point clouds should have a lower probability of being foreground targets in the target detection task. In addition, in actual production, the cost of obtaining training data for point cloud semantic segmentation and training data for 3D target detection is quite different. If a multi-task model needs to be trained, the amount of segmentation data is usually difficult to reach a comparable amount of data as the 3D target detection data. This requires considering a staged training method to adapt to problems in actual production.

[0046] The multi-task training method for detection and segmentation in the embodiments of this application can effectively improve the performance of the detection and segmentation multi-task model and fully utilize data from actual production environments. Compared to independent detection and segmentation heads that output their own task content, the two-stage multi-task training framework used in the embodiments of this application can better improve model performance and utilize data from actual situations at the cost of a small increase in inference time.

[0047] The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0048] like Figure 1 As shown in the figure, the training framework mainly includes the backbone network, the segmentation head module, the combination of the segmentation head features and the pearlescent network, and the detection head module.

[0049] The backbone network is used to extract point cloud features. It can use features extracted at the BEV perspective, at the voxel level, or by combining point cloud-level features with the aforementioned BEV / Voxel features. In short, the backbone network extracts overall point cloud features.

[0050] The segmentation head module extracts features from the backbone network in the form of BEV grids or voxels. These features are re-projected to point cloud granularity through linear interpolation and devoxelization to produce point cloud segmentation results. The segmentation head is the part of the model specifically dedicated to performing image segmentation tasks.

[0051] The fusion of segmentation features and backbone network, since the features output by the backbone network and the features output by the segmentation features have the same length and width but different dimensions, they can be merged.

[0052] The detection head module inputs the result of the fusion of segmentation features and the backbone network, and outputs each final point cloud 3D box category and point cloud 3D box size.

[0053] The present application embodiment provides a multi-task training method for detection and segmentation, such as Figure 2 As shown, a flowchart of a multi-task training method for detection and segmentation in an embodiment of the present application is provided, wherein the method comprises at least the following steps S210 to S230:

[0054] Step S210: training a backbone network and a segmentation head module based on the first point cloud data as a pre-trained model, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results.

[0055] The multi-task training method in the embodiment of the present application is a necessary step in the model training phase, and the steps in the model verification phase correspond to the multi-task training method. The "first point cloud data" used as training data is annotated with the point cloud detection results (usually 3D point cloud detection boxes) and the point cloud segmentation results (the content represented in the 3D scene is obtained according to each point (xyz)). The "segmentation head module" is a part of the deep learning point cloud segmentation model specifically used to perform point cloud segmentation tasks. It is usually located at the end of the model, immediately after the feature extraction part, and is responsible for converting the feature map into pixel-level classification predictions.

[0056] In actual training, in the first phase, integrated data labeled for both detection and segmentation can be used to train the backbone network and segmentation head module, serving as a pre-trained model. Pre-trained models are an important technique in deep learning. By pre-training on large datasets, they learn common feature representations, providing foundational support for subsequent specific tasks. By learning common features from large-scale data, pre-trained models provide a powerful foundation for subsequent tasks and are a key technology for improving efficiency and performance in deep learning.

[0057] Step S220: Freeze the backbone network and the segmentation head module. The backbone network uses the feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on the second point cloud data. The second point cloud data at least includes a result marked with a detection box.

[0058] During actual training, the segmentation head module and the backbone network are frozen, while the backbone network loads the parameters trained in the first phase. The detection head module can then be trained using data labeled with only the detection boxes. Freezing during model training means setting certain layers or parameters of the model to a non-trainable state during training, meaning that these layers or parameters will not be updated during backpropagation.

[0059] The "second point cloud data" serves as training data, annotated only with 3D detection boxes. The detection head module is trained by combining the point cloud feature output from the backbone network with the feature parameters of the point cloud data from the pre-trained model. The "detection head module" is primarily used for object detection tasks, with its core function being to identify and locate specific objects in an image. Object detection requires not only determining the object's category but also its location within the image. The detection head typically achieves this by predicting a bounding box. This bounding box is defined by the position of the top left corner, as well as its width and height.

[0060] In related technologies, the cost of labeling segmentation data and detection data differs significantly, resulting in inconsistent data volumes. This means there's a larger amount of pure detection data and a smaller amount of data labeled for both detection and segmentation. Therefore, by having the first point cloud data include at least the labeled point cloud detection and segmentation results, and the second point cloud data include at least the labeled detection bounding boxes, this problem can be overcome.

[0061] Step S230: Based on the backbone network, the segmentation head module and the detection head module, a detection and segmentation model is trained, wherein the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0062] The backbone network, the segmentation head module, and the detection head module are used to train a detection and segmentation model. The backbone network extracts point cloud feature data, the segmentation head module segments the point cloud to obtain segmentation results, and the detection head module outputs the category and size information of the point cloud 3D detection box as the final output.

[0063] Through the above method, point cloud segmentation is performed first and then 3D point cloud target detection is performed. The point cloud features extracted by the backbone network are put together for detection. Based on the segmentation results and the intermediate features of the detection, the backbone and segmentation features are fused before detection.

[0064] Compared with using independent detection and segmentation heads to output their respective task contents, the above method adopts a two-stage multi-task training framework, which can better improve the performance of the model and utilize the actual data at the cost of a small increase in inference time.

[0065] Unlike related techniques that use two prediction heads to output segmentation and detection results separately, without considering that the output of the segmentation task can provide strong insights for the detection task, this method takes advantage of the strong correlation between segmentation and detection tasks and explicitly adds relevant feature transfer methods during training.

[0066] Different from related technologies, the cost of obtaining training data for point cloud semantic segmentation and 3D object detection is quite different. If a multi-task model needs to be trained, the amount of segmentation data is usually difficult to reach a comparable amount of data as the 3D object detection data. This requires considering a phased training method to adapt to problems in actual production. With the above method, in the first training phase, the backbone network and segmentation head module are trained using integrated data that is labeled with both detection and segmentation. After obtaining the pre-trained model, the detection head module is trained in the second phase using data that is only labeled with the detection box.

[0067] In one embodiment of the present application, it also includes: the point cloud segmentation result in the segmentation head module is used as the fifth point cloud feature; the fifth point cloud feature is pooled to obtain a third point cloud feature; the first point cloud feature in the pre-trained model is merged with the third point cloud feature to obtain a fourth point cloud feature.

[0068] like Figure 1 As shown in the figure, the "fifth point cloud feature" refers to Feat_class_seg obtained after the Segmentation Head module. The "third point cloud feature" refers to Feat_class_seg_revert obtained after max pooling by combining the segmentation head features and the backbone network. The "first point cloud feature" is Feat_common extracted by the backbone network. The "fourth point cloud feature" is Feat_merge_det, the result of merging Feat_common and Feat_class_seg_revert.

[0069] In one embodiment of the present application, the dimension of the third point cloud feature is [B, C3, H, W], which is the same as the size of the point cloud feature data in the pre-trained model; the dimension of the fourth point cloud feature is [B, C1+C3, H, W].

[0070] like Figure 1As shown, the segmentation feature Feat_class_seg can be further subjected to projection methods such as maxpooling to obtain the feature Feat_class_seg_revert, whose dimension is [B, C3, H, W]. By concat Feat_class_seg_revert and Feat_common, we can obtain the Feat_merge_det feature with a dimension of [B, C1+C3, H, W]. C is the dimension of the feature, and H and W are the grid length and width of the BEV feature. B is the batch size of the processed data. In the related art, the integrated detection and segmentation model is a backbone network that extracts features and then outputs different task classifications through two independent prediction heads. If you want to optimize a task separately, it may bring some unexpected changes to another task. The above problem is overcome by merging Feat_common and Feat_class_seg_revert, which have the same length and width as the output of the backbone network.

[0071] In one embodiment of the present application, the backbone network and the segmentation head module are trained according to the first point cloud data as a pre-training model, including: extracting the overall point cloud features through the backbone network, and outputting the first point cloud features, the dimension of the first point cloud features is [B, C1, H, W], where B is the batch size of the processed data, C1 is the dimension of the feature, and H and W are the grid length and width of the planar features.

[0072] like Figure 1 As shown, in the first stage, the backbone network and segmentation head module can be trained using integrated data that is labeled with both detection and segmentation, that is, Figure 1 The backbone network and segmentation head module in the pre-trained model are used as the model. In the second stage, the segmentation head module is frozen, and the backbone network loads the parameters trained in the first stage. At this time, the detection head module can be trained using data with only detection boxes labeled.

[0073] In one embodiment of the present application, the segmentation head module is used to process the first point cloud feature to obtain a second point cloud feature of dimension [B, C2, N], wherein N represents the number of point clouds and C2 represents the feature dimension of each point cloud; the dimension of the output fifth point cloud feature is [B, Class_num, N], wherein Class_num represents the number of categories of point cloud segmentation and N represents the number of classified point clouds.

[0074] like Figure 1 As shown in FIG, the “second point cloud feature” refers to the point cloud feature obtained after bilinear interpolation in the segmentation head module.

[0075] like Figure 1As shown in the figure, the features extracted by the backbone network are in the form of BEV grids or voxels. These features can be re-projected to point cloud granularity through linear interpolation, devoxelization, and other methods. The dimensions of these features are [B, C2, N], where N represents the number of point clouds and C2 represents the feature dimension of each point cloud at this stage. Finally, through a segmentation head module, the output feature Feat_class_seg has the dimensions [B, Class_num, N], where Class_num represents the number of point cloud segmentation categories and N represents the number of classified point clouds.

[0076] In one embodiment of the present application, the backbone network and the segmentation head module are frozen, and the backbone network adopts the feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on the second point cloud data, including: the detection and segmentation model outputs the geometric information dimension of each detection box as [B, C, M]; the detection and segmentation model also outputs the category information dimension as [B, det_class, M], wherein C represents the feature dimension of the detection box, M is the number of output detection boxes, and det_class represents the number of detection categories.

[0077] like Figure 1 As shown, the input of the detection head is Feat_merge_det, which outputs the geometric information of each box [B,7,M].

[0078] The 7th dimension is box (center_x, center_y, center_z, size_x, size_y, size_z, yaw), and M is the number of output detection boxes.

[0079] The detection and segmentation model also outputs class information in the dimension [B, det_class, M], where det_class represents the number of detection classes of interest. It is understood that the result in det_class can be used to obtain the final 3D detection information through NMS (Non-Maximum Suppression).

[0080] As you can understand, non-maximum suppression is a common technique in object detection tasks, primarily used to address the problem of detection algorithms generating multiple detection boxes for the same object. NMS suppresses detection boxes that have a high degree of overlap with the highest-scoring detection box, ensuring that only a single detection box is retained for each object.

[0081] The present application also provides a method for processing point cloud data, wherein the processing method includes:

[0082] A point cloud detection and segmentation model is obtained by adopting the multi-task training method of detection and segmentation;

[0083] According to the point cloud detection and segmentation model, the point cloud data is processed to obtain category information and size information results of the point cloud 3D detection box.

[0084] The point cloud detection and segmentation model obtained through training is used to process the point cloud data to obtain the category information and size information results of the point cloud 3D detection box.

[0085] The present application also provides a multi-task training device 300 for detection and segmentation, such as Figure 3 As shown, a schematic diagram of the structure of a multi-task training device for detection and segmentation in an embodiment of the present application is provided. The device 400 includes at least: a pre-training module 310, a detection module 320, and a training module 330, wherein:

[0086] In one embodiment of the present application, the pre-training module 310 is specifically used to: train the backbone network and the segmentation head module based on the first point cloud data as a pre-training model, and the first point cloud data at least includes the results of the annotated point cloud detection and the results of the point cloud segmentation.

[0087] The multi-task training method in the embodiment of the present application is a necessary step in the model training phase, and the steps in the model verification phase correspond to the multi-task training method. The "first point cloud data" used as training data is annotated with the point cloud detection results (usually 3D point cloud detection boxes) and the point cloud segmentation results (the content represented in the 3D scene is obtained according to each point (xyz)). The "segmentation head module" is a part of the deep learning point cloud segmentation model specifically used to perform point cloud segmentation tasks. It is usually located at the end of the model, immediately after the feature extraction part, and is responsible for converting the feature map into pixel-level classification predictions.

[0088] In actual training, in the first phase, integrated data labeled for both detection and segmentation can be used to train the backbone network and segmentation head module, serving as a pre-trained model. Pre-trained models are an important technique in deep learning. By pre-training on large datasets, they learn common feature representations, providing foundational support for subsequent specific tasks. By learning common features from large-scale data, pre-trained models provide a powerful foundation for subsequent tasks and are a key technology for improving efficiency and performance in deep learning.

[0089] In one embodiment of the present application, the detection module 320 is specifically used to: freeze the backbone network and the segmentation head module, the backbone network adopts the feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on the second point cloud data, and the second point cloud data at least includes the result of the detection box marked.

[0090] During actual training, the segmentation head module is frozen, while the backbone network loads the parameters trained in the first phase. At this point, the detection head module can be trained using data labeled with only the detection boxes. Freezing during model training means setting certain layers or parameters of the model to a non-trainable state during training, meaning that these layers or parameters will not be updated during backpropagation.

[0091] The "second point cloud data" serves as training data, annotated only with 3D detection boxes. The detection head module is trained by combining the point cloud feature output from the backbone network with the feature parameters of the point cloud data from the pre-trained model. The "detection head module" is primarily used for object detection tasks, with its core function being to identify and locate specific objects in an image. Object detection requires not only determining the object's category but also its location within the image. The detection head typically achieves this by predicting a bounding box. This bounding box is defined by the position of the top left corner, as well as its width and height.

[0092] In related technologies, the labeling costs for segmentation data and detection data differ significantly, resulting in inconsistent data volumes. This means there's a larger amount of pure detection data and a smaller amount of data labeled for both detection and segmentation. Therefore, by having the first point cloud data include at least the labeled point cloud detection and segmentation results, and the second point cloud data include at least the labeled detection bounding boxes, this problem can be overcome.

[0093] In one embodiment of the present application, the training module 330 is specifically used to: train a detection and segmentation model based on the backbone network, the segmentation head module and the detection head module, wherein the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0094] The backbone network, the segmentation head module, and the detection head module are used to train a detection and segmentation model. The backbone network extracts point cloud feature data, the segmentation head module segments the point cloud to obtain segmentation results, and the detection head module outputs the category and size information of the point cloud 3D detection box as the final output.

[0095] In one embodiment of the present application, it further includes:

[0096] The point cloud segmentation result in the segmentation head module is used as the fifth point cloud feature;

[0097] The fifth point cloud feature is pooled to obtain a third point cloud feature;

[0098] The first point cloud feature in the pre-trained model is combined with the third point cloud feature to obtain a fourth point cloud feature.

[0099] In one embodiment of the present application, the dimension of the third point cloud feature is [B, C3, H, W], and is the same as the size of the point cloud feature data in the pre-trained model;

[0100] The dimension of the fourth point cloud feature is [B, C1+C3, H, W].

[0101] In one embodiment of the present application, the pre-training module 310 is further configured to:

[0102] The backbone network extracts the overall point cloud features and outputs the first point cloud features. The dimension of the first point cloud features is [B, C1, H, W].

[0103] Among them, B is the batch size of the processed data, C1 is the dimension of the feature, and H and W are the grid length and width of the plane feature.

[0104] In one embodiment of the present application, the segmentation head module is used to

[0105] The dimension of the second point cloud feature obtained by processing the first point cloud feature is [B, C2, N], where N represents the number of point clouds and C2 represents the feature dimension of each point cloud;

[0106] The dimension of the fifth point cloud feature output is [B, Class_num, N], where Class_num represents the number of categories for point cloud segmentation, and N represents the number of classified point clouds.

[0107] In one embodiment of the present application, the detection module 320 is further configured to:

[0108] The detection and segmentation model outputs the geometric information dimension of each detection box as [B, C, M];

[0109] The detection and segmentation model also outputs category information of dimension [B, det_class, M],

[0110] Wherein, C represents the feature dimension of the detection box, M is the number of output detection boxes, and det_class represents the number of detection categories.

[0111] It can be understood that the above-mentioned multi-task training device for detection and segmentation can implement the various steps of the multi-task training method for detection and segmentation provided in the aforementioned embodiments. The relevant explanations on the multi-task training method for detection and segmentation are applicable to the multi-task training device for detection and segmentation and will not be repeated here.

[0112] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 4 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0113] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0114] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0115] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming a multi-task training device for detection and segmentation at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0116] Training a backbone network and a segmentation head module as a pre-trained model based on first point cloud data, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results;

[0117] Freezing the segmentation head module, and using the backbone network to train the detection head module based on the second point cloud data using the feature parameters of the point cloud data in the pre-trained model, wherein the second point cloud data at least includes a result marked with a detection box;

[0118] According to the backbone network, the segmentation head module and the detection head module, a detection and segmentation model is trained.

[0119] Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0120] The above application Figure 2 The methods performed by the multi-task training device for detection and segmentation disclosed in the illustrated embodiments can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be performed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0121] The electronic device may also perform Figure 2 Method for executing a multi-task training device for detection and segmentation, and realizing a multi-task training device for detection and segmentation in Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0122] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 2 The method performed by the multi-task training device for detection and segmentation in the illustrated embodiment is specifically used to perform:

[0123] Training a backbone network and a segmentation head module as a pre-trained model based on first point cloud data, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results;

[0124] Freezing the segmentation head module, and using the backbone network to train the detection head module based on the second point cloud data using the feature parameters of the point cloud data in the pre-trained model, wherein the second point cloud data at least includes a result marked with a detection box;

[0125] According to the backbone network, the segmentation head module and the detection head module, a detection and segmentation model is trained.

[0126] Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

[0127] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0132] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0133] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A multi-task training method for detection and segmentation, where: The training method comprises: Training a backbone network and a segmentation head module as a pre-trained model based on first point cloud data, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results; Freezing the backbone network and the segmentation head module, wherein the backbone network uses feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on second point cloud data, wherein the second point cloud data at least includes a result marked with a detection box; According to the backbone network, the segmentation head module and the detection head module, a detection and segmentation model is trained. Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

2. The method of claim 1, further comprising: The point cloud segmentation result in the segmentation head module is used as the fifth point cloud feature; The fifth point cloud feature is pooled to obtain a third point cloud feature; The first point cloud feature in the pre-trained model is combined with the third point cloud feature to obtain a fourth point cloud feature.

3. The method according to claim 2, wherein: The dimension of the third point cloud feature is [B, C3, H, W], and is the same as the size of the point cloud feature data in the pre-trained model; The dimension of the fourth point cloud feature is [B, C1+C3, H, W].

4. The method according to claim 1, wherein: The training of the backbone network and the segmentation head module according to the first point cloud data as a pre-training model includes: The backbone network extracts the overall point cloud features and outputs the first point cloud features. The dimension of the first point cloud features is [B, C1, H, W]. Among them, B is the batch size of the processed data, C1 is the dimension of the feature, and H and W are the grid length and width of the plane feature.

5. The method according to claim 2, wherein: The segmentation head module is used to The dimension of the second point cloud feature obtained by processing the first point cloud feature is [B, C2, N], where N represents the number of point clouds and C2 represents the feature dimension of each point cloud; The dimension of the fifth point cloud feature output is [B, Class_num, N], where Class_num represents the number of categories for point cloud segmentation, and N represents the number of classified point clouds.

6. The method of claim 1, wherein: Freezing the backbone network and the segmentation head module; the backbone network adopts the feature parameters of the point cloud data in the pre-trained model and trains the detection head module according to the second point cloud data, including: The detection and segmentation model outputs the geometric information dimension of each detection box as [B, C, M]; The detection and segmentation model also outputs category information of dimension [B, det_class, M], Wherein, C represents the feature dimension of the detection box, M is the number of output detection boxes, and det_class represents the number of detection categories.

7. A point cloud data processing method, wherein: The processing method comprises: A point cloud detection and segmentation model is obtained by adopting the multi-task training method for detection and segmentation according to any one of claims 1 to 6; According to the point cloud detection and segmentation model, the point cloud data is processed to obtain category information and size information results of the point cloud 3D detection box.

8. A multi-task training device for detection and segmentation, wherein: The device comprises: A pre-training module, configured to train a backbone network and a segmentation head module based on first point cloud data as a pre-training model, wherein the first point cloud data at least includes annotated point cloud detection results and point cloud segmentation results; a detection module, configured to freeze the backbone network and the segmentation head module, wherein the backbone network uses feature parameters of the point cloud data in the pre-trained model and trains the detection head module based on second point cloud data, wherein the second point cloud data at least includes a result marked with a detection box; A training module is used to train a detection and segmentation model based on the backbone network, the segmentation head module and the detection head module. Among them, the backbone network is used to extract point cloud feature data, the segmentation head module is used to obtain point cloud segmentation results, and the detection head module is used to output category information and size information of the point cloud 3D detection box.

9. An electronic device comprising: processor; as well as A memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 6 and / or the method of claim 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device comprising a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 6 and / or the method of claim 7.

Images