Model training, point cloud data processing method and device, equipment and storage medium

Abstract

Embodiments of the present application provide a model training, point cloud data processing method and device, equipment and storage medium, relating to the technical field of data processing, the above-mentioned method comprises: obtaining sample point cloud data and a plurality of task annotation information; input the sample point cloud data into a feature extraction layer to obtain a sample feature map; input the sample feature map into each task processing layer respectively to obtain a task processing result processed by each task processing layer; calculate a training loss of the task processing layer according to the task processing result and the task annotation information; adjust the model parameters of the to-be-trained model according to the training loss corresponding to each task processing layer, realize the training of the to-be-trained model, and obtain a point cloud data processing model. The model trained according to the model training scheme provided by the embodiments of the present application can realize multi-task processing of point cloud data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for model training and point cloud data processing. Background Technology

[0002] In autonomous driving scenarios, the data processing equipment installed on the vehicle can obtain point cloud data of the surrounding environment collected by the lidar installed on the vehicle. In order to accurately plan the vehicle's travel path, monitor road traffic conditions, and perform other operations, the point cloud data needs to be processed for various tasks.

[0003] In view of the above, there is a need to provide a point cloud data processing solution for multiple tasks, so as to realize multi-task processing of point cloud data. Summary of the Invention

[0004] The purpose of this invention is to provide a model training and point cloud data processing method, apparatus, device, and storage medium to achieve multi-task processing of point cloud data. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide a model training method, the method comprising:

[0006] Obtain sample point cloud data and various task annotation information of the sample point cloud data;

[0007] The sample point cloud data is input into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers.

[0008] The sample feature maps are input into each task processing layer to obtain the task processing results of each task processing layer based on the sample feature maps.

[0009] For each task processing layer, the training loss of the task processing layer is calculated based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer.

[0010] The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer to achieve training of the model to be trained and obtain the point cloud data processing model.

[0011] In one embodiment of the present invention, the step of inputting the sample point cloud data into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data includes:

[0012] The sample point cloud data is input into the feature extraction layer of the model to be trained, and the sample feature map is obtained after the feature extraction layer performs the following processing on the sample point cloud data:

[0013] The sample point cloud data is divided into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data. The division direction is determined according to the parameters of the lidar that collects the sample point cloud data.

[0014] Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number;

[0015] Feature extraction is performed on the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region.

[0016] Based on the position of each effective point column region in the sample point cloud data and the corresponding point cloud features, a pseudo image is obtained. The width and height dimensions of the pseudo image correspond to the dimensions of the two division directions, respectively. The image channel dimension of the pseudo image corresponds to the depth dimension of the point cloud features. The pixel value of each pixel in the pseudo image is the feature value at the position corresponding to the pixel in the point cloud features.

[0017] Feature extraction is performed on the pseudo-image to obtain the feature map of the pseudo-image, which is used as the sample feature map of the sample point cloud data.

[0018] In one embodiment of the present invention, the model to be trained includes at least two of the following task processing layers:

[0019] Task processing layers for object recognition, object classification, object velocity prediction, semantic segmentation, lane detection, and instance segmentation.

[0020] In one embodiment of the present invention, when the model to be trained includes a first task processing layer for implementing an object recognition task, the step of inputting the sample feature map into each task processing layer respectively to obtain the task processing result of each task processing layer performing task processing based on the sample feature map includes:

[0021] The sample feature map is input into the first task processing layer, and the task processing result is output by the first task processing layer after performing the following processing on the sample feature map:

[0022] According to the preset feature transformation coefficients corresponding to various preset object types, the sample feature map is transformed to obtain the transformed classification feature map corresponding to various preset object types.

[0023] For each object type, the center point of the object in the sample point cloud data is determined based on the position of the pixel point with a pixel value greater than a preset threshold in the classification feature map corresponding to that object type.

[0024] In one embodiment of the present invention, when the model to be trained includes a second task processing layer for implementing the object velocity prediction task, the method further includes:

[0025] Each classification feature map is input into the second task processing layer to obtain the motion velocity of each collected object corresponding to the sample point cloud data at the time of sample point cloud data acquisition, as predicted by the second task processing layer based on each classification feature map.

[0026] In one embodiment of the present invention, when the model to be trained includes a third task processing layer for implementing an object classification task, the step of inputting the sample feature map into each task processing layer respectively to obtain the task processing result of each task processing layer based on the sample feature map includes:

[0027] The sample feature map is input into the third task processing layer, and the task processing result is output by the third task processing layer after the sample feature map is processed as follows:

[0028] For each valid point column region, the information of each data point in the valid point column region and the feature in the sample feature map corresponding to the valid point column region are superimposed to obtain the superimposed information of each data point in the valid point column region.

[0029] For each data point, the superposition information of the data point is multiplied by a preset transformation matrix to obtain the calculation result. Based on the calculation result, the object type corresponding to the data point is determined from multiple preset object types.

[0030] In one embodiment of the present invention, adjusting the model parameters of the model to be trained according to the training loss corresponding to each task processing layer includes:

[0031] For each task processing layer, the weight coefficient of the training loss corresponding to the task processing layer is determined based on the training loss corresponding to the task processing layer in multiple consecutive completed training iterations, and the calculated training loss corresponding to the task processing layer is weighted.

[0032] The model parameters of the model to be trained are adjusted according to the training loss corresponding to each weighted task processing layer.

[0033] Secondly, embodiments of the present invention also provide a point cloud data processing method, the method comprising:

[0034] Obtain the target point cloud data to be processed;

[0035] The target point cloud data is input into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training method described in the first aspect above. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. One task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

[0036] Thirdly, embodiments of the present invention also provide a model training apparatus, the apparatus comprising:

[0037] The sample acquisition module is used to acquire sample point cloud data and various task annotation information of the sample point cloud data.

[0038] The feature extraction module is used to input the sample point cloud data into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers.

[0039] The task processing module is used to input the sample feature map into each task processing layer respectively, and obtain the task processing results of each task processing layer based on the sample feature map.

[0040] The loss calculation module is used to calculate the training loss of each task processing layer based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer.

[0041] The parameter adjustment module is used to adjust the model parameters of the model to be trained according to the training loss corresponding to each task processing layer, so as to train the model to be trained and obtain the point cloud data processing model.

[0042] In one embodiment of the present invention, the feature extraction module is specifically used for:

[0043] The sample point cloud data is input into the feature extraction layer of the model to be trained, and the sample feature map is obtained after the feature extraction layer performs the following processing on the sample point cloud data:

[0044] The sample point cloud data is divided into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data. The division direction is determined according to the parameters of the lidar that collects the sample point cloud data.

[0045] Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number;

[0046] Feature extraction is performed on the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region.

[0047] Based on the position of each effective point column region in the sample point cloud data and the corresponding point cloud features, a pseudo image is obtained. The width and height dimensions of the pseudo image correspond to the dimensions of the two division directions, respectively. The image channel dimension of the pseudo image corresponds to the depth dimension of the point cloud features. The pixel value of each pixel in the pseudo image is the feature value at the position corresponding to the pixel in the point cloud features.

[0048] Feature extraction is performed on the pseudo-image to obtain the feature map of the pseudo-image, which is used as the sample feature map of the sample point cloud data.

[0049] In one embodiment of the present invention, the model to be trained includes at least two of the following task processing layers:

[0050] Task processing layers for object recognition, object classification, object velocity prediction, semantic segmentation, lane detection, and instance segmentation.

[0051] In one embodiment of the present invention, when the model to be trained includes a first task processing layer for implementing an object recognition task, the task processing module is specifically used for:

[0052] The sample feature map is input into the first task processing layer, and the task processing result is output by the first task processing layer after performing the following processing on the sample feature map:

[0053] According to the preset feature transformation coefficients corresponding to various preset object types, the sample feature map is transformed to obtain the transformed classification feature map corresponding to various preset object types.

[0054] For each object type, the center point of the object in the sample point cloud data is determined based on the position of the pixel point with a pixel value greater than a preset threshold in the classification feature map corresponding to that object type.

[0055] In one embodiment of the present invention, when the model to be trained includes a second task processing layer for implementing the object velocity prediction task, the apparatus further includes:

[0056] The velocity prediction module is used to input each classification feature map into the second task processing layer to obtain the motion velocity of each acquisition object corresponding to the sample point cloud data at the time of acquisition of the sample point cloud data, which is predicted by the second task processing layer based on each classification feature map.

[0057] In one embodiment of the present invention, when the model to be trained includes a third task processing layer for implementing an object classification task, the task processing module is specifically used for:

[0058] The sample feature map is input into the third task processing layer, and the task processing result is output by the third task processing layer after the sample feature map is processed as follows:

[0059] For each valid point column region, the information of each data point in the valid point column region and the feature in the sample feature map corresponding to the valid point column region are superimposed to obtain the superimposed information of each data point in the valid point column region.

[0060] For each data point, the superposition information of the data point is multiplied by a preset transformation matrix to obtain the calculation result. Based on the calculation result, the object type corresponding to the data point is determined from multiple preset object types.

[0061] In one embodiment of the present invention, the parameter adjustment module is specifically used for:

[0062] For each task processing layer, the weight coefficient of the training loss corresponding to the task processing layer is determined based on the training loss corresponding to the task processing layer in multiple consecutive completed training iterations, and the calculated training loss corresponding to the task processing layer is weighted.

[0063] The model parameters of the model to be trained are adjusted according to the training loss corresponding to each weighted task processing layer.

[0064] Fourthly, embodiments of the present invention also provide a point cloud data processing apparatus, the apparatus comprising:

[0065] The target acquisition module is used to acquire the target point cloud data to be processed;

[0066] The point cloud processing module is used to input the target point cloud data into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training method described in the first aspect above. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. Each task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

[0067] Fifthly, embodiments of the present invention also provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0068] Memory, used to store computer programs;

[0069] When a processor executes a program stored in memory, it implements the steps of the method described in the first or second aspect above.

[0070] In a sixth aspect, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described in the first or second aspect above.

[0071] Beneficial effects of the embodiments of the present invention:

[0072] As can be seen from the above, when training a model using the scheme provided in this embodiment of the invention, the model to be trained includes a feature extraction layer and multiple task processing layers. Each task processing layer can perform task processing based on the features extracted by the feature extraction layer. Thus, during model training, each task processing layer can output a task processing result. Based on the task processing results output by each task processing layer and the task annotation information corresponding to the tasks implemented by each task processing layer, the training loss for each task processing layer is calculated. The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer. This enables the model to learn the features of the tasks implemented by each task processing layer. Therefore, after model training, the trained point cloud data processing model can accurately perform multi-task processing on the point cloud data input to the model based on the learned features of each task. Therefore, by applying the model training scheme provided in this embodiment of the invention, the trained point cloud data model can not only perform multi-task processing on point cloud data but also improve the accuracy of multi-task processing of point cloud data. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0074] Figure 1a A flowchart illustrating the first model training method provided in an embodiment of the present invention;

[0075] Figure 1b This is a schematic diagram of the structure of the first type of training model provided in an embodiment of the present invention;

[0076] Figure 2 A flowchart illustrating the second model training method provided in this embodiment of the invention;

[0077] Figure 3 A flowchart illustrating the third model training method provided in this embodiment of the invention;

[0078] Figure 4 This is a schematic diagram of the structure of the second type of training model provided in an embodiment of the present invention;

[0079] Figure 5 A flowchart illustrating a point cloud data processing method provided in an embodiment of the present invention;

[0080] Figure 6 This is a schematic diagram of the structure of the first model training device provided in an embodiment of the present invention;

[0081] Figure 7 This is a schematic diagram of the structure of the second model training device provided in an embodiment of the present invention;

[0082] Figure 8 This is a schematic diagram of the structure of a point cloud data processing device provided in an embodiment of the present invention;

[0083] Figure 9 This is a flowchart illustrating a model training method provided in an embodiment of the present invention. Detailed Implementation

[0084] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0085] See Figure 1a , Figure 1aThis is a flowchart illustrating the first model training method provided in an embodiment of the present invention. In this embodiment, the method includes the following steps S101-S105.

[0086] Step S101: Obtain sample point cloud data and various task annotation information for the sample point cloud data.

[0087] The aforementioned sample point cloud data is used to train the model to be trained.

[0088] Each type of task annotation information in the above sample point cloud data corresponds to a specific task. Different tasks will have different task annotation information.

[0089] For example, the task annotation information for an object classification task could be the type of object corresponding to a data point in the sample point cloud data, and the task annotation information for an object velocity detection task could be the motion velocity of the object corresponding to a data point in the sample point cloud data.

[0090] Specifically, when obtaining task annotation information for sample point cloud data, the tasks implemented by the task processing layers included in the model to be trained can be known in advance, thereby obtaining task annotation information corresponding to the known tasks.

[0091] Step S102: Input the sample point cloud data into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data.

[0092] The model to be trained also includes multiple task processing layers.

[0093] For a description of the task processing layer described above, please refer to the subsequent step S103, which will not be detailed here.

[0094] Specifically, the feature extraction layer in the model to be trained is used to extract the features of the point cloud data input to the model. In this way, when the sample point cloud data is input into the feature extraction layer, the feature extraction layer can extract the features of the sample point cloud data and obtain the sample feature map of the sample point cloud data.

[0095] Step S103: Input the sample feature maps into each task processing layer to obtain the task processing results of each task processing layer based on the sample feature maps.

[0096] One of the task processing layers is used to perform a task on the point cloud data.

[0097] In one embodiment of the present invention, the model to be trained includes at least two of the following task processing layers:

[0098] Task processing layers for object recognition, object classification, object velocity prediction, semantic segmentation, lane detection, and instance segmentation.

[0099] In this scheme, each task processing layer can perform one task processing on point cloud data. Thus, when the point cloud data processing model is trained using the model training scheme provided in this embodiment of the invention, the task processing layers included in the model can accurately perform their corresponding processing tasks. Furthermore, the point cloud data processing model includes at least two task processing layers, so using the point cloud data processing model can accurately perform multi-task processing on point cloud data.

[0100] In the model to be trained, each task processing layer is connected to the feature extraction layer. After the feature extraction layer outputs the sample feature map of the sample point cloud data, the sample feature map can be input into each task processing layer, so that each task processing layer can perform task processing based on the sample feature map.

[0101] See Figure 1b , Figure 1b This is a schematic diagram of the structure of a model to be trained. Figure 1b The model to be trained includes a feature extraction layer and three task processing layers. These three task processing layers are connected to the feature extraction layer, so that when the feature extraction layer outputs the features of the point cloud data, the extracted features can be input into the three task processing layers respectively.

[0102] Figure 1b The schematic diagram shown is only one case of the model to be trained. In actual application scenarios, the number of task processing layers included in the model to be trained can be two, four or other numbers. This embodiment of the invention does not limit this.

[0103] Step S104: For each task processing layer, calculate the training loss of the task processing layer based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer.

[0104] Based on the task processing results output by the task processing layer and the task labeling information corresponding to the tasks implemented by the task processing layer, the training loss of the task processing layer for task processing can be calculated using existing loss calculation techniques.

[0105] For example, for a task processing layer that performs an object velocity prediction task, the task processing result output by the task processing layer is the object velocity of the collected object predicted by the model. The collected object is the object collected from the sample point cloud data. The task annotation information corresponding to the object velocity prediction task performed by the task processing layer is the actual velocity of the collected object. Based on the object velocity predicted by the model and the actual velocity, the training loss of the task processing layer for task processing can be calculated using existing loss calculation techniques.

[0106] Step S105: Adjust the model parameters of the model to be trained according to the training loss corresponding to each task processing layer, so as to train the model to be trained and obtain the point cloud data processing model.

[0107] Specifically, after obtaining the training loss corresponding to each task processing layer, the model parameters of the model to be trained can be adjusted according to each training loss, and it can be determined whether the model meets the training termination condition. If it does not meet the condition, the process returns to step S101 above, obtains the sample point cloud data and task annotation information again, and continues to train the model to be trained until the model meets the training termination condition, thus completing the training of the model to be trained and obtaining the point cloud data processing model.

[0108] The training termination condition mentioned above can be that the number of training iterations reaches a preset number, the training termination condition can also be that the calculated training loss corresponding to each task processing layer is less than a preset threshold, or other conditions used to terminate training, which will not be detailed here.

[0109] In one embodiment of the present invention, when adjusting the model parameters of the model to be trained according to each training loss, the adjustment can be carried out in any of the following two ways.

[0110] In the first implementation, the model parameters of the model to be trained can be adjusted multiple times according to each training loss.

[0111] In the second implementation, the total training loss of the model to be trained can be calculated based on the obtained training losses, and the model parameters of the model to be trained can be adjusted according to the total training loss.

[0112] For example, the individual training losses can be added together to obtain the total training loss, which can be used as the total training loss.

[0113] In addition, when adjusting the model parameters of the model to be trained according to each training loss, the weight coefficients of the training loss corresponding to each task processing layer can be set, so as to adjust the model parameters of the model to be trained according to the weighted training loss corresponding to each task processing layer.

[0114] In one embodiment of the present invention, the weight coefficients of the training loss can be set by either of the following two implementation methods.

[0115] In the first implementation method, it can be achieved through subsequent... Figure 2 In the illustrated embodiment, step S105A sets the weight coefficients of the training loss, which will not be described in detail here.

[0116] In the second implementation, the user's accuracy requirements for the tasks implemented by each task processing layer can be known in advance. For task processing layers where the user requires higher accuracy, the weight coefficient of the training loss corresponding to that task processing layer can be set to a larger weight coefficient. For task processing layers where the user requires lower accuracy, the weight coefficient of the training loss corresponding to that task processing layer can be set to a smaller weight coefficient.

[0117] As can be seen from the above, when training a model using the scheme provided in this embodiment of the invention, the model to be trained includes a feature extraction layer and multiple task processing layers. Each task processing layer can perform task processing based on the features extracted by the feature extraction layer. Thus, during model training, each task processing layer can output a task processing result. Based on the task processing results output by each task processing layer and the task annotation information corresponding to the tasks implemented by each task processing layer, the training loss for each task processing layer is calculated. The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer. This enables the model to learn the features of the tasks implemented by each task processing layer. Therefore, after model training, the trained point cloud data processing model can accurately perform multi-task processing on the point cloud data input to the model based on the learned features of each task. Therefore, by applying the model training scheme provided in this embodiment of the invention, the trained point cloud data model can not only perform multi-task processing on point cloud data but also improve the accuracy of multi-task processing of point cloud data.

[0118] During model training, due to differences in the difficulty of different tasks and the varying amounts of data used to train different task processing layers in the model, the training losses for different task processing layers in the model to be trained can vary significantly. For example, the model may learn tasks quickly for some task processing layers and slowly for others. Here, tasks learned quickly by the model are called simple tasks, and tasks learned slowly are called difficult tasks. When adjusting model parameters according to the training loss, a larger weight coefficient can be assigned to the training loss of the task processing layer that implements difficult tasks, and a smaller weight coefficient can be assigned to the training loss of the task processing layer that implements simple tasks.

[0119] In view of the above, in one embodiment of the present invention, see... Figure 2 The flowchart of the second model training method is provided. In this embodiment, the above step S105 can be achieved through the following steps S105A-S105B.

[0120] Step S105A: For each task processing layer, determine the weight coefficient of the training loss corresponding to the task processing layer based on the training loss corresponding to the task processing layer in multiple consecutive completed training iterations, and perform weighted calculation on the calculated training loss corresponding to the task processing layer.

[0121] Specifically, for each task processing layer, the weight coefficients of the training loss corresponding to that task processing layer can be determined by any of the following two implementation methods.

[0122] In the first implementation, the training loss corresponding to the task processing layer can be obtained during multiple consecutive completed training iterations. Based on the obtained training loss, the trend of training loss change can be known, and the weight coefficient of the training loss corresponding to the task processing layer can be determined based on the trend of training loss change.

[0123] For example, if the training loss corresponding to the task processing layer converges quickly during multiple training iterations, it indicates that the task implemented by the task processing layer is a simple task, and the model has high learning efficiency for the task implemented by the task processing layer. In this case, the weight coefficient of the training loss corresponding to the task processing layer can be set to a smaller weight coefficient. If the training loss corresponding to the task processing layer converges slowly during multiple training iterations, it indicates that the task implemented by the task processing layer is a difficult task, and the model has low learning efficiency for the task implemented by the task processing layer. In this case, the weight coefficient of the training loss corresponding to the task processing layer can be set to a larger weight coefficient.

[0124] In the second implementation method, the weight coefficient of the training loss can be determined by combining the trend of the training loss and the magnitude of the training loss.

[0125] Furthermore, the weight coefficients of the training loss corresponding to each of the above task processing layers can be adjusted in real time. During training, the training loss from the most recent completed training iterations can be obtained in real time, and the weight coefficients can be determined based on the obtained training loss.

[0126] For example, for difficult tasks, if the training loss of the task processing layer for the difficult task converges slowly, a larger weight coefficient can be set. As the model to be trained continues to be trained, when the training loss of the task processing layer for the difficult task converges faster, the set weight coefficient can be reduced.

[0127] Step S105B: Adjust the model parameters of the model to be trained according to the training loss corresponding to each task processing layer after weighting, so as to train the model to be trained and obtain the point cloud data processing model.

[0128] This step is the same as adjusting the model parameters of the model to be trained in step S105 above, and will not be repeated here.

[0129] As can be seen from the above, when training a model using the scheme provided in the embodiments of the present invention, the training loss in multiple completed training iterations can reflect the difficulty of the tasks implemented by each task processing layer in the model. Therefore, based on the above multiple training losses, the weight coefficients of the training loss corresponding to the task processing layer can be accurately determined. In this way, adjusting the model parameters of the model to be trained according to the weighted training loss can accelerate the model training efficiency.

[0130] The following section explains the specific implementation method of extracting sample feature maps from sample point cloud data using the feature extraction layer.

[0131] In one embodiment of the present invention, after the sample point cloud data is input into the feature extraction layer of the model to be trained, the feature extraction layer can perform the following step AE to obtain the sample feature map of the sample point cloud data.

[0132] Step A: Divide the sample point cloud data into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data.

[0133] The division direction is determined based on the parameters of the lidar used to collect the point cloud data of the samples.

[0134] When LiDAR collects sample point cloud data, it usually needs to pre-construct the point cloud space. This way, when the LiDAR receives laser information reflected back from the outside, it generates various data points in the point cloud space based on the received laser information.

[0135] The aforementioned point cloud space can be understood as the space determined according to the parameters of the LiDAR, and the aforementioned division direction can be understood as the direction in which the point cloud space is divided. When determining the division direction, the aforementioned point cloud space is a three-dimensional space, thus two of the dimensions of the point cloud space can be determined as two division directions.

[0136] For example, the point cloud space mentioned above can be represented as a spatial coordinate system, and the division direction mentioned above can be the two coordinate axis directions of this spatial coordinate system.

[0137] Specifically, since the above-mentioned sample point cloud data is usually point cloud data within a certain spatial range, when dividing the sample point cloud data into regions along the division direction, the length of the sample point cloud data in the dimensions corresponding to the two division directions can be obtained. Based on the obtained lengths of the two dimensions, the division distance in these two dimensions is determined. Thus, based on the determined division distance, the sample point cloud data is divided into regions to obtain multiple point column regions after division.

[0138] Step B: Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number.

[0139] The aforementioned preset quantity can be any quantity set manually.

[0140] Specifically, after dividing the sample point cloud data into multiple point column regions, the number of data points included in each point column region can be determined, thereby finding a valid point column region in each point column region where the number of data points in the region is greater than a preset number.

[0141] When obtaining the effective point column region, in addition to using the methods provided in steps AB above, the sample point cloud data can also be processed using existing point cloud point columnization processing technology to obtain the effective point column region within the sample point cloud data.

[0142] Step C: Extract features from the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region.

[0143] Specifically, existing point cloud data feature extraction techniques can be used to extract features from the point cloud data of each effective point column region.

[0144] For example, point cloud data of the effective point column regions can be input into an existing PointNet network to obtain the point cloud features of the network's output point cloud data. Using the PointNet network, global features of the effective point column regions within the sample point cloud data can be extracted. The specific implementation method of using the PointNet network to extract point cloud features will not be detailed here.

[0145] Step D: Based on the position of each valid point column region in the sample point cloud data and the corresponding point cloud features, obtain the pseudo image.

[0146] In this context, the width and height dimensions of the pseudo-image correspond to the dimensions of the two division directions mentioned above, the image channel dimension of the pseudo-image corresponds to the depth dimension of the point cloud feature, and the pixel value of each pixel in the pseudo-image is the feature value at the position corresponding to the pixel in the point cloud feature.

[0147] The position of the effective point column region in the sample point cloud data can be represented as the position of the effective point column region in the multiple point column regions after division.

[0148] For example, if the sample point cloud data is divided twice along the length dimension of the point cloud space and three times along the width dimension of the point cloud space when dividing the point column regions, then 3*4=12 point column regions can be obtained. If the position information of the effective point column region is (1,2), it means that the effective point column region is the first point column region in the length dimension and the second point column region in the width dimension.

[0149] The size of the point cloud feature corresponding to each point column region can be 1*1*c, where c is the number of dimension values ​​in the depth dimension of the point cloud feature.

[0150] Specifically, after determining the effective point column region, the position of the effective point column region in the sample image can be obtained, and multiple pixels in the pseudo image at the same position as the effective point column region can be determined. After obtaining the point cloud features of the point cloud data of the effective point column region, the pixel values ​​of the determined multiple pixels can be assigned to the various feature values ​​included in the point cloud features, and the pixel values ​​of the unassigned pixels in the pseudo image can be set to 0, thus obtaining the pseudo image.

[0151] In the example above, if the location information of the effective point column region is (1, 2), then multiple pixels in the pseudo image with a width dimension value of 1, a height dimension value of 2, and an image channel dimension value of any dimension can be determined as pixels at the same position as the effective point column region. At this time, the c feature values ​​of the point cloud feature corresponding to the effective point column region are used as the pixel values ​​of the determined multiple pixels.

[0152] Since the point column regions are obtained by dividing the sample point cloud data along two preset division directions, when extracting features from the point cloud data of the effective point column regions and obtaining pseudo images based on the position of each effective region and the corresponding point cloud features, it is possible to convert the sample point cloud data into pseudo images without losing information in the third dimension other than the dimensions of the two division directions mentioned above.

[0153] Step E: Extract features from the pseudo-image to obtain the feature map of the pseudo-image, which serves as the sample feature map of the sample point cloud data.

[0154] Specifically, after obtaining the pseudo-image, its features can be extracted using existing feature extraction techniques, and these features can be used as the sample feature map of the sample point cloud data. The implementation method for extracting the features of the pseudo-image will not be detailed here.

[0155] For example, ResNet-18 or ResNet-50 backbone networks can be used to extract features from pseudo-images. Using ResNet networks for feature extraction from pseudo-images not only extracts semantic information from features from shallow to deep, but also prevents gradient explosion and other issues during model training, thereby accelerating model convergence.

[0156] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the feature extraction layer extracts features from the point cloud data of the effective point column region to obtain the point cloud features corresponding to the effective point column region. Since the effective point cloud region is a point column region with more than a preset number of data points in the region, the point cloud data of the effective point cloud region includes rich data point information. Thus, the extracted point cloud features are more accurate. Furthermore, the point cloud features corresponding to each effective point column region retain all information in the third dimension other than the dimensions of the two division directions mentioned above. In this way, based on the position of each effective point column region and the corresponding point cloud features, pseudo images can be accurately obtained. Feature extraction is performed on the pseudo images, and the feature map of the pseudo images is used as the sample feature map of the sample point cloud data, which can improve the accuracy of the sample feature map.

[0157] The following section explains the specific implementation methods of different task processing layers based on sample feature maps for task processing.

[0158] In one embodiment of the present invention, when the model to be trained includes a first task processing layer for implementing an object recognition task, after the sample feature map is input into the first task processing layer, the first task processing layer can perform the following steps FG to obtain the task processing result.

[0159] Step F: Perform feature transformation on the sample feature map according to the preset feature transformation coefficients corresponding to various preset object types to obtain the transformed classification feature map corresponding to various preset object types.

[0160] Since the object recognition task described above is usually a recognition task for a specific object type, the preset object type can be the type of object that the object recognition task expects to recognize.

[0161] For example, the above object types could be roads, vehicles, roadblocks, etc.

[0162] The initial values ​​of the above transformation coefficients can be set manually. During model training, the transformation coefficients can be continuously adjusted according to the training loss.

[0163] Feature transformation of a sample feature map can be understood as compressing the features in the image channel dimension of the sample feature map according to preset feature transformation coefficients to obtain two-dimensional classification feature maps corresponding to various preset object types.

[0164] For example, if there are five preset object types, and the size of the above sample feature map is 15*15*128, by performing feature transformation on the sample feature map, the sample feature map can be converted into five classification feature maps with a size of 15*15, and each classification feature map corresponds to one object type.

[0165] Specifically, feature transformation of the sample feature map can be performed using either of the following two methods.

[0166] In the first implementation, each object type can correspond to a feature transformation coefficient. In this way, the sample feature map can be transformed multiple times according to each feature transformation coefficient to obtain the classification feature map corresponding to each object type.

[0167] In the example above, the five object types can correspond to five feature transformation coefficients of size 128*1. By performing five feature transformations on the sample feature map according to these five feature transformation coefficients, the classification feature maps corresponding to the five object types are obtained.

[0168] In the second implementation, each object type can correspond to a feature transformation coefficient. By performing feature transformation on the sample feature map according to the feature transformation coefficient, the sample feature map can be compressed into a three-dimensional feature map with the number of dimension values ​​in the image channel dimension equal to the number of object types. The two-dimensional feature map corresponding to each dimension value in the image channel dimension of this three-dimensional feature map is the classification feature map corresponding to each object type.

[0169] In the example above, the five object types can correspond to a feature transformation coefficient of size 128*5. In this way, the sample feature map is transformed according to the feature transformation coefficient, and a feature map of size 15*15*5 is obtained. The two-dimensional feature map corresponding to each dimension value of the image channel dimension in this feature map is the classification feature map corresponding to the five object types.

[0170] Step G: For each object type, determine the object center point in the sample point cloud data based on the positions of pixels with pixel values ​​greater than a preset threshold in the classification feature map corresponding to that object type.

[0171] The aforementioned preset threshold can be a pixel value set manually.

[0172] In the classification feature map, the pixel value of a pixel is the feature value at that pixel location.

[0173] The pseudo-image is obtained based on the position of each effective point column region in the sample point cloud data and the point cloud features. Each pixel in the pseudo-image corresponds to a point column region in the width and height dimensions. The sample feature map is the feature map of the pseudo-image. The sample feature map and the pseudo-image can have the same dimensions in the width and height dimensions. Each feature in the sample feature map corresponds to a pixel in the pseudo-image in these two dimensions. The feature transformation only compresses the features in the image channel dimension of the sample feature map; the other two dimensions remain unchanged. The classification feature map corresponding to the object type has the same dimensions in the width and height dimensions as the sample feature map in these two dimensions, and the features in these two feature maps correspond to each other in these two dimensions. Therefore, each feature in the classification feature map corresponding to the object type corresponds to a point column region in the width and height dimensions.

[0174] For example, if the sample point cloud data is divided into w1 regions along its length and h1 regions along its width, the resulting pseudo-image will have a size of w1*h1*c, where c is the number of depth dimensions of the point cloud features. Extracting features from the pseudo-image yields a sample feature map of size w1*h1*c1, where c1 is the number of depth dimensions. Performing feature transformation on the sample feature map yields a classification feature map of size w1*h1.

[0175] Specifically, for each object type, pixels with values ​​greater than a preset threshold can be identified in the classification feature map corresponding to that object type. Based on the correspondence between pixels and dotted regions in the classification feature map and the positions of the identified pixels, the dotted region containing the center point of the object for that object type can be determined. Furthermore, when performing feature transformation on the sample feature map, the feature transformation coefficients corresponding to that object type can be adjusted so that the classification feature map for that object type is a dual-channel feature map. Based on the positions of pixels with values ​​greater than a preset threshold in the feature map of each channel, the offset of the object's center point is calculated. Therefore, based on the dotted region containing the determined object's center point and the calculated offset, the object's center point for that object type can be obtained, thus determining the object's position.

[0176] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the first task processing layer is used to implement the object recognition task. The first task processing layer can perform feature transformation on the sample feature map according to the feature transformation coefficients to obtain the classification feature map corresponding to the object type. Based on the positions of pixels in the classification feature map whose pixel values ​​are greater than a preset preset value, the center point of the object in the sample point cloud data is determined. During model training, the model parameters in the first task processing layer can be continuously optimized, thereby improving the accuracy of the first task processing layer in performing feature transformation on the sample feature map and determining the center point of the object in the sample point cloud data. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of achieving the object recognition task using the trained model can be improved.

[0177] In one embodiment of the present invention, see Figure 3 The flowchart of the third model training method is provided. In this embodiment, when the model to be trained includes a second task processing layer for implementing the object velocity prediction task, the above method further includes the following step S106.

[0178] Step S106: Input each classification feature map into the second task processing layer to obtain the motion velocity of each collected object corresponding to the sample point cloud data at the time of sample point cloud data acquisition, which is predicted by the second task processing layer based on each classification feature map, and used as the task processing result of the second task processing layer.

[0179] Specifically, after inputting the classification feature maps into the second task processing layer, the second task processing layer can predict the motion speed of each collected object corresponding to the sample point cloud data at the time of sample point cloud data acquisition based on the classification feature maps and using existing velocity prediction techniques. This will not be elaborated here.

[0180] For example, the second task processing layer can use existing OD (Object Detection) prediction technology to predict the motion speed of the collected objects.

[0181] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the second task processing layer is used to implement the object velocity prediction task. During model training, the model parameters in the second task processing layer can be continuously optimized, thereby improving the accuracy of the second task processing layer in predicting object velocity. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of using the trained model to achieve the object velocity prediction task can be improved.

[0182] In one embodiment of the present invention, when the model to be trained includes a third task processing layer for implementing an object classification task, after the sample feature map is input into the third task processing layer, the third task processing layer can perform the following step HI to obtain the task processing result.

[0183] Step H: For each valid point column region, overlay the information of each data point in the valid point column region with the features in the sample feature map corresponding to the valid point column region to obtain the overlay information of each data point in the valid point column region.

[0184] The information of the aforementioned data points may include their location, color, and intensity in the sample point cloud data.

[0185] For example, the information of a data point can be represented as (x, y, z, intensity), where (x, y, z) are the three-dimensional coordinates of the data point, and intensity is the intensity information of the data point.

[0186] Specifically, according to step G in the embodiment describing the first task processing layer, each feature of the sample feature map in the width and height dimensions corresponds to each pixel of the pseudo-image in these two dimensions. Each pixel of the pseudo-image in the width and height dimensions corresponds to a defined point column region. Therefore, each feature of the sample feature map in the width and height dimensions corresponds to a point column region. Thus, for each valid point column region, a feature of size 1*1*c corresponding to that valid point column region can be determined in the sample feature map. Superimposing the information of each data point in the valid point column region with the determined feature can be understood as concatenating the information of the data point with the determined feature to obtain the superimposed information of the concatenated data points.

[0187] For example, if the information of the data points within the effective point column region is (x, y, z, intensity), then this information is a 4-dimensional information. The feature 1*1*c corresponding to the effective point column region is a c-dimensional information. To superimpose the data point information with the feature, a 4+c-dimensional information can be obtained, which serves as the superimposed information of the data points.

[0188] Step 1: For each data point, perform a matrix multiplication calculation between the superposition information of the data point and the preset transformation matrix to obtain the calculation result. Based on the calculation result, determine the object type corresponding to the data point from multiple preset object types.

[0189] The initial value of the aforementioned preset transformation matrix can be set manually, and during model training, the transformation matrix can be continuously adjusted according to the training loss.

[0190] Specifically, each preset object type can be pre-coded. After performing matrix multiplication on the superimposed information of each data point and the transformation matrix to obtain the calculation result, the object type corresponding to each data point can be determined according to the coding shown in the calculation result.

[0191] For example, if there are three object types k1, k2, and k3, let the encoding of type k1 be (0, 0, 1), the encoding of type k2 be (0, 1, 0), and the encoding of type k3 be (1, 0, 0). After obtaining the 4+c dimensional superposition information of the data points, a matrix multiplication can be performed on the 4+c dimensional superposition information and a transformation matrix of size (4+c, 3) to obtain a 3-dimensional calculation result. If the calculation result is (0, 0, 1), then the object type corresponding to the data point can be determined to be type K1; if the calculation result is (0, 1, 0), then the object type corresponding to the data point can be determined to be type K2; and if the calculation result is (1, 0, 0), then the object type corresponding to the data point can be determined to be type K3.

[0192] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the third task processing layer is used to implement the object classification task. During the model training process, the model parameters in the third task processing layer can be continuously optimized, which can improve the accuracy of the third task processing layer in obtaining the superposition information of each data point within each effective point column region and in performing matrix multiplication calculations. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of achieving the object classification task using the trained model can be improved.

[0193] In one embodiment of the present invention, see Figure 4 It provides a schematic diagram of the structure of the second type of model to be trained. Figure 4 In the model to be trained shown, point cloud data 1 can be called point_valid1. The PP network layer is used to perform point cloud columnarization on point cloud data 1 of size (N, 4), where N represents the number of data points in point cloud data 1, and 4 represents that each data point in point cloud data 1 has 4-dimensional information (3-dimensional coordinates and reflection intensity information). The columnar feature extraction layer, also known as the pillar feature heatmap network layer, is used to extract the point cloud features of the point cloud data in the valid columnar regions. The RPNresnet network layer is used to obtain pseudo-images based on the point cloud features of the point cloud data in each valid columnar region, and to extract features from the pseudo-images to obtain feature maps of size (C, H, W), where C is the depth dimension of the feature map, H is the height dimension of the feature map, and W is the width dimension of the feature map. After obtaining the above feature maps, the feature maps can be input into the semantic segmentation task processing layer and the object detection task processing layer, respectively. The semantic segmentation task processing layer can also be called the segmentHead task processing layer, and the object detection task processing layer can also be called the OD Head task processing layer.

[0194] In the segmentHead task processing layer, the fv network layer is used to extract features corresponding to the positions of each valid point column region from the feature map of (C, H, W), obtaining the features corresponding to each valid point column region. The size of the feature corresponding to each valid point column region is 1*1*C. The point cloud feature overlay layer, also known as the point_features network layer, is used to overlay the features corresponding to each valid point column region with the information of each data point in that valid point column region, obtaining point cloud data 2. Point cloud data 2 can also be called point_valid2, where point_valid2 represents the global point cloud feature obtained after overlaying the data points from each valid point column region. The MLP network layer, an artificial neural network layer, transforms the global features of a point cloud of size (N, 4+C) into features of size (N, NUM_C), where NUM_C can be any pre-set dimension value. The MLP network layer is used to calculate the classification features representing that each data point belongs to a preset classification type based on the information of each data point itself and the features corresponding to the point column region where each data point is located. After feature transformation, max pooling is performed on the features of size (N, NUM_C) in the dimension where NUM_C is located to obtain a processing result of (N, 1). This processing result can be understood as each data point in the point cloud data corresponding to a value, and the object type corresponding to the data point is determined based on this value.

[0195] In the OD Head task processing layer, the feature transformation layer, also known as the CONV network layer, uses feature transformation coefficients of size (192, H, W) to transform the feature map, resulting in a feature map of size (64, H, W). This transformed feature map of size (64, H, W) is the classification feature map mentioned in step F of the aforementioned embodiment. This feature map of size (64, H, W) is then input into the network prediction layer. The network prediction layer, also known as the predict network layer, can make predictions based on the transformed feature map, obtaining the object bounding box (Box), scores, and the class of the object corresponding to the object bounding box. The scores can be understood as the confidence level of the model's predicted object bounding box and class.

[0196] After training the point cloud data processing model using the scheme provided in the above embodiments, the model can be used to perform multi-task processing on the point cloud data.

[0197] In one embodiment of the present invention, see Figure 5 A flowchart of a point cloud data processing method is provided, which includes the following steps S501-S502.

[0198] Step S501: Obtain the target point cloud data to be processed.

[0199] The aforementioned target point cloud data can be selected from a certain range of point cloud data collected by the lidar.

[0200] Specifically, after the lidar acquires point cloud data, point cloud data in a preset area can be extracted from the acquired point cloud data and used as target point cloud data. Alternatively, based on the distribution of data points in the acquired point cloud data, areas with dense data point distribution can be used as target point cloud data.

[0201] Step S502: Input the target point cloud data into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model.

[0202] The point cloud data processing model is trained according to the training scheme provided in the above model training method embodiment. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. One task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

[0203] The descriptions of the feature extraction layer and task processing layer can be found in the above-described model training method embodiments, and will not be repeated here.

[0204] The point cloud data processing model includes a feature extraction layer and multiple task processing layers. After the target point cloud data is input into the point cloud data processing model, the feature extraction layer can extract the feature map of the target point cloud data and input the extracted feature map into each task processing layer. Each task processing layer can perform task processing based on the input feature map, thereby outputting the task processing result.

[0205] As can be seen from the above, when processing point cloud data using the solution provided in this embodiment of the invention, the point cloud data processing model includes a feature extraction layer and multiple task processing layers. After training, the feature extraction layer in the model can learn the characteristics of features applicable to the tasks implemented by each task processing layer. Furthermore, the feature extraction layer can learn the feature relationships between multiple tasks that are difficult to learn under a single task. Each task processing layer can learn the characteristics of the task it implements. Thus, when the target point cloud data is input into the point cloud data processing model, each network layer in the model can accurately process the target point cloud data based on the learned characteristics and feature relationships, thereby improving the accuracy of the task processing results output by the model. Therefore, the point cloud data processing solution provided in this embodiment of the invention can not only achieve multi-task processing of point cloud data but also improve the accuracy of multi-task processing.

[0206] In existing technologies, when processing point cloud data for multiple tasks, features of the point cloud data are typically extracted using a feature extraction method specific to each task. Subsequent processing for that task is then performed based on the extracted features. Different tasks may use different feature extraction methods. However, in this solution, all task processing layers share features extracted by a single feature extraction layer. This eliminates the need to extract features from the point cloud data multiple times during multi-task processing. This approach can ensure multi-task processing of point cloud data and achieve the analysis and representation of complex spatial environments while reducing data computation and time costs.

[0207] Furthermore, the task processing layers in the aforementioned point cloud data processing model are arranged in parallel, and each task processing layer can be considered independent of the others. Therefore, task processing layers can be added or removed from the point cloud data processing model according to actual scenario requirements, thereby achieving the actual scenario requirements based on the new point cloud data processing model. It is evident that applying the point cloud data processing scheme provided in this embodiment of the invention can improve the scalability of multi-task processing of point cloud data.

[0208] Corresponding to the above-described model training method, this embodiment of the invention also provides a model training apparatus.

[0209] In one embodiment of the present invention, see Figure 6 The first model training apparatus is provided, the apparatus comprising:

[0210] The sample acquisition module 601 is used to acquire sample point cloud data and various task annotation information of the sample point cloud data.

[0211] The feature extraction module 602 is used to input the sample point cloud data into the feature extraction layer in the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers.

[0212] Task processing module 603 is used to input the sample feature map into each task processing layer respectively, and obtain the task processing result of each task processing layer based on the sample feature map.

[0213] The loss calculation module 604 is used to calculate the training loss of each task processing layer for task processing based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer.

[0214] The parameter adjustment module 605 is used to adjust the model parameters of the model to be trained according to the training loss corresponding to each task processing layer, so as to train the model to be trained and obtain the point cloud data processing model.

[0215] As can be seen from the above, when training a model using the scheme provided in this embodiment of the invention, the model to be trained includes a feature extraction layer and multiple task processing layers. Each task processing layer can perform task processing based on the features extracted by the feature extraction layer. Thus, during model training, each task processing layer can output a task processing result. Based on the task processing results output by each task processing layer and the task annotation information corresponding to the tasks implemented by each task processing layer, the training loss for each task processing layer is calculated. The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer. This enables the model to learn the features of the tasks implemented by each task processing layer. Therefore, after model training, the trained point cloud data processing model can accurately perform multi-task processing on the point cloud data input to the model based on the learned features of each task. Therefore, by applying the model training scheme provided in this embodiment of the invention, the trained point cloud data model can not only perform multi-task processing on point cloud data but also improve the accuracy of multi-task processing of point cloud data.

[0216] In one embodiment of the present invention, the feature extraction module 602 is specifically used for:

[0217] The sample point cloud data is input into the feature extraction layer of the model to be trained, and the sample feature map is obtained after the feature extraction layer performs the following processing on the sample point cloud data:

[0218] The sample point cloud data is divided into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data. The division direction is determined according to the parameters of the lidar that collects the sample point cloud data.

[0219] Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number;

[0220] Feature extraction is performed on the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region.

[0221] Based on the position of each effective point column region in the sample point cloud data and the corresponding point cloud features, a pseudo image is obtained. The width and height dimensions of the pseudo image correspond to the dimensions of the two division directions, respectively. The image channel dimension of the pseudo image corresponds to the depth dimension of the point cloud features. The pixel value of each pixel in the pseudo image is the feature value at the position corresponding to the pixel in the point cloud features.

[0222] Feature extraction is performed on the pseudo-image to obtain the feature map of the pseudo-image, which is used as the sample feature map of the sample point cloud data.

[0223] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the feature extraction layer extracts features from the point cloud data of the effective point column region to obtain the point cloud features corresponding to the effective point column region. Since the effective point cloud region is a point column region with more than a preset number of data points in the region, the point cloud data of the effective point cloud region includes rich data point information. Thus, the extracted point cloud features are more accurate. Furthermore, the point cloud features corresponding to each effective point column region retain all information in the third dimension other than the dimensions of the two division directions mentioned above. In this way, based on the position of each effective point column region and the corresponding point cloud features, pseudo images can be accurately obtained. Feature extraction is performed on the pseudo images, and the feature map of the pseudo images is used as the sample feature map of the sample point cloud data, which can improve the accuracy of the sample feature map.

[0224] In one embodiment of the present invention, the model to be trained includes at least two of the following task processing layers:

[0225] Task processing layers for object recognition, object classification, object velocity prediction, semantic segmentation, lane detection, and instance segmentation.

[0226] In this scheme, each task processing layer can perform one task processing on point cloud data. Thus, when the point cloud data processing model is trained using the model training scheme provided in this embodiment of the invention, the task processing layers included in the model can accurately perform their corresponding processing tasks. Furthermore, the point cloud data processing model includes at least two task processing layers, so using the point cloud data processing model can accurately perform multi-task processing on point cloud data.

[0227] In one embodiment of the present invention, when the model to be trained includes a first task processing layer for implementing an object recognition task, the task processing module 603 is specifically used for:

[0228] The sample feature map is input into the first task processing layer, and the task processing result is output by the first task processing layer after performing the following processing on the sample feature map:

[0229] According to the preset feature transformation coefficients corresponding to various preset object types, the sample feature map is transformed to obtain the transformed classification feature map corresponding to various preset object types.

[0230] For each object type, the center point of the object in the sample point cloud data is determined based on the position of the pixel point with a pixel value greater than a preset threshold in the classification feature map corresponding to that object type.

[0231] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the first task processing layer is used to implement the object recognition task. The first task processing layer can perform feature transformation on the sample feature map according to the feature transformation coefficients to obtain the classification feature map corresponding to the object type. Based on the positions of pixels in the classification feature map whose pixel values ​​are greater than a preset preset value, the center point of the object in the sample point cloud data is determined. During model training, the model parameters in the first task processing layer can be continuously optimized, thereby improving the accuracy of the first task processing layer in performing feature transformation on the sample feature map and determining the center point of the object in the sample point cloud data. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of achieving the object recognition task using the trained model can be improved.

[0232] In one embodiment of the present invention, see Figure 7 A schematic diagram of a second model training device is provided. In this embodiment, when the model to be trained includes a second task processing layer for implementing the object velocity prediction task, the device further includes:

[0233] The velocity prediction module 606 is used to input each classification feature map into the second task processing layer to obtain the motion velocity of each acquisition object corresponding to the sample point cloud data at the time of acquisition of the sample point cloud data, which is predicted by the second task processing layer based on each classification feature map, and is used as the task processing result of the second task processing layer.

[0234] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the second task processing layer is used to implement the object velocity prediction task. During model training, the model parameters in the second task processing layer can be continuously optimized, thereby improving the accuracy of the second task processing layer in predicting object velocity. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of using the trained model to achieve the object velocity prediction task can be improved.

[0235] In one embodiment of the present invention, when the model to be trained includes a third task processing layer for implementing an object classification task, the task processing module 603 is specifically used for:

[0236] The sample feature map is input into the third task processing layer, and the task processing result is output by the third task processing layer after the sample feature map is processed as follows:

[0237] For each valid point column region, the information of each data point in the valid point column region and the feature in the sample feature map corresponding to the valid point column region are superimposed to obtain the superimposed information of each data point in the valid point column region.

[0238] For each data point, the superposition information of the data point is multiplied by a preset transformation matrix to obtain the calculation result. Based on the calculation result, the object type corresponding to the data point is determined from multiple preset object types.

[0239] As can be seen from the above, when training the model using the scheme provided in this embodiment of the invention, the third task processing layer is used to implement the object classification task. During the model training process, the model parameters in the third task processing layer can be continuously optimized, which can improve the accuracy of the third task processing layer in obtaining the superposition information of each data point within each effective point column region and in performing matrix multiplication calculations. Therefore, by training the model using the scheme provided in this embodiment of the invention, the accuracy of achieving the object classification task using the trained model can be improved.

[0240] In one embodiment of the present invention, the parameter adjustment module 605 is specifically used for:

[0241] For each task processing layer, the weight coefficient of the training loss corresponding to the task processing layer is determined based on the training loss corresponding to the task processing layer in multiple consecutive completed training iterations, and the calculated training loss corresponding to the task processing layer is weighted.

[0242] The model parameters of the model to be trained are adjusted according to the training loss corresponding to each weighted task processing layer.

[0243] As can be seen from the above, when training a model using the scheme provided in the embodiments of the present invention, the training loss in multiple completed training iterations can reflect the difficulty of the tasks implemented by each task processing layer in the model. Therefore, based on the above multiple training losses, the weight coefficients of the training loss corresponding to the task processing layer can be accurately determined. In this way, adjusting the model parameters of the model to be trained according to the weighted training loss can accelerate the model training efficiency.

[0244] Corresponding to the point cloud data processing method described above, this embodiment of the invention also provides a point cloud data processing device.

[0245] In one embodiment of the present invention, see Figure 8 A schematic diagram of a point cloud data processing device is provided, the device comprising:

[0246] The target acquisition module 801 is used to acquire target point cloud data to be processed.

[0247] The point cloud processing module 802 is used to input the target point cloud data into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training scheme provided in the above model training method embodiment. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. Each task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

[0248] As can be seen from the above, when processing point cloud data using the solution provided in this embodiment of the invention, the point cloud data processing model includes a feature extraction layer and multiple task processing layers. After training, the feature extraction layer in the model can learn the characteristics of features applicable to the tasks implemented by each task processing layer. Furthermore, the feature extraction layer can learn the feature relationships between multiple tasks that are difficult to learn under a single task. Each task processing layer can learn the characteristics of the task it implements. Thus, when the target point cloud data is input into the point cloud data processing model, each network layer in the model can accurately process the target point cloud data based on the learned characteristics and feature relationships, thereby improving the accuracy of the task processing results output by the model. Therefore, the point cloud data processing solution provided in this embodiment of the invention can not only achieve multi-task processing of point cloud data but also improve the accuracy of multi-task processing.

[0249] In existing technologies, when processing point cloud data for multiple tasks, features of the point cloud data are typically extracted using a feature extraction method specific to each task. Subsequent processing for that task is then performed based on the extracted features. Different tasks may use different feature extraction methods. However, in this solution, all task processing layers share features extracted by a single feature extraction layer. This eliminates the need to extract features from the point cloud data multiple times during multi-task processing. This approach can ensure multi-task processing of point cloud data and achieve the analysis and representation of complex spatial environments while reducing data computation and time costs.

[0250] Furthermore, the task processing layers in the aforementioned point cloud data processing model are arranged in parallel, and each task processing layer can be considered independent of the others. Therefore, task processing layers can be added or removed from the point cloud data processing model according to actual scenario requirements, thereby achieving the actual scenario requirements based on the new point cloud data processing model. It is evident that applying the point cloud data processing scheme provided in this embodiment of the invention can improve the scalability of multi-task processing of point cloud data.

[0251] This invention also provides an electronic device, such as... Figure 9As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0252] Memory 903 is used to store computer programs;

[0253] When processor 901 executes a program stored in memory 903, it performs the following steps:

[0254] Obtain sample point cloud data and various task annotation information of the sample point cloud data;

[0255] The sample point cloud data is input into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers.

[0256] The sample feature maps are input into each task processing layer to obtain the task processing results of each task processing layer based on the sample feature maps.

[0257] For each task processing layer, the training loss of the task processing layer is calculated based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer.

[0258] The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer to achieve training of the model to be trained and obtain the point cloud data processing model.

[0259] Other schemes for the processor 901 to execute the program stored in the memory 903 to implement model training are the same as those mentioned in the aforementioned model training method embodiments, and will not be repeated here.

[0260] When the processor 901 described above executes the program stored in the memory 903, it can also perform the following steps:

[0261] Obtain the target point cloud data to be processed;

[0262] The target point cloud data is input into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training method described in the aforementioned model training method embodiment. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. One task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

[0263] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0264] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0265] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0266] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0267] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described model training or point cloud data processing methods.

[0268] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the model training or point cloud data processing methods described in the above embodiments.

[0269] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0270] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0271] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, systems, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0272] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that, The method includes: Obtain sample point cloud data and various task annotation information of the sample point cloud data; The sample point cloud data is input into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers. The sample feature maps are input into each task processing layer to obtain the task processing results of each task processing layer based on the sample feature maps. For each task processing layer, the training loss of the task processing layer is calculated based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer. The model parameters of the model to be trained are adjusted according to the training loss corresponding to each task processing layer to achieve training of the model to be trained and obtain the point cloud data processing model. The step of inputting the sample point cloud data into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data includes: The sample point cloud data is input into the feature extraction layer of the model to be trained, and the sample feature map is obtained after the feature extraction layer performs the following processing on the sample point cloud data: The sample point cloud data is divided into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data. The preset division directions are determined according to the parameters of the lidar that collects the sample point cloud data. Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number; Feature extraction is performed on the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region. Based on the position of each effective point column region in the sample point cloud data and the corresponding point cloud features, a pseudo image is obtained. The width and height dimensions of the pseudo image correspond to the dimensions of the two division directions, respectively. The image channel dimension of the pseudo image corresponds to the depth dimension of the point cloud features. The pixel value of each pixel in the pseudo image is the feature value at the position corresponding to the pixel in the point cloud features. Feature extraction is performed on the pseudo-image to obtain the feature map of the pseudo-image, which is used as the sample feature map of the sample point cloud data.

2. The method according to claim 1, characterized in that, The model to be trained includes at least two of the following task processing layers: Task processing layers for object recognition, object classification, object velocity prediction, semantic segmentation, lane detection, and instance segmentation.

3. The method according to claim 1, characterized in that, When the model to be trained includes a first task processing layer for implementing the object recognition task, the step of inputting the sample feature map into each task processing layer respectively to obtain the task processing result of each task processing layer based on the sample feature map includes: The sample feature map is input into the first task processing layer, and the task processing result is output by the first task processing layer after performing the following processing on the sample feature map: According to the preset feature transformation coefficients corresponding to various preset object types, the sample feature map is transformed to obtain the transformed classification feature map corresponding to various preset object types. For each object type, the center point of the object in the sample point cloud data is determined based on the position of the pixel point with a pixel value greater than a preset threshold in the classification feature map corresponding to that object type.

4. The method according to claim 3, characterized in that, When the model to be trained includes a second task processing layer for implementing the object velocity prediction task, the method further includes: Each classification feature map is input into the second task processing layer to obtain the motion velocity of each collected object corresponding to the sample point cloud data at the time of sample point cloud data acquisition, as predicted by the second task processing layer based on each classification feature map.

5. The method according to claim 1, characterized in that, When the model to be trained includes a third task processing layer for implementing an object classification task, the step of inputting the sample feature map into each task processing layer respectively to obtain the task processing results of each task processing layer based on the sample feature map includes: The sample feature map is input into the third task processing layer, and the task processing result is output by the third task processing layer after the sample feature map is processed as follows: For each valid point column region, the information of each data point in the valid point column region and the feature in the sample feature map corresponding to the valid point column region are superimposed to obtain the superimposed information of each data point in the valid point column region. For each data point, the superposition information of the data point is multiplied by a preset transformation matrix to obtain the calculation result. Based on the calculation result, the object type corresponding to the data point is determined from multiple preset object types.

6. The method according to claim 1, characterized in that, The step of adjusting the model parameters of the model to be trained according to the training loss corresponding to each task processing layer includes: For each task processing layer, the weight coefficient of the training loss corresponding to the task processing layer is determined based on the training loss corresponding to the task processing layer in multiple consecutive completed training iterations, and the calculated training loss corresponding to the task processing layer is weighted. The model parameters of the model to be trained are adjusted according to the training loss corresponding to each weighted task processing layer.

7. A point cloud data processing method, characterized in that, The method includes: Obtain the target point cloud data to be processed; The target point cloud data is input into the trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training method of any one of claims 1-6. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. One task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

8. A model training device, characterized in that, The device includes: The sample acquisition module is used to acquire sample point cloud data and various task annotation information of the sample point cloud data. The feature extraction module is used to input the sample point cloud data into the feature extraction layer of the model to be trained to obtain the sample feature map of the sample point cloud data. The model to be trained further includes multiple task processing layers. The task processing module is used to input the sample feature map into each task processing layer respectively, and obtain the task processing results of each task processing layer based on the sample feature map. The loss calculation module is used to calculate the training loss of each task processing layer based on the task processing result output by the task processing layer and the task annotation information corresponding to the task implemented by the task processing layer. The parameter adjustment module is used to adjust the model parameters of the model to be trained according to the training loss corresponding to each task processing layer, so as to train the model to be trained and obtain the point cloud data processing model. The feature extraction module is specifically used for: The sample point cloud data is input into the feature extraction layer of the model to be trained, and the sample feature map is obtained after the feature extraction layer performs the following processing on the sample point cloud data: The sample point cloud data is divided into regions along two preset division directions to obtain multiple point column regions included in the sample point cloud data. The division direction is determined according to the parameters of the lidar that collects the sample point cloud data. Find valid point column regions in the obtained multiple point column regions where the number of data points in the region is greater than the preset number; Feature extraction is performed on the point cloud data of each valid point column region to obtain the point cloud features corresponding to each valid point column region. Based on the position of each effective point column region in the sample point cloud data and the corresponding point cloud features, a pseudo image is obtained. The width and height dimensions of the pseudo image correspond to the dimensions of the two division directions, respectively. The image channel dimension of the pseudo image corresponds to the depth dimension of the point cloud features. The pixel value of each pixel in the pseudo image is the feature value at the position corresponding to the pixel in the point cloud features. Feature extraction is performed on the pseudo-image to obtain the feature map of the pseudo-image, which is used as the sample feature map of the sample point cloud data.

9. A point cloud data processing device, characterized in that, The device includes: The target acquisition module is used to acquire the target point cloud data to be processed; A point cloud processing module is used to input the target point cloud data into a trained point cloud data processing model to obtain the task processing results output by each task processing layer in the point cloud data processing model. The point cloud data processing model is trained according to the training method of any one of claims 1-6. The point cloud data processing model includes a feature extraction layer and multiple task processing layers. A task processing layer is used to perform a processing task on the point cloud data based on the features extracted by the feature extraction layer.

10. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-6 or 7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-6 or 7.

Images