Training method of three-dimensional point cloud processing model and semantic segmentation model and related equipment

Abstract

The application discloses a three-dimensional point cloud processing model and a semantic segmentation model training method and related equipment, and relates to the technical field of three-dimensional point clouds. The three-dimensional point cloud processing model training method comprises the following steps: processing three-dimensional point cloud data by using a three-dimensional point cloud processing model to obtain information of points in the three-dimensional point cloud data; obtaining information of pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; determining alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data by using the information of the points and the information of the pixels; determining classification loss of the three-dimensional point cloud processing model by using classification information of the points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; and training the three-dimensional point cloud processing model by using the alignment constraint loss and the classification loss. The embodiment of the application realizes cross-modal guidance of two-dimensional images to three-dimensional point clouds, and improves the accuracy of model training.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud technology, and in particular to a training method and related equipment for a 3D point cloud processing model and a semantic segmentation model. Background Technology

[0002] 3D point clouds refer to spatial 3D point information acquired by scanning devices such as LiDAR and depth cameras. They typically include spatial 3D position information, color information, and intensity information. 3D point cloud semantic segmentation, which assigns a specific semantic label to each point in the point cloud, is a crucial step in scene understanding and has wide applications in many fields such as robotics, augmented reality, and autonomous driving. Therefore, 3D point cloud semantic segmentation has always been a research hotspot in the field of computer vision.

[0003] In related technologies, weakly supervised point cloud semantic segmentation methods based on scene classification supervision signals typically use the idea of ​​point cloud category activation graphs to generate pseudo-labels for each point. This method generates pseudo-labels based on point cloud data. Summary of the Invention

[0004] Analysis revealed that the method of generating pseudo-labels based on point cloud category activation maps only considers the point cloud data as a single modality, and inaccurate pseudo-labels will lead to inaccurate semantic segmentation results of 3D point clouds.

[0005] One of the technical problems to be solved by the embodiments of the present invention is: how to improve the annotation accuracy of three-dimensional point cloud data.

[0006] According to a first aspect of some embodiments of the present invention, a method for training a three-dimensional point cloud processing model is provided, comprising: processing three-dimensional point cloud data using the three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data; obtaining information about pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; determining an alignment constraint loss between the three-dimensional point cloud data and the two-dimensional image data using the point information and the pixel information; determining a classification loss of the three-dimensional point cloud processing model using classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; and training the three-dimensional point cloud processing model using the alignment constraint loss and the classification loss.

[0007] In some embodiments, the information of a point includes at least one of point classification information and feature information; the information of a pixel includes at least one of pixel classification information and feature information.

[0008] In some embodiments, determining the alignment constraint loss between 3D point cloud data and 2D image data using point information and pixel information includes: determining the degree of matching between 3D point cloud data and 2D image data using point information and pixel information; and determining the alignment constraint loss based on the degree of matching.

[0009] In some embodiments, the degree of matching includes at least one of the following: the degree of feature matching between points and pixels, the degree of global feature matching between 3D point cloud data and 2D image data, and the degree of semantic matching between points and pixels.

[0010] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel.

[0011] In some embodiments, where the matching degree includes the feature matching degree between points and pixels, determining the alignment constraint loss based on the matching degree includes: mapping the features of points and pixels to the same dimension to obtain the first point cloud feature of each point and the first pixel feature of each pixel; randomly acquiring a first number of points and corresponding pixels; for each of the first number of points, determining the contrastive learning loss between the point and the corresponding pixel using the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel; and determining the alignment constraint loss based on the sum of the contrastive learning losses of each of the first number of points.

[0012] In some embodiments, for each of the first number of points: the contrastive learning loss between the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.

[0013] In some embodiments, the degree of global feature matching is determined based on the similarity between any two points in the 3D point cloud data and the similarity between any two pixels in the 2D image data.

[0014] In some embodiments, where the matching degree includes the global feature matching degree, determining the alignment constraint loss based on the matching degree includes: mapping the features of points and pixels to the same dimension to obtain the second point cloud features of each point and the second pixel features of each pixel; randomly acquiring a second number of points and corresponding pixels; for any two points in the second number of points, determining the first similarity information between the second point cloud features of the two points and the second similarity information between the second pixel features of the two corresponding pixels in the second number of pixels, and determining the similarity loss corresponding to the two points based on the difference between the first similarity information and the second similarity information; and determining the alignment constraint loss based on the sum of the similarity losses of the points in the 3D point cloud data.

[0015] In some embodiments, the first similarity information and the second similarity information are a similarity matrix.

[0016] In some embodiments, the degree of semantic matching is determined based on the first classification information of each point and the second classification information of each pixel.

[0017] In some embodiments, the second classification information is obtained by processing each pixel using a two-dimensional image processing model, and when the matching degree includes the semantic matching degree, determining the alignment constraint loss based on the matching degree includes: mapping the three-dimensional point cloud data to the same dimension as the two-dimensional image data; processing the mapped three-dimensional point cloud data using a two-dimensional image processing model to obtain the third classification information for each point; determining the semantic matching loss using the second classification information and the third classification information, and determining the alignment constraint loss based on the semantic matching loss.

[0018] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.

[0019] In some embodiments, obtaining information from two-dimensional image data includes: processing the two-dimensional image data using a two-dimensional image processing model to obtain information from the two-dimensional image data.

[0020] In some embodiments, determining the classification loss of a 3D point cloud processing model using classification information of points in 3D point cloud data predicted by a 3D point cloud processing model and pre-labeled scene classification information includes: determining a first classification loss based on the classification information of points in 3D point cloud data predicted by a 3D point cloud processing model and pre-labeled scene classification information; determining a second classification loss based on the classification information of pixels in 2D image data predicted by a 2D image processing model and scene classification information; and determining the classification loss of the 3D point cloud processing model based on the first classification loss and the second classification loss.

[0021] In some embodiments, a first classification loss is determined based on the pooling processing result of the classification information of the 3D point cloud data predicted by the 3D point cloud processing model and the scene classification information; a second classification loss is determined based on the classification information of the pixels in the 2D image data predicted by the 2D image processing model and the scene classification information.

[0022] In some embodiments, a 3D point cloud processing model is trained using a weighted result of alignment constraint loss and classification loss.

[0023] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network.

[0024] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network.

[0025] According to a second aspect of some embodiments of the present invention, a method for training a semantic segmentation model is provided, comprising: predicting classification information in a three-dimensional point cloud data to be processed using a three-dimensional point cloud processing model, wherein the three-dimensional point cloud processing model is trained using any of the aforementioned training methods for three-dimensional point cloud processing models; generating pseudo-labels for points in the three-dimensional point cloud data to be processed based on the classification information; and training a semantic segmentation model for the three-dimensional point cloud using the pseudo-labels.

[0026] According to a third aspect of some embodiments of the present invention, a training apparatus for a three-dimensional point cloud processing model is provided, comprising: a first information acquisition module configured to process three-dimensional point cloud data using the three-dimensional point cloud processing model to acquire information about points in the three-dimensional point cloud data; a second information acquisition module configured to acquire information about pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; an alignment constraint loss determination module configured to determine an alignment constraint loss between the three-dimensional point cloud data and the two-dimensional image data using the point information and the pixel information; a classification loss determination module configured to determine a classification loss of the three-dimensional point cloud processing model using classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; and a three-dimensional point cloud processing model training module configured to train the three-dimensional point cloud processing model using the alignment constraint loss and the classification loss.

[0027] According to a fourth aspect of some embodiments of the present invention, a training apparatus for a three-dimensional point cloud processing model is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any of the aforementioned training methods for a three-dimensional point cloud processing model based on instructions stored in the memory.

[0028] According to a fifth aspect of some embodiments of the present invention, a training system for a semantic segmentation model is provided, comprising: a training apparatus for any of the aforementioned three-dimensional point cloud processing models; a prediction module configured to predict classification information in three-dimensional point cloud data to be processed using the three-dimensional point cloud processing model; a generation module configured to generate pseudo-labels for points in the three-dimensional point cloud data to be processed based on the classification information; and a semantic segmentation model training module configured to train a semantic segmentation model for the three-dimensional point cloud using the pseudo-labels.

[0029] According to a sixth aspect of some embodiments of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the aforementioned training methods for a three-dimensional point cloud processing model.

[0030] Some embodiments of the above invention have the following advantages or beneficial effects: In the process of training a three-dimensional point cloud processing model, the embodiments of the present invention refer to the alignment constraint loss of three-dimensional point cloud data and two-dimensional image data, thereby introducing two-dimensional images into the training process of the three-dimensional point cloud processing model, realizing cross-modal guidance of two-dimensional images for three-dimensional point clouds, and improving the accuracy of model training.

[0031] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A flowchart illustrating a training method for a processing model according to some embodiments of the present invention is shown.

[0034] Figure 2 A flowchart illustrating a method for determining alignment constraint loss according to some embodiments of the present invention is shown.

[0035] Figure 3 A flowchart illustrating a method for determining alignment constraint loss according to other embodiments of the present invention is shown.

[0036] Figure 4 A flowchart illustrating a method for determining alignment constraint loss according to further embodiments of the present invention is shown.

[0037] Figure 5A flowchart illustrating a classification loss determination method according to some embodiments of the present invention is shown.

[0038] Figure 6 A flowchart illustrating a training method for a semantic segmentation model of a 3D point cloud according to some embodiments of the present invention is shown.

[0039] Figure 7 A schematic diagram of the structure of a training device for a three-dimensional point cloud processing model according to some embodiments of the present invention is shown.

[0040] Figure 8 A schematic diagram of the structure of a training system for a semantic segmentation model according to some embodiments of the present invention is shown.

[0041] Figure 9 A schematic diagram of the structure of a training device for a three-dimensional point cloud processing model according to other embodiments of the present invention is shown.

[0042] Figure 10 A schematic diagram of the structure of a training device for a three-dimensional point cloud processing model according to some embodiments of the present invention is shown. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0045] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0046] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0048] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0049] Figure 1 A flowchart illustrating a training method for a processing model according to some embodiments of the present invention is shown. Figure 1 As shown, the training method of the processing model in this embodiment includes steps S102 to S110.

[0050] In step S102, the three-dimensional point cloud data is processed using a three-dimensional point cloud processing model to obtain information about the points in the three-dimensional point cloud data.

[0051] 3D point cloud data includes data from a number of points, and the data for each point may include location information and color information. In some embodiments, 3D point cloud data is... The expression is represented by N, where N represents the number of points and 6 represents the 6 dimensions (3D color information and 3D position information).

[0052] 3D point cloud processing models are used to classify each point in 3D point cloud data. For example, they can output the probability of each point belonging to each category, and then determine one or more categories to which the point belongs based on probability thresholds.

[0053] The 3D point cloud processing model can also output the features of each point. For example, intermediate calculation results of the 3D point cloud processing model can be output, representing the features of the points.

[0054] In some embodiments, the information of points in the 3D point cloud data includes at least one of the classification information and feature information of the points in the 3D point cloud data.

[0055] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network. The 3D point cloud processing model can simultaneously output point-by-point feature maps of the 3D point cloud data. and classification prediction probability Where N represents the number of points, L represents the dimension of the 3D point cloud point features, and C represents the number of categories.

[0056] In step S104, information about pixels in the two-dimensional image data is obtained, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area.

[0057] Two-dimensional image data includes data for several pixels, and the data for each pixel may include color information; of course, pixels may also be represented in other ways. In some embodiments, two-dimensional image data is... The expression is represented as H×W, where H×W represents the length multiplied by the width of the image, and 3 represents the three color channels (e.g., red, green, and blue channels).

[0058] Due to differences in acquisition equipment, in some cases, the areas covered by 3D point cloud data and 2D image data may not be exactly the same. However, there is overlap between the areas covered by the two.

[0059] In some embodiments, the information of pixels in two-dimensional image data includes at least one of pixel classification information and feature information. The information type of two-dimensional image data can correspond to the information type of three-dimensional point cloud data; for example, both may be feature information, both may be classification information, or both may include both feature information and classification information. Thus, three-dimensional point cloud data and two-dimensional image data can be described from the perspectives of features and semantics.

[0060] In some embodiments, a two-dimensional image processing model is used to process two-dimensional image data to obtain information from the two-dimensional image data. This improves the efficiency of information acquisition. Depending on the needs, the two-dimensional image processing model can also be trained together with a three-dimensional point cloud processing model to improve the overall prediction accuracy.

[0061] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network. It simultaneously outputs pixel-by-pixel feature maps of the two-dimensional image. and classification prediction probability Where H×W represents the length multiplied by the width of the image, i.e. the number of pixels, D represents the dimension of the pixel features of the two-dimensional image, and C represents the number of categories for semantic segmentation.

[0062] In step S106, the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data is determined using the information of points and pixels.

[0063] Alignment constraints refer to constraints that align 3D point cloud data and 2D image data. In some embodiments, constraints can be based on the features of points and pixels, or on the semantics of points and pixels. Furthermore, constraints can also be based on global attributes (e.g., global features) of the 3D point cloud and the 2D image.

[0064] In some embodiments, the matching degree between 3D point cloud data and 2D image data is determined using point information and pixel information; an alignment constraint loss is determined based on the matching degree. By determining the alignment constraint loss based on the matching degree between 3D point cloud data and 2D image data and training the model, the 3D point cloud processing model can align the 3D point cloud data and 2D image data. For example, it can bring similar points and pixels closer together, move dissimilar points and pixels further apart, or strive to make them have the same semantic meaning at corresponding positions.

[0065] In some embodiments, the degree of matching between 3D point cloud data and 2D image data includes at least one of the following: the degree of feature matching between points in the 3D point cloud data and pixels in the 2D image data, the degree of global feature matching between the 3D point cloud data and the 2D image data, and the degree of semantic matching between points in the 3D point cloud data and pixels in the 2D image data. Embodiments of how these three constraints are determined will be described exemplarily below. Those skilled in the art may also use other methods to determine the degree of matching as needed.

[0066] In step S108, the classification loss of the 3D point cloud processing model is determined using the point classification information predicted by the 3D point cloud processing model, the point classification information in the 3D point cloud data, and the pre-labeled scene classification information. That is, the greater the difference between the predicted value and the labeled value, the greater the classification loss.

[0067] Scene classification information describes the classification result of the entire scene. For example, if a scene includes tables and chairs, its classification information can include information corresponding to "tables" and "chairs". In other words, in the embodiments of this invention, it is not necessary to label the 3D point cloud data point by point before training the 3D point cloud processing model; instead, the entire data is labeled.

[0068] Since the predicted classification information is point-by-point, pooling can be used to obtain the prediction result for the entire scene in order to map it to scene classification information, and then the classification loss can be calculated. This saves on pre-labeling work and improves overall training efficiency.

[0069] In some embodiments, the classification loss is determined based on the cross-entropy loss function. Other methods can also be used to calculate the classification loss as needed, which will not be elaborated here.

[0070] In step S110, a 3D point cloud processing model is trained using alignment constraint loss and classification loss.

[0071] In some embodiments, a 3D point cloud processing model is trained using a weighted result of alignment constraint loss and classification loss. When the alignment constraint loss includes multiple categories, or when the classification loss includes multiple categories, each category may also have a predefined weight.

[0072] In some embodiments, when information about two-dimensional image data is obtained using a two-dimensional image processing model, the classification loss of the two-dimensional image data can also be used to train the two-dimensional image processing model. This allows the two-dimensional image processing model to provide more accurate prediction results.

[0073] In the training process of the 3D point cloud processing model, the embodiments of the present invention refer to the alignment constraint loss of 3D point cloud data and 2D image data, thereby introducing 2D images into the training process of the 3D point cloud processing model, realizing cross-modal guidance of 2D images for 3D point clouds, and improving the accuracy of model training.

[0074] The following examples illustrate how to determine alignment constraints based on three different levels of matching.

[0075] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel. See below for reference. Figure 2 An embodiment of the method for determining alignment constraint loss is described.

[0076] Figure 2 A flowchart illustrating a method for determining alignment constraint loss according to some embodiments of the present invention is shown. Figure 2 As shown, the alignment constraint loss determination method in this embodiment includes steps S202 to S208.

[0077] In step S202, the features of points and pixels are mapped to the same dimension to obtain the first point cloud feature of each point and the first pixel feature of each pixel.

[0078] For example, the features of both points and pixels can be processed into a new E-dimensional data set. Point and pixel mapping can be achieved separately through two fully connected layer networks.

[0079] In step S204, a first number of points and their corresponding pixels are randomly obtained.

[0080] The corresponding points and pixels can be determined in advance through methods such as feature matching. For example, when a scene includes a table corner, the points corresponding to the table corner in the 3D point cloud data and the pixels corresponding to the table corner in the 2D image data are the corresponding points and pixels.

[0081] In step S206, for each of the first number of points, the contrastive learning loss between the point and the corresponding pixel is determined by using the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel.

[0082] For ease of description, assume that the corresponding points and pixels have the same identifier, i.e., the i-th point corresponds to the i-th pixel. Then, for the i-th point, we use the similarity between the first point cloud feature of the i-th point and the first pixel feature of the i-th pixel, and the similarity between the first point cloud feature of the i-th point and the 1st, 2nd, ..., Nth pixel features. c The sum of the similarity of the first pixel features of each pixel in the first number of pixels is used to determine the contrastive learning loss between the i-th point and the i-th pixel.

[0083] In some embodiments, the contrastive learning loss between a point and its corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.

[0084] In step S208, the alignment constraint loss is determined based on the sum of the contrastive learning losses for each of the first number of points. The alignment constraint loss is positively correlated with the sum of the contrastive learning losses.

[0085] Let the sum of the contrastive learning losses be denoted as the "point-to-pixel contrast loss". In some embodiments, the point-to-pixel contrast loss... Determined according to formula (1).

[0086]

[0087] in, Features are the mapped points in a 3D point cloud data. N represents the mapped features of pixels in a two-dimensional image data. c Let τ be the number of points and pixels, i be the adjustment parameter, and k be the identifiers of the points and pixels. The purpose of this loss function is to increase the feature similarity between a point and its corresponding pixel (positive sample), while decreasing the similarity between a point and its non-corresponding pixel (negative sample). The adjustment parameter controls the degree to which the loss function focuses on difficult negative samples (negative samples with high similarity). A larger adjustment parameter results in lower focus; a smaller adjustment parameter results in higher focus. In practical applications, appropriate adjustment parameters can be selected through experimentation.

[0088] Through the above embodiments, the features of corresponding pixels and the features of points are brought closer together, while non-corresponding features are moved further apart, thus achieving the goal of aligning the features of points to the features of the image.

[0089] In some embodiments, the global feature matching degree is determined based on the similarity between any two points in the 3D point cloud data and the similarity between any two pixels in the 2D image data. See below for reference. Figure 3 An embodiment of the method for determining alignment constraint loss is described.

[0090] Figure 3 A flowchart illustrating a method for determining alignment constraint loss according to other embodiments of the present invention is shown. Figure 3 As shown, the alignment constraint loss determination method in this embodiment includes steps S302 to S308.

[0091] In step S302, the features of points and pixels are mapped to the same dimension to obtain the second point cloud feature of each point and the second pixel feature of each pixel.

[0092] For example, the features of points and pixels are processed into a new F-dimensional data. Point and pixel mapping can be implemented using two fully connected layer networks respectively. Step S302 can be step S202 or additional processing performed outside of step S202. That is, E and F can be equal or unequal, and the fully connected layer network used in step S302 can be the same as or different from the fully connected layer network used in step S202.

[0093] In step S304, a second number of points and their corresponding pixels are randomly obtained. The second number may be the same as or different from the first number.

[0094] In step S306, for any two points in the second number of points, the first similarity information between the second point cloud features of the two points and the second similarity information between the second pixel features of two corresponding pixels in the second number of pixels are determined, and the similarity loss corresponding to the two points is determined based on the difference between the first similarity information and the second similarity information.

[0095] In some embodiments, the first similarity information and the second similarity information are similarity matrices. The similarity matrix describes the similarity between any two points or any two pixels and can be regarded as a kind of global structural information.

[0096] In step S308, the alignment constraint loss is determined based on the sum of the similarity losses of the points in the 3D point cloud data.

[0097] Let the sum of similarity losses be denoted as the "global structural similarity loss". In some embodiments, the global structural similarity loss... Determined according to formulas (2)-(4).

[0098]

[0099]

[0100]

[0101] in, Features of points in 3D point cloud data A represents the features of pixels in two-dimensional image data. 3d (i,j) represents the first similarity information, A 2d (i,j) represents the second similarity information, N a The number of points and pixels is represented by i, j, and k, which are the identifiers of the points and pixels.

[0102] Through the above embodiments, the processing model of 3D point cloud can learn the global structural information of image features, thereby achieving alignment of 3D point cloud and 2D image at the global structural level.

[0103] In some embodiments, the semantic matching degree is determined based on a first classification information for each point and a second classification information for each pixel. The first classification information for each point can be determined by the output of a 3D point cloud processing model, and the second classification information for each pixel can be determined by the output of a 2D image processing model. (See below for reference.) Figure 4 An embodiment of the method for determining alignment constraint loss is described.

[0104] Figure 4 A flowchart illustrating a method for determining alignment constraint loss according to further embodiments of the present invention is shown. Figure 4 As shown, the alignment constraint loss determination method in this embodiment includes steps S402 to S406.

[0105] In step S402, the three-dimensional point cloud data is mapped to the same dimension as the two-dimensional image data.

[0106] In step S404, the mapped 3D point cloud data is processed using a 2D image processing model to obtain the third classification information for each point.

[0107] In step S406, the semantic matching loss is determined using the second classification information and the third classification information, and the constraint loss is aligned based on the semantic matching loss.

[0108] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.

[0109] In some embodiments, semantic matching loss Determined according to formula (5).

[0110]

[0111] Where N represents the number of points, C represents the number of categories, and M represents the number of categories. 2d (i,c) represents the prediction result of the two-dimensional image processing model for pixels in two-dimensional image data. This represents the prediction result of a two-dimensional image processing model for points in the mapped three-dimensional point cloud data.

[0112] Through the above embodiments, the 3D point cloud processing model can learn the semantic information of image features, thereby achieving semantic alignment.

[0113] When using a two-dimensional image processing model to obtain information from two-dimensional image data, the classification loss of the two-dimensional image processing model can also be considered when determining the classification loss. See below for reference. Figure 5 This describes an embodiment of the classification loss determination method of the present invention.

[0114] Figure 5 A flowchart illustrating a classification loss determination method according to some embodiments of the present invention is shown. Figure 5 As shown, the classification loss determination method in this embodiment includes steps S502 to S506.

[0115] In step S502, the first classification loss is determined based on the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information.

[0116] In step S504, the second classification loss is determined based on the classification information of pixels in the two-dimensional image data predicted by the two-dimensional image processing model and the scene classification information.

[0117] Since 3D point cloud data and 2D image data cover the same area, i.e. the same scene, they can share pre-labeled scene classification information.

[0118] In step S506, the classification loss of the 3D point cloud processing model is determined based on the first classification loss and the second classification loss.

[0119] In some embodiments, the classification information is the classification probability. The classification loss includes a first classification loss determined based on the classification information of the 3D point cloud data after global average pooling and the scene classification information, and a second classification loss determined based on the classification information of the 3D point cloud data after global average pooling and the scene classification information.

[0120] For example, the pooled classification information can be calculated using formulas (6) and (7), where Z 3d M represents the classification information predicted by the pooled 3D point cloud processing model. 3d (n) represents the classification information predicted by the 3D point cloud processing model before pooling, Z 2d M represents the classification information predicted by the pooled two-dimensional image processing model. 2d (h,w) represents the classification information predicted by the 2D image processing model before pooling, N represents the number of points in the 3D point cloud, and H and W represent the length and width of the 2D image, respectively.

[0121]

[0122]

[0123] Then, the first classification loss can be calculated using formulas (8) and (9). Second category loss Where C represents the number of categories, c represents the category identifier, and Y c The multi-class classification label represents the scene, and σ represents the sigmoid activation function.

[0124]

[0125]

[0126] Through the above embodiments, when calculating the classification loss, the classification accuracy of the 3D point cloud processing model is considered in addition to the classification accuracy of the 3D point cloud processing model. This takes into account the influence of the 2D image processing model on the prediction results, and further improves the training effect of the 3D point cloud processing model.

[0127] After training the 3D point cloud processing model, it can be applied to the training of a 3D point cloud semantic segmentation model. See below for reference. Figure 6 An example of a training method for a semantic segmentation model of 3D point clouds.

[0128] Figure 6 A flowchart illustrating a training method for a semantic segmentation model of a 3D point cloud according to some embodiments of the present invention is shown. Figure 6 As shown, the training method of this embodiment includes steps S602 to S606.

[0129] In step S602, the classification information in the three-dimensional point cloud data to be processed is predicted using a three-dimensional point cloud processing model.

[0130] The 3D point cloud processing model was trained using the method described in any of the foregoing embodiments.

[0131] In step S604, pseudo-labels are generated for the points in the 3D point cloud data to be processed based on the classification information. Each pseudo-label represents the category of the point.

[0132] In step S606, a semantic segmentation model for 3D point clouds is trained using pseudo-labels. That is, the prediction results of the 3D point cloud processing model are used as labeling information for the training data, and the training data is further used to train the semantic segmentation model.

[0133] In some embodiments, when generating pseudo-label categories, for the input point cloud First, the classification probability of each point is obtained through a 3D point cloud processing model. Then, for each point, the category with the highest predicted probability among the C categories is selected as the pseudo-label, resulting in point-by-point pseudo-labels. Then, use it as a supervisory signal to train the semantic segmentation model.

[0134] In some embodiments, a 3D U-Net segmentation model based on a 3D residual network is trained using cross-entropy loss. Other types of semantic segmentation networks can also be trained as needed, or the trained 3D point cloud processing model can be applied to other weakly supervised 3D point cloud learning tasks.

[0135] The following is for reference. Figure 7 An embodiment of a training device for a 3D point cloud processing model is described.

[0136] Figure 7 A schematic diagram of a training apparatus for a three-dimensional point cloud processing model according to some embodiments of the present invention is shown. Figure 7 As shown, the training device 700 for the 3D point cloud processing model in this embodiment includes: a first information acquisition module 7100, configured to process 3D point cloud data using the 3D point cloud processing model to obtain information about points in the 3D point cloud data; a second information acquisition module 7200, configured to acquire information about pixels in 2D image data, wherein the 2D image data and the 3D point cloud data cover the same area; an alignment constraint loss determination module 7300, configured to determine the alignment constraint loss between the 3D point cloud data and the 2D image data using the point information and the pixel information; a classification loss determination module 7400, configured to determine the classification loss of the 3D point cloud processing model using the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information; and a 3D point cloud processing model training module 7500, configured to train the 3D point cloud processing model using the alignment constraint loss and the classification loss.

[0137] In some embodiments, the information of a point includes at least one of point classification information and feature information; the information of a pixel includes at least one of pixel classification information and feature information.

[0138] In some embodiments, the alignment constraint loss determination module 7300 is further configured to use point information and pixel information to determine the degree of matching between the three-dimensional point cloud data and the two-dimensional image data; and to determine the alignment constraint loss based on the degree of matching.

[0139] In some embodiments, the degree of matching includes at least one of the following: the degree of feature matching between points and pixels, the degree of global feature matching between 3D point cloud data and 2D image data, and the degree of semantic matching between points and pixels.

[0140] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel.

[0141] In some embodiments, the alignment constraint loss determination module 7300 is further configured to: map the features of the points and the features of the pixels to the same dimension, and obtain the first point cloud feature of each point and the first pixel feature of each pixel, respectively; randomly obtain a first number of points and corresponding pixels; for each of the first number of points, determine the contrastive learning loss between the point and the corresponding pixel using the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel; and determine the alignment constraint loss based on the sum of the contrastive learning losses of each of the first number of points.

[0142] In some embodiments, for each of the first number of points: the contrastive learning loss between the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.

[0143] In some embodiments, the degree of global feature matching is determined based on the similarity between any two points in the 3D point cloud data and the similarity between any two pixels in the 2D image data.

[0144] In some embodiments, the alignment constraint loss determination module 7300 is further configured to: map the features of points and the features of pixels to the same dimension, respectively, to obtain the second point cloud features of each point and the second pixel features of each pixel, when the matching degree includes the global feature matching degree; randomly obtain a second number of points and corresponding pixels; for any two points in the second number of points, determine the first similarity information between the second point cloud features of the two points and the second similarity information between the second pixel features of the two corresponding pixels in the second number of pixels, and determine the similarity loss corresponding to the two points based on the difference between the first similarity information and the second similarity information; determine the alignment constraint loss based on the sum of the similarity losses of the points in the three-dimensional point cloud data.

[0145] In some embodiments, the first similarity information and the second similarity information are a similarity matrix.

[0146] In some embodiments, the degree of semantic matching is determined based on the first classification information of each point and the second classification information of each pixel.

[0147] In some embodiments, the second classification information is obtained by processing each pixel using a two-dimensional image processing model, and the alignment constraint loss determination module 7300 is further configured to: map the three-dimensional point cloud data to the same dimension as the two-dimensional image data when the matching degree includes the semantic matching degree; process the mapped three-dimensional point cloud data using a two-dimensional image processing model to obtain the third classification information for each point; determine the semantic matching loss using the second classification information and the third classification information, and determine the alignment constraint loss based on the semantic matching loss.

[0148] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.

[0149] In some embodiments, the second information acquisition module 7200 is further configured to process the two-dimensional image data using a two-dimensional image processing model to obtain information from the two-dimensional image data.

[0150] In some embodiments, the classification loss determination module 7400 is further configured to determine a first classification loss based on the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information; determine a second classification loss based on the classification information of pixels in the 2D image data predicted by the 2D image processing model and the scene classification information; and determine the classification loss of the 3D point cloud processing model based on the first classification loss and the second classification loss.

[0151] In some embodiments, the classification loss determination module 7400 is further configured to determine a first classification loss based on the pooling processing result of the classification information of the 3D point cloud data predicted by the 3D point cloud processing model and the scene classification information; and to determine a second classification loss based on the classification information of the pixels in the 2D image data predicted by the 2D image processing model and the scene classification information.

[0152] In some embodiments, the training apparatus 700 further includes a two-dimensional image processing model training module 7600, configured to train a two-dimensional image processing model using alignment constraint loss and classification loss.

[0153] In some embodiments, the 3D point cloud processing model training module 7500 is further configured to train a 3D point cloud processing model using a weighted result of alignment constraint loss and classification loss.

[0154] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network.

[0155] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network.

[0156] The following is for reference. Figure 8An example of a training system for a semantic segmentation model.

[0157] Figure 8 A schematic diagram of the structure of a training system for a semantic segmentation model according to some embodiments of the present invention is shown. Figure 8 As shown, the semantic segmentation model training system 80 of this embodiment includes: a training device 810 for a 3D point cloud processing model, the specific implementation of which can be referred to any embodiment of the present invention; a prediction module 820 configured to predict classification information in the 3D point cloud data to be processed using the 3D point cloud processing model; a generation module 830 configured to generate pseudo-labels for points in the 3D point cloud data to be processed based on the classification information; and a semantic segmentation model training module 840 configured to train a semantic segmentation model for the 3D point cloud using the pseudo-labels.

[0158] Figure 9 A schematic diagram of a training apparatus for a three-dimensional point cloud processing model according to other embodiments of the present invention is shown. Figure 9 As shown, the training apparatus 90 for the three-dimensional point cloud processing model in this embodiment includes: a memory 910 and a processor 920 coupled to the memory 910. The processor 920 is configured to execute the training method for the three-dimensional point cloud processing model in any of the foregoing embodiments based on instructions stored in the memory 910.

[0159] The memory 910 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, and other programs.

[0160] Figure 10 A schematic diagram of a training apparatus for a three-dimensional point cloud processing model according to some embodiments of the present invention is shown. Figure 10 As shown, the training device 100 for the 3D point cloud processing model in this embodiment includes a memory 1010 and a processor 1020, and may also include an input / output interface 1030, a network interface 1040, and a storage interface 1050. These interfaces 1030, 1040, 1050, and the memory 1010 and processor 1020 can be connected, for example, via a bus 1060. The input / output interface 1030 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 1040 provides a connection interface for various networked devices. The storage interface 1050 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0161] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, implements any of the aforementioned training methods for a three-dimensional point cloud processing model.

[0162] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0163] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0164] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0165] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0166] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A training method for a 3D point cloud processing model, comprising: The three-dimensional point cloud data is processed using a three-dimensional point cloud processing model to obtain information about the points in the three-dimensional point cloud data; Obtain pixel information from two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; Using the information of the points and the information of the pixels, determine the alignment constraint loss between the 3D point cloud data and the 2D image data; The classification loss of the 3D point cloud processing model is determined by using the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information. The alignment constraint loss and the classification loss are used to train the 3D point cloud processing model.

2. The training method according to claim 1, wherein: The information of the point includes at least one of the point's classification information and feature information; The information of the pixel includes at least one of the pixel's classification information and feature information.

3. The training method according to claim 1 or 2, wherein, The step of determining the alignment constraint loss between the 3D point cloud data and the 2D image data using the information of the points and the pixels includes: Using the information of the points and the information of the pixels, the degree of matching between the three-dimensional point cloud data and the two-dimensional image data is determined; The alignment constraint loss is determined based on the degree of matching.

4. The training method according to claim 3, wherein, The matching degree includes at least one of the following: the feature matching degree between the point and the pixel, the global feature matching degree between the three-dimensional point cloud data and the two-dimensional image data, and the semantic matching degree between the point and the pixel.

5. The training method according to claim 4, wherein, The degree of feature matching between the point and the pixel is determined based on the similarity between the point and the corresponding pixel.

6. The training method according to claim 5, wherein, When the matching degree includes the feature matching degree between the point and the pixel, determining the alignment constraint loss based on the matching degree includes: Map the features of the points and the features of the pixels to the same dimension to obtain the first point cloud feature of each point and the first pixel feature of each pixel. Randomly obtain the first number of points and their corresponding pixels; For each of the first number of points, the contrastive learning loss between the point and the corresponding pixel is determined by using the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel. The alignment constraint loss is determined based on the sum of the contrastive learning losses for each of the first number of points.

7. The training method according to claim 6, wherein, For each of the first number of points: The comparison learning loss between the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.

8. The training method according to claim 4, wherein, The global feature matching degree is determined based on the similarity between any two points in the three-dimensional point cloud data and the similarity between any two pixels in the two-dimensional image data.

9. The training method according to claim 8, wherein, When the matching degree includes the global feature matching degree, determining the alignment constraint loss based on the matching degree includes: Map the features of the points and the features of the pixels to the same dimension to obtain the second point cloud feature of each point and the second pixel feature of each pixel; Randomly obtain a second number of points and their corresponding pixels; For any two points in the second number of points, determine the first similarity information between the second point cloud features of the two points and the second similarity information between the second pixel features of two corresponding pixels in the second number of pixels, and determine the similarity loss corresponding to the two points based on the difference between the first similarity information and the second similarity information. The alignment constraint loss is determined based on the sum of the similarity losses of the points in the 3D point cloud data.

10. The training method according to claim 9, wherein, The first similarity information and the second similarity information form a similarity matrix.

11. The training method according to claim 4, wherein, The semantic matching degree is determined based on the first classification information of each point and the second classification information of each pixel.

12. The training method according to claim 11, wherein, The second classification information is obtained by processing each pixel using a two-dimensional image processing model, and when the matching degree includes the semantic matching degree, determining the alignment constraint loss based on the matching degree includes: Map the three-dimensional point cloud data to the same dimension as the two-dimensional image data; Using a two-dimensional image processing model, the mapped three-dimensional point cloud data is processed to obtain the third classification information of each point; The semantic matching loss is determined using the second classification information and the third classification information, and the alignment constraint loss is determined based on the semantic matching loss.

13. The training method according to claim 12, wherein, The alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.

14. The training method according to claim 1 or 2, wherein, The information for acquiring the two-dimensional image data includes: The two-dimensional image data is processed using a two-dimensional image processing model to obtain information from the two-dimensional image data.

15. The training method according to claim 14, wherein, The determination of the classification loss of the 3D point cloud processing model, using the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information, includes: The first classification loss is determined based on the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information. The second classification loss is determined based on the classification information of pixels in the two-dimensional image data predicted by the two-dimensional image processing model and the scene classification information. The classification loss of the 3D point cloud processing model is determined based on the first classification loss and the second classification loss.

16. The training method according to claim 15, wherein: A first classification loss is determined based on the pooling results of the classification information of the 3D point cloud data predicted by the 3D point cloud processing model and the scene classification information; a second classification loss is determined based on the classification information of the pixels in the 2D image data predicted by the 2D image processing model and the scene classification information.

17. The training method according to claim 1 or 2, wherein, The 3D point cloud processing model is trained using the weighted result of the alignment constraint loss and the classification loss.

18. The training method according to claim 1 or 2, wherein, The three-dimensional point cloud processing model is implemented using a three-dimensional residual network.

19. The training method according to claim 14, wherein, The two-dimensional image processing model is implemented using a two-dimensional residual network.

20. A method for training a semantic segmentation model, comprising: A three-dimensional point cloud processing model is used to predict classification information in three-dimensional point cloud data to be processed, wherein the three-dimensional point cloud processing model is trained using the training method of the three-dimensional point cloud processing model of any one of claims 1 to 19. Based on the classification information, generate pseudo-labels for the points in the three-dimensional point cloud data to be processed; Using the pseudo-labels, a semantic segmentation model for 3D point clouds is trained.

21. A training device for a three-dimensional point cloud processing model, comprising: The first information acquisition module is configured to process the three-dimensional point cloud data using a three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data. The second information acquisition module is configured to acquire information about pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; The alignment constraint loss determination module is configured to determine the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data using the information of the points and the information of the pixels. The classification loss determination module is configured to determine the classification loss of the 3D point cloud processing model by using the classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and the pre-labeled scene classification information. The 3D point cloud processing model training module is configured to train the 3D point cloud processing model using the alignment constraint loss and the classification loss.

22. A training device for a three-dimensional point cloud processing model, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute a training method for a three-dimensional point cloud processing model as described in any one of claims 1 to 19, based on instructions stored in the memory.

23. A training system for a semantic segmentation model, comprising: The training device for the three-dimensional point cloud processing model as described in claim 21 or 22; The prediction module is configured to use a 3D point cloud processing model to predict classification information in the 3D point cloud data to be processed. The generation module is configured to generate pseudo-labels for points in the three-dimensional point cloud data to be processed based on the classification information. The semantic segmentation model training module is configured to train a semantic segmentation model for a 3D point cloud using the pseudo-labels.

24. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a three-dimensional point cloud processing model according to any one of claims 1 to 19.

Images