Three-dimensional point cloud processing model training method and semantic segmentation model training method, and related device
By combining the alignment constraints and classification losses of three-dimensional point clouds and two-dimensional image data, the three-dimensional point cloud processing model is trained, and the problem of pseudo-label inaccuracy is solved, and the accuracy and labeling accuracy of three-dimensional point cloud semantic segmentation are improved.
Patent Information
- Application Number
- PCT/CN2024/082639
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-03-20
- Publication Date
- 2024-11-14
AI Technical Summary
In the existing three-dimensional point cloud semantic segmentation method, the pseudo-label generated by the weakly supervised point cloud semantic segmentation method based on scene classification supervised signals is inaccurate, resulting in inaccurate semantic segmentation results of the three-dimensional point cloud.
By using the combination of the three-dimensional point cloud processing model and the two-dimensional image processing model, the alignment constraint loss and classification loss of point cloud data and image data are used to train the three-dimensional point cloud processing model to improve the labeling accuracy and model training accuracy. The specific methods include obtaining the alignment constraint loss of three-dimensional point cloud data and two-dimensional image data, aligning based on the degree of feature and semantic matching, and calculating the classification loss using pre-labeled scene classification information, and performing model training.
The labeling accuracy of three-dimensional point cloud data and the accuracy of model training are improved. Through cross-modal guidance, the training effect of the three-dimensional point cloud processing model is improved, and the accuracy of semantic segmentation results is enhanced.
Smart Images

Figure CN2024082639_14112024_PF_FP_ABST
Abstract
Description
Training methods and related equipment for 3D point cloud processing models and semantic segmentation models
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application is based on and claims priority to an application filed in China with application number 202310518670.1 and filing date May 9, 2023. The disclosed content of the Chinese application is hereby introduced as a whole into this application. Technical Field
[0003] The present disclosure relates to the field of three-dimensional point cloud technology, and in particular to a training method and related equipment for a three-dimensional point cloud processing model and a semantic segmentation model. Background Art
[0004] A 3D point cloud refers to the three-dimensional spatial point information acquired by scanning devices such as lidar and depth cameras. It generally contains information such as 3D position, color, and intensity. Semantic segmentation of 3D point clouds, which assigns specific semantic labels to each point in the point cloud, is a key step in scene understanding and has widespread applications in fields such as robotics, augmented reality, and autonomous driving. Therefore, 3D point cloud semantic segmentation has long been a research hotspot in computer vision.
[0005] In related technologies, weakly supervised point cloud semantic segmentation methods based on scene classification supervision signals usually adopt the idea of point cloud category activation maps to generate pseudo labels for each point. This method generates pseudo labels based on point cloud data.
[0006] Summary of the Invention
[0007] According to a first aspect of some embodiments of the present disclosure, a training method for a three-dimensional point cloud processing model is provided, comprising: processing three-dimensional point cloud data using a three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data; obtaining information about pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; determining the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data using the information about points and the information about pixels; determining the classification loss of the three-dimensional point cloud processing model using classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; and training the three-dimensional point cloud processing model using the alignment constraint loss and the classification loss.
[0008] In some embodiments, the point information includes at least one of the classification information and feature information of the point; the pixel information includes at least one of the classification information and feature information of the pixel.
[0009] In some embodiments, using point information and pixel information to determine the alignment constraint loss of three-dimensional point cloud data and two-dimensional image data includes: using point information and pixel information to determine the degree of matching between three-dimensional point cloud data and two-dimensional image data; and determining the alignment constraint loss based on the degree of matching.
[0010] In some embodiments, the degree of matching includes at least one of: a degree of feature matching between points and pixels, a degree of global feature matching between three-dimensional point cloud data and two-dimensional image data, and a degree of semantic matching between points and pixels.
[0011] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel.
[0012] In some embodiments, when the degree of matching includes the degree of feature matching between points and pixels, determining the alignment constraint loss based on the degree of matching includes: mapping the features of the points and the features of the pixels to the same dimension, and obtaining the first point cloud feature of each point and the first pixel feature of each pixel respectively; randomly acquiring a first number of points and corresponding pixels; for each of the first number of points, determining the contrastive learning loss between the point and the corresponding pixel using the sum of the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel; and determining the alignment constraint loss based on the sum of the contrastive learning losses of each of the first number of points.
[0013] In some embodiments, for each of the first number of points: the contrastive learning loss of the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.
[0014] In some embodiments, the degree of global feature matching is determined based on the similarity between any two points in the three-dimensional point cloud data and the similarity between any two pixels in the two-dimensional image data.
[0015] In some embodiments, when the degree of matching includes the degree of global feature matching, determining the alignment constraint loss based on the degree of matching includes: mapping the features of the points and the features of the pixels to the same dimension, and obtaining the second point cloud features of each point and the second pixel features of each pixel respectively; randomly obtaining a second number of points and corresponding pixels; for any two points in the second number of points, determining the first similarity information between the second point cloud features of the two points, and the second similarity information between the second pixel features of two corresponding pixels in the second number of pixels, and determining the similarity loss corresponding to the two points based on the gap between the first similarity information and the second similarity information; determining the alignment constraint loss based on the sum of the similarity losses of the points in the three-dimensional point cloud data.
[0016] In some embodiments, the first similarity information and the second similarity information are similarity matrices.
[0017] In some embodiments, the degree of semantic matching is determined based on the first classification information of each point and the second classification information of each pixel.
[0018] In some embodiments, the second classification information is obtained by processing each pixel using a two-dimensional image processing model, and when the degree of matching includes a semantic matching degree, determining the alignment constraint loss based on the degree of matching includes: mapping the three-dimensional point cloud data to the same dimension as the two-dimensional image data; using the two-dimensional image processing model, processing the mapped three-dimensional point cloud data to obtain third classification information for each point; determining the semantic matching loss using the second classification information and the third classification information, and determining the alignment constraint loss based on the semantic matching loss.
[0019] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.
[0020] In some embodiments, obtaining the information of the two-dimensional image data includes: processing the two-dimensional image data using a two-dimensional image processing model to obtain the information of the two-dimensional image data.
[0021] In some embodiments, determining the classification loss of the three-dimensional point cloud processing model using the classification information of the points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the pre-labeled scene classification information includes: determining a first classification loss based on the classification information of the points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the pre-labeled scene classification information; determining a second classification loss based on the classification information of the pixels in the two-dimensional image data and the scene classification information predicted by the two-dimensional image processing model; and determining the classification loss of the three-dimensional point cloud processing model based on the first classification loss and the second classification loss.
[0022] In some embodiments, a first classification loss is determined based on the pooling processing results of the classification information of the three-dimensional point cloud data and the scene classification information predicted by the three-dimensional point cloud processing model; and a second classification loss is determined based on the classification information of the pixels in the two-dimensional image data and the scene classification information predicted by the two-dimensional image processing model.
[0023] In some embodiments, a 3D point cloud processing model is trained using a weighted result of an alignment constraint loss and a classification loss.
[0024] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network.
[0025] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network.
[0026] According to a second aspect of some embodiments of the present disclosure, a method for training a semantic segmentation model is provided, comprising: using a three-dimensional point cloud processing model to predict classification information in three-dimensional point cloud data to be processed, wherein the three-dimensional point cloud processing model is trained using any of the aforementioned training methods for three-dimensional point cloud processing models; generating pseudo labels for points in the three-dimensional point cloud data to be processed based on the classification information; and using the pseudo labels to train a semantic segmentation model of the three-dimensional point cloud.
[0027] According to a third aspect of some embodiments of the present disclosure, a training device for a three-dimensional point cloud processing model is provided, comprising: a first information acquisition module, configured to process three-dimensional point cloud data using a three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data; a second information acquisition module, configured to obtain information about pixels in two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; an alignment constraint loss determination module, configured to determine the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data using information about points and information about pixels; a classification loss determination module, configured to determine the classification loss of the three-dimensional point cloud processing model using classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; and a three-dimensional point cloud processing model training module, configured to train the three-dimensional point cloud processing model using alignment constraint loss and classification loss.
[0028] According to a fourth aspect of some embodiments of the present disclosure, a training device for a three-dimensional point cloud processing model is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute any one of the aforementioned training methods for the three-dimensional point cloud processing model based on instructions stored in the memory.
[0029] According to a fifth aspect of some embodiments of the present disclosure, a training system for a semantic segmentation model is provided, comprising: a training device for any one of the aforementioned three-dimensional point cloud processing models; a prediction module, configured to use the three-dimensional point cloud processing model to predict classification information in the three-dimensional point cloud data to be processed; a generation module, configured to generate pseudo labels for points in the three-dimensional point cloud data to be processed based on the classification information; and a semantic segmentation model training module, configured to use the pseudo labels to train the semantic segmentation model of the three-dimensional point cloud.
[0030] According to a sixth aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any one of the aforementioned training methods for the three-dimensional point cloud processing model is implemented.
[0031] According to a sixth aspect of some embodiments of the present disclosure, there is provided a computer program comprising: instructions, which, when executed by a processor, cause the processor to execute any one of the aforementioned training methods for a three-dimensional point cloud processing model.
[0032] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0034] FIG1 shows a flowchart of a method for training a processing model according to some embodiments of the present disclosure.
[0035] FIG2 shows a schematic flow chart of a method for determining an alignment constraint loss according to some embodiments of the present disclosure.
[0036] FIG3 shows a schematic flow chart of a method for determining alignment constraint loss according to other embodiments of the present disclosure.
[0037] FIG4 shows a schematic flow chart of a method for determining alignment constraint loss according to further embodiments of the present disclosure.
[0038] FIG5 shows a flow chart of a method for determining classification loss according to some embodiments of the present disclosure.
[0039] FIG6 shows a flowchart of a method for training a semantic segmentation model of a three-dimensional point cloud according to some embodiments of the present disclosure.
[0040] FIG7 shows a schematic structural diagram of a training device for a three-dimensional point cloud processing model according to some embodiments of the present disclosure.
[0041] FIG8 shows a schematic structural diagram of a training system for a semantic segmentation model according to some embodiments of the present disclosure.
[0042] FIG9 shows a schematic structural diagram of a training device for a three-dimensional point cloud processing model according to other embodiments of the present disclosure.
[0043] FIG10 shows a schematic structural diagram of a training device for a three-dimensional point cloud processing model according to some further embodiments of the present disclosure. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.
[0045] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.
[0046] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0047] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0048] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.
[0049] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0050] After analysis, it was found that the method of generating pseudo labels based on point cloud category activation maps only considers the point cloud data modality, and the inaccurate pseudo labels generated will lead to inaccurate semantic segmentation results of the three-dimensional point cloud.
[0051] A technical problem to be solved by the embodiments of the present disclosure is: how to improve the annotation accuracy of three-dimensional point cloud data.
[0052] Figure 1 shows a flow chart of a method for training a processing model according to some embodiments of the present disclosure. As shown in Figure 1 , the method for training a processing model in this embodiment includes steps S102 to S110.
[0053] In step S102 , the three-dimensional point cloud data is processed using a three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data.
[0054] The three-dimensional point cloud data includes data of several points, and the data of each point may include position information and color information. In some embodiments, the three-dimensional point cloud data is Represents, where N represents the number of points and 6 represents 6 dimensions (3D color information and 3D position information).
[0055] The 3D point cloud processing model is used to classify each point in the 3D point cloud data. For example, it can output the probability of each point belonging to each category, and then determine the class or classes to which the point belongs based on the probability threshold.
[0056] The 3D point cloud processing model can also output the features of each point. For example, the intermediate calculation results of the 3D point cloud processing model can be output, and the output represents the features of the point.
[0057] In some embodiments, the information of the points in the three-dimensional point cloud data includes at least one of classification information and feature information of the points in the three-dimensional point cloud data.
[0058] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network. The 3D point cloud processing model can simultaneously output a point-by-point feature map of the 3D point cloud data. and classification prediction probability Where N represents the number of points, L represents the dimension of the three-dimensional point cloud point features, and C represents the number of categories.
[0059] In step S104 , pixel information in the two-dimensional image data is obtained, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area.
[0060] The two-dimensional image data includes data of a plurality of pixels, and the data of each pixel may include color information. Of course, other methods may also be used to represent pixels. In some embodiments, the two-dimensional image data is represented by Indicates that H×W represents the length and width of the image, and 3 represents three color channels (such as red, green, and blue channels).
[0061] Due to differences in acquisition equipment, in some cases, the areas covered by 3D point cloud data and 2D image data may not be completely consistent. However, the areas covered by the two do overlap.
[0062] In some embodiments, pixel information in the 2D image data includes at least one of classification information and feature information of the pixel. The information type of the 2D image data can correspond to the information type of the 3D point cloud data. For example, both can be feature information, both can be classification information, or both can include both feature information and classification information. Thus, the 3D point cloud data and the 2D image data can be described from both a feature perspective and a semantic perspective.
[0063] In some embodiments, a 2D image processing model is used to process 2D image data to obtain information about the 2D image data. This improves the efficiency of information acquisition. If needed, the 2D image processing model can also be trained together with the 3D point cloud processing model to improve overall prediction accuracy.
[0064] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network. At the same time, a pixel-by-pixel feature map of the two-dimensional image is output. and classification prediction probability Where H×W represents the length and width of the image, that is, the number of pixels, D represents the dimension of the two-dimensional image pixel features, and C represents the number of categories for semantic segmentation.
[0065] In step S106 , the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data is determined using the point information and the pixel information.
[0066] Alignment constraints are constraints that align 3D point cloud data with 2D image data. In some embodiments, these constraints can be based on point and pixel characteristics or semantics. Furthermore, they can be based on global properties (e.g., global features) of the 3D point cloud and 2D image.
[0067] In some embodiments, point and pixel information is used to determine the degree of match between the 3D point cloud data and the 2D image data; based on the degree of match, an alignment constraint loss is determined. Determining the alignment constraint loss based on the degree of match between the 3D point cloud data and the 2D image data, and then training the model, allows the 3D point cloud processing model to align the 3D point cloud data with the 2D image data, for example, by bringing similar points and pixels closer together, moving dissimilar points and pixels further apart, or maximizing semantic similarity between the two at corresponding locations.
[0068] In some embodiments, the degree of match between 3D point cloud data and 2D image data includes at least one of the following: the degree of feature match between points in the 3D point cloud data and pixels in the 2D image data; the degree of global feature match between the 3D point cloud data and the 2D image data; and the degree of semantic match between points in the 3D point cloud data and pixels in the 2D image data. Examples of determining these three constraints are described below. Those skilled in the art may also use other methods to determine the degree of match as needed.
[0069] In step S108, the classification loss of the 3D point cloud processing model is determined using the point classification information predicted by the 3D point cloud processing model and the pre-labeled scene classification information. That is, the greater the difference between the predicted value and the labeled value, the greater the classification loss.
[0070] Scene classification information describes the classification results of the entire scene. For example, if a scene includes a table and a chair, its classification information may include information corresponding to "table" and "chair." In other words, in the embodiments of the present disclosure, the 3D point cloud data does not need to be annotated point by point before training the 3D point cloud processing model. Instead, the data is annotated as a whole.
[0071] Since the predicted classification information is point-by-point, in order to match it with the scene classification information, we can use pooling to obtain the prediction result for the entire scene and further calculate the classification loss. This can save the workload of pre-labeling and improve the overall training efficiency.
[0072] In some embodiments, the classification loss is determined based on a cross entropy loss function. Other methods can also be used to calculate the classification loss as needed, which will not be described here.
[0073] In step S110 , the three-dimensional point cloud processing model is trained using alignment constraint loss and classification loss.
[0074] In some embodiments, a weighted result of the alignment constraint loss and the classification loss is used to train a 3D point cloud processing model. When the alignment constraint loss includes multiple categories, or when the classification loss includes multiple categories, each category can also be assigned a preset weight.
[0075] In some embodiments, when a two-dimensional image processing model is used to obtain information about the two-dimensional image data, the classification loss of the two-dimensional image data can also be used to train the two-dimensional image processing model, thereby enabling the two-dimensional image processing model to provide more accurate prediction results.
[0076] In the process of training the three-dimensional point cloud processing model, the embodiment of the present disclosure refers to the alignment constraint loss of three-dimensional point cloud data and two-dimensional image data, thereby introducing two-dimensional images into the training process of the three-dimensional point cloud processing model, realizing cross-modal guidance of two-dimensional images to three-dimensional point clouds, and improving the accuracy of model training.
[0077] The following describes, by way of example, how to determine the alignment constraints based on three levels of matching.
[0078] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel. An embodiment of a method for determining an alignment constraint loss is described below with reference to FIG2 .
[0079] Fig. 2 shows a flow chart of a method for determining alignment constraint loss according to some embodiments of the present disclosure. As shown in Fig. 2 , the method for determining alignment constraint loss in this embodiment includes steps S202 to S208.
[0080] In step S202, the point features and the pixel features are mapped to the same dimension to obtain the first point cloud feature of each point and the first pixel feature of each pixel respectively.
[0081] For example, both point features and pixel features are processed into a new E-dimensional data. Point and pixel mapping can be implemented through two fully connected layer networks respectively.
[0082] In step S204, a first number of points and corresponding pixels are randomly acquired.
[0083] The corresponding points and pixels can be determined in advance by feature matching or other means. For example, when a scene includes a table corner, the point corresponding to the table corner in the 3D point cloud data and the pixel corresponding to the table corner in the 2D image data are the corresponding points and pixels.
[0084] In step S206, for each of the first number of points, the contrastive learning loss of the point and the corresponding pixel is determined using the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.
[0085] For the sake of convenience, it is assumed that the corresponding points and pixels have the same identifier, that is, the i-th point corresponds to the i-th pixel. Then for the i-th point, the similarity between the first point cloud feature of the i-th point and the first pixel feature of the i-th pixel, and the similarity between the first point cloud feature of the i-th point and the 1st, 2nd, ... Nth points are used. c The sum of the similarities of the first pixel features of each pixel in (representing the first number) pixels is used to determine the contrastive learning loss between the i-th point and the i-th pixel.
[0086] In some embodiments, the contrastive learning loss between a point and a corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.
[0087] In step S208, an alignment constraint loss is determined based on the sum of the contrastive learning losses of each of the first number of points. The alignment constraint loss is positively correlated with the sum of the contrastive learning losses.
[0088] Let the sum of contrastive learning losses be “point-to-pixel contrastive loss”. In some embodiments, point-to-pixel contrastive loss Determined according to formula (1).
[0089] in, is the mapped feature of the points in the 3D point cloud data, is the mapped feature of the pixels in the two-dimensional image data, N c is the number of points and pixels, τ is the tuning parameter, and i and k are the identifiers of the points and pixels. The goal of this loss function is to increase the feature similarity between a point and its corresponding pixel (positive example), while reducing the similarity between its non-corresponding pixels (negative examples). The tuning parameter controls the degree of attention the loss function pays to difficult negative examples (negative examples with high similarity). A larger tuning parameter results in less attention; a smaller tuning parameter results in more attention. In specific applications, appropriate tuning parameters can be selected through experimentation.
[0090] Through the above embodiment, the corresponding pixel features and point features are brought closer together, and the non-corresponding features are moved further apart, thereby achieving the purpose of aligning the point features with the image features.
[0091] In some embodiments, the degree of global feature matching is determined based on the similarity between any two points in the 3D point cloud data and the similarity between any two pixels in the 2D image data.
[0092] Fig. 3 shows a flow chart of a method for determining alignment constraint loss according to some other embodiments of the present disclosure. As shown in Fig. 3 , the method for determining alignment constraint loss in this embodiment includes steps S302 to S308.
[0093] In step S302, the point features and the pixel features are mapped to the same dimension to obtain the second point cloud features of each point and the second pixel features of each pixel respectively.
[0094] For example, both point features and pixel features are processed into a new F-dimensional data set. Point and pixel mapping can be implemented using two fully connected layer networks, respectively. Step S302 can be step S202 or an additional process performed outside of step S202. That is, E and F can be equal or unequal, and the fully connected layer network used in step S302 can be the same as or different from the fully connected layer network used in step S202.
[0095] In step S304, a second number of points and corresponding pixels are randomly acquired. The second number may be the same as or different from the first number.
[0096] In step S306, for any two points among the second number of points, first similarity information between the second point cloud features of the two points and second similarity information between the second pixel features of two corresponding pixels in the second number of pixels are determined, and the similarity loss corresponding to the two points is determined based on the gap between the first similarity information and the second similarity information.
[0097] In some embodiments, the first similarity information and the second similarity information are similarity matrices. The similarity matrix describes the similarity between any two points or any two pixels and can be regarded as a kind of global structural information.
[0098] In step S308 , the alignment constraint loss is determined according to the sum of similarity losses of points in the three-dimensional point cloud data.
[0099] Let the sum of similarity losses be “global structural similarity loss”. In some embodiments, the global structural similarity loss Determined according to formulas (2)-(4).
[0100] in, is the feature of the point in the 3D point cloud data, is the feature of the pixel in the two-dimensional image data, A 3d (i, j) is the first similarity information, A 2d (i, j) is the second similarity information, N a is the number of points and pixels, i, j and k are the identifiers of points and pixels.
[0101] Through the above embodiments, the processing model of the three-dimensional point cloud can learn the global structural information of the image features, thereby achieving global structural level alignment of the three-dimensional point cloud and the two-dimensional image.
[0102] In some embodiments, the degree of semantic matching is determined based on the first classification information for each point and the second classification information for each pixel. The first classification information for each point can be determined by the output of a 3D point cloud processing model, and the second classification information for each pixel can be determined by the output of a 2D image processing model. An embodiment of a method for determining alignment constraint loss is described below with reference to FIG4 .
[0103] Fig. 4 shows a flow chart of a method for determining alignment constraint loss according to some other embodiments of the present disclosure. As shown in Fig. 4 , the method for determining alignment constraint loss in this embodiment includes steps S402 to S406.
[0104] In step S402 , the three-dimensional point cloud data is mapped to the same dimension as the two-dimensional image data.
[0105] In step S404, the mapped three-dimensional point cloud data is processed using a two-dimensional image processing model to obtain third classification information of each point.
[0106] In step S406 , a semantic matching loss is determined using the second classification information and the third classification information, and the constraint loss is aligned according to the semantic matching loss.
[0107] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.
[0108] In some embodiments, the semantic matching loss Determined according to formula (5).
[0109] Among them, N represents the number of points, C represents the number of categories, and M 2d (i,c) represents the prediction result of the two-dimensional image processing model for the pixels in the two-dimensional image data. Represents the prediction result of the two-dimensional image processing model for the midpoint of the mapped three-dimensional point cloud data.
[0110] Through the above embodiments, the three-dimensional point cloud processing model can learn the semantic information of image features, thereby achieving alignment at the semantic level.
[0111] When using a two-dimensional image processing model to obtain information about two-dimensional image data, the classification loss of the two-dimensional image processing model may also be considered when determining the classification loss. An embodiment of the classification loss determination method disclosed herein is described below with reference to FIG5 .
[0112] Figure 5 shows a schematic flow chart of a classification loss determination method according to some embodiments of the present disclosure. As shown in Figure 5 , the classification loss determination method of this embodiment includes steps S502 to S506.
[0113] In step S502 , a first classification loss is determined based on classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information.
[0114] In step S504, a second classification loss is determined based on the classification information of the pixels in the two-dimensional image data and the scene classification information predicted by the two-dimensional image processing model.
[0115] Since the 3D point cloud data and the 2D image data cover the same area, that is, target the same scene, they can share pre-labeled scene classification information.
[0116] In step S506 , the classification loss of the three-dimensional point cloud processing model is determined according to the first classification loss and the second classification loss.
[0117] In some embodiments, the classification information is a classification probability. The classification loss includes a first classification loss determined based on the classification information of the three-dimensional point cloud data after global average pooling and the scene classification information, and a second classification loss determined based on the classification information after global average pooling and the scene classification information.
[0118] For example, the classification information after pooling can be calculated by formulas (6) and (7), where Z 3d Represents the classification information predicted by the pooled 3D point cloud processing model, M 3d (n) represents the classification information predicted by the 3D point cloud processing model before pooling, Z 2d Represents the classification information predicted by the pooled two-dimensional image processing model, M 2d (h, w) represents the classification information predicted by the two-dimensional image processing model before pooling, N represents the number of points in the three-dimensional point cloud, and H and W represent the length and width of the two-dimensional image respectively.
[0119] Then, the first classification loss can be calculated by formulas (8) and (9) and the second classification loss Among them, C represents the number of categories, c represents the category identifier, and Y c represents the multi-category classification label of the scene, and σ represents the sigmoid activation function.
[0120] Through the above embodiments, when calculating the classification loss, in addition to considering the classification accuracy of the three-dimensional point cloud processing model, the classification accuracy of the two-dimensional image processing model can also be considered, thereby taking into account the influence of the two-dimensional image processing model on the prediction result, and further improving the training effect of the three-dimensional point cloud processing model.
[0121] After the 3D point cloud processing model is trained, it can be applied to the training of a 3D point cloud semantic segmentation model. An embodiment of a 3D point cloud semantic segmentation model training method is described below with reference to FIG6 .
[0122] Figure 6 shows a flow chart of a method for training a semantic segmentation model for a 3D point cloud according to some embodiments of the present disclosure. As shown in Figure 6 , the training method of this embodiment includes steps S602 to S606.
[0123] In step S602 , the classification information in the three-dimensional point cloud data to be processed is predicted using the three-dimensional point cloud processing model.
[0124] The three-dimensional point cloud processing model is trained using the method in any one of the aforementioned embodiments.
[0125] In step S604, pseudo labels of points in the three-dimensional point cloud data to be processed are generated according to the classification information. Each pseudo label represents a category of the point.
[0126] In step S606, the pseudo labels are used to train a semantic segmentation model for the 3D point cloud. That is, the prediction results of the 3D point cloud processing model are used as label information for the training data, and the training data is further used to train the semantic segmentation model.
[0127] In some embodiments, when generating pseudo-label categories, for the input point cloud First, the classification probability of each point is obtained through the 3D point cloud processing model Then, for each point, the category with the largest predicted probability among the C categories is selected as the pseudo label to obtain the point-by-point pseudo label. It is then used as a supervisory signal to train the semantic segmentation model.
[0128] In some embodiments, a 3D U-Net segmentation model based on a 3D residual network is trained using cross-entropy loss. Alternatively, other types of semantic segmentation networks can be trained, or the trained 3D point cloud processing model can be applied to other 3D point cloud weakly supervised learning tasks.
[0129] An embodiment of a training device for a three-dimensional point cloud processing model is described below with reference to FIG7 .
[0130] FIG7 is a schematic diagram of a training apparatus for a 3D point cloud processing model according to some embodiments of the present disclosure. As shown in FIG7 , the training apparatus 700 for a 3D point cloud processing model in this embodiment includes: a first information acquisition module 7100 configured to process 3D point cloud data using the 3D point cloud processing model to obtain information about points in the 3D point cloud data; a second information acquisition module 7200 configured to obtain information about pixels in 2D image data, where the 2D image data and the 3D point cloud data cover the same area; an alignment constraint loss determination module 7300 configured to determine an alignment constraint loss for the 3D point cloud data and the 2D image data using information about points and pixels; a classification loss determination module 7400 configured to determine a classification loss for the 3D point cloud processing model using classification information of points in the 3D point cloud data predicted by the 3D point cloud processing model and pre-labeled scene classification information; and a 3D point cloud processing model training module 7500 configured to train the 3D point cloud processing model using an alignment constraint loss and a classification loss.
[0131] In some embodiments, the point information includes at least one of the classification information and feature information of the point; the pixel information includes at least one of the classification information and feature information of the pixel.
[0132] In some embodiments, the alignment constraint loss determination module 7300 is further configured to use point information and pixel information to determine the degree of matching between the three-dimensional point cloud data and the two-dimensional image data; and determine the alignment constraint loss based on the degree of matching.
[0133] In some embodiments, the degree of matching includes at least one of: a degree of feature matching between points and pixels, a degree of global feature matching between three-dimensional point cloud data and two-dimensional image data, and a degree of semantic matching between points and pixels.
[0134] In some embodiments, the degree of feature matching between a point and a pixel is determined based on the similarity between the point and the corresponding pixel.
[0135] In some embodiments, the alignment constraint loss determination module 7300 is further configured to, when the degree of matching includes the degree of feature matching between the point and the pixel: map the features of the point and the features of the pixel to the same dimension, and obtain the first point cloud feature of each point and the first pixel feature of each pixel respectively; randomly obtain a first number of points and corresponding pixels; for each of the first number of points, determine the contrastive learning loss between the point and the corresponding pixel using the sum of the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and the sum of the similarity between the first point cloud feature of the point and the first pixel feature of each pixel; determine the alignment constraint loss based on the sum of the contrastive learning losses of each of the first number of points.
[0136] In some embodiments, for each of the first number of points: the contrastive learning loss of the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.
[0137] In some embodiments, the degree of global feature matching is determined based on the similarity between any two points in the three-dimensional point cloud data and the similarity between any two pixels in the two-dimensional image data.
[0138] In some embodiments, the alignment constraint loss determination module 7300 is further configured to, when the degree of matching includes the degree of global feature matching: map the features of the points and the features of the pixels to the same dimension, and obtain the second point cloud features of each point and the second pixel features of each pixel respectively; randomly obtain a second number of points and corresponding pixels; for any two points in the second number of points, determine the first similarity information between the second point cloud features of the two points, and the second similarity information between the second pixel features of two corresponding pixels in the second number of pixels, and determine the similarity loss corresponding to the two points based on the gap between the first similarity information and the second similarity information; determine the alignment constraint loss based on the sum of the similarity losses of the points in the three-dimensional point cloud data.
[0139] In some embodiments, the first similarity information and the second similarity information are similarity matrices.
[0140] In some embodiments, the degree of semantic matching is determined based on the first classification information of each point and the second classification information of each pixel.
[0141] In some embodiments, the second classification information is obtained by processing each pixel using a two-dimensional image processing model, and the alignment constraint loss determination module 7300 is further configured to: map the three-dimensional point cloud data to the same dimension as the two-dimensional image data when the degree of matching includes the degree of semantic matching; use the two-dimensional image processing model to process the mapped three-dimensional point cloud data to obtain third classification information for each point; use the second classification information and the third classification information to determine the semantic matching loss, and determine the alignment constraint loss based on the semantic matching loss.
[0142] In some embodiments, the alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.
[0143] In some embodiments, the second information acquisition module 7200 is further configured to process the two-dimensional image data using a two-dimensional image processing model to obtain information about the two-dimensional image data.
[0144] In some embodiments, the classification loss determination module 7400 is further configured to determine a first classification loss based on the classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the pre-labeled scene classification information; determine a second classification loss based on the classification information of pixels in the two-dimensional image data predicted by the two-dimensional image processing model and the scene classification information; and determine the classification loss of the three-dimensional point cloud processing model based on the first classification loss and the second classification loss.
[0145] In some embodiments, the classification loss determination module 7400 is further configured to determine a first classification loss based on the pooling processing results of the classification information of the three-dimensional point cloud data and the scene classification information predicted by the three-dimensional point cloud processing model; and to determine a second classification loss based on the classification information of the pixels in the two-dimensional image data and the scene classification information predicted by the two-dimensional image processing model.
[0146] In some embodiments, the training device 700 further includes: a two-dimensional image processing model training module 7600, configured to train the two-dimensional image processing model using alignment constraint loss and classification loss.
[0147] In some embodiments, the 3D point cloud processing model training module 7500 is further configured to train the 3D point cloud processing model using a weighted result of the alignment constraint loss and the classification loss.
[0148] In some embodiments, the 3D point cloud processing model is implemented using a 3D residual network.
[0149] In some embodiments, the two-dimensional image processing model is implemented using a two-dimensional residual network.
[0150] An embodiment of a training system for a semantic segmentation model is described below with reference to FIG8 .
[0151] Figure 8 shows a schematic diagram of a semantic segmentation model training system according to some embodiments of the present disclosure. As shown in Figure 8, the semantic segmentation model training system 80 of this embodiment includes: a 3D point cloud processing model training device 810, the specific implementation of which can refer to any embodiment of the present disclosure; a prediction module 820, configured to use the 3D point cloud processing model to predict classification information in the 3D point cloud data to be processed; a generation module 830, configured to generate pseudo labels for points in the 3D point cloud data to be processed based on the classification information; and a semantic segmentation model training module 840, configured to use the pseudo labels to train the semantic segmentation model of the 3D point cloud.
[0152] Figure 9 illustrates a schematic diagram of a 3D point cloud processing model training apparatus according to other embodiments of the present disclosure. As shown in Figure 9 , the 3D point cloud processing model training apparatus 90 of this embodiment includes a memory 910 and a processor 920 coupled to the memory 910 . The processor 920 is configured to execute the 3D point cloud processing model training method of any of the aforementioned embodiments based on instructions stored in the memory 910 .
[0153] The memory 910 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.
[0154] FIG10 shows a schematic structural diagram of a training device for a three-dimensional point cloud processing model according to some other embodiments of the present disclosure. As shown in FIG10 , the training device 100 for a three-dimensional point cloud processing model of this embodiment includes: a memory 1010 and a processor 1020, and may also include an input / output interface 1030, a network interface 1040, a storage interface 1050, and the like. These interfaces 1030, 1040, 1050 and the memory 1010 and the processor 1020 can be connected, for example, via a bus 1060. The input / output interface 1030 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touch screen. The network interface 1040 provides a connection interface for various networked devices. The storage interface 1050 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0155] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any one of the aforementioned training methods for the three-dimensional point cloud processing model.
[0156] Those skilled in the art will appreciate that embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0157] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0158] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0159] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0160] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of protection of the present disclosure.
Claims
1. A training method for a three-dimensional point cloud processing model, comprising: Processing the three-dimensional point cloud data using a three-dimensional point cloud processing model to obtain information about points in the three-dimensional point cloud data; Acquire pixel information in the two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; Determine the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data by using the information of the point and the information of the pixel; Determining a classification loss of the three-dimensional point cloud processing model using classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; The three-dimensional point cloud processing model is trained using the alignment constraint loss and the classification loss.
2. The training method according to claim 1, wherein: The information of the point includes at least one of classification information and feature information of the point; The pixel information includes at least one of classification information and feature information of the pixel.
3. The training method according to claim 1 or 2, wherein: The determining the alignment constraint loss of the three-dimensional point cloud data and the two-dimensional image data by using the information of the point and the information of the pixel comprises: Determine the degree of matching between the three-dimensional point cloud data and the two-dimensional image data by using the information of the points and the information of the pixels; The alignment constraint loss is determined according to the matching degree.
4. The training method according to claim 3, wherein: The matching degree includes at least one of: a feature matching degree between the point and the pixel, a global feature matching degree between the three-dimensional point cloud data and the two-dimensional image data, and a semantic matching degree between the point and the pixel.
5. The training method according to claim 4, wherein: The degree of feature matching between the point and the pixel is determined based on the similarity between the point and the corresponding pixel.
6. The training method according to claim 5, wherein: In a case where the matching degree includes a feature matching degree between the point and the pixel, determining the alignment constraint loss according to the matching degree includes: Mapping the features of the points and the features of the pixels to the same dimension, respectively obtaining a first point cloud feature of each point and a first pixel feature of each pixel; Randomly obtain a first number of points and corresponding pixels; For each of the first number of points, determine a contrastive learning loss between the point and the corresponding pixel using a similarity between a first point cloud feature of the point and a first pixel feature of the corresponding pixel and a sum of similarities between the first point cloud feature of the point and the first pixel feature of each pixel; The alignment constraint loss is determined based on a sum of contrastive learning losses for each of the first number of points.
7. The training method according to claim 6, wherein: For each of the first number of points: The contrastive learning loss between the point and the corresponding pixel is negatively correlated with the similarity between the first point cloud feature of the point and the first pixel feature of the corresponding pixel, and positively correlated with the sum of the similarities between the first point cloud feature of the point and the first pixel feature of each pixel.
8. The training method according to any one of claims 4 to 7, wherein: The global feature matching degree is determined according to the similarity between any two points in the three-dimensional point cloud data and the similarity between any two pixels in the two-dimensional image data.
9. The training method according to claim 8, wherein: In a case where the matching degree includes the global feature matching degree, determining the alignment constraint loss according to the matching degree includes: Mapping the features of the points and the features of the pixels to the same dimension, and obtaining a second point cloud feature of each point and a second pixel feature of each pixel respectively; Randomly obtain a second number of points and corresponding pixels; For any two points in the second number of points, determine first similarity information between second point cloud features of the two points and second similarity information between second pixel features of two corresponding pixels in the second number of pixels, and determine similarity losses corresponding to the two points according to a gap between the first similarity information and the second similarity information; The alignment constraint loss is determined according to the sum of similarity losses of points in the three-dimensional point cloud data.
10. The training method according to claim 9, wherein: The first similarity information and the second similarity information are similarity matrices.
11. The training method according to any one of claims 4 to 10, wherein: The semantic matching degree is determined according to the first classification information of each point and the second classification information of each pixel.
12. The training method according to claim 11, wherein: The second classification information is obtained by processing each pixel using a two-dimensional image processing model, and when the matching degree includes the semantic matching degree, determining the alignment constraint loss according to the matching degree includes: Mapping the three-dimensional point cloud data to the same dimension as the two-dimensional image data; Using a two-dimensional image processing model, the mapped three-dimensional point cloud data is processed to obtain the third classification information of each point; A semantic matching loss is determined using the second classification information and the third classification information, and the alignment constraint loss is determined according to the semantic matching loss.
13. The training method according to claim 12, wherein: The alignment constraint loss is positively correlated with the second classification information and negatively correlated with the third classification information.
14. The training method according to any one of claims 1 to 13, wherein: The information for acquiring the two-dimensional image data includes: The two-dimensional image data is processed using a two-dimensional image processing model to obtain information of the two-dimensional image data.
15. The training method according to claim 14, wherein: Determining the classification loss of the three-dimensional point cloud processing model by using the classification information of the points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the pre-labeled scene classification information comprises: Determining a first classification loss according to classification information of points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and pre-labeled scene classification information; determining a second classification loss according to the classification information of pixels in the two-dimensional image data predicted by the two-dimensional image processing model and the scene classification information; The classification loss of the three-dimensional point cloud processing model is determined according to the first classification loss and the second classification loss.
16. The training method according to claim 15, wherein: A first classification loss is determined based on the pooling processing results of the classification information of the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the scene classification information; a second classification loss is determined based on the classification information of the pixels in the two-dimensional image data predicted by the two-dimensional image processing model and the scene classification information.
17. The training method according to any one of claims 1 to 16, wherein: The three-dimensional point cloud processing model is trained using a weighted result of the alignment constraint loss and the classification loss.
18. The training method according to any one of claims 1 to 17, wherein: The three-dimensional point cloud processing model is implemented using a three-dimensional residual network.
19. The training method according to any one of claims 14 to 18, wherein: The two-dimensional image processing model is implemented using a two-dimensional residual network.
20. A method for training a semantic segmentation model, comprising: Predicting classification information in the three-dimensional point cloud data to be processed using a three-dimensional point cloud processing model, wherein the three-dimensional point cloud processing model is trained using the training method for a three-dimensional point cloud processing model of any one of claims 1 to 19; Generating pseudo labels of points in the three-dimensional point cloud data to be processed according to the classification information; The pseudo labels are used to train a semantic segmentation model of the three-dimensional point cloud.
21. A training device for a three-dimensional point cloud processing model, comprising: A first information acquisition module is configured to process the three-dimensional point cloud data using a three-dimensional point cloud processing model to obtain information of points in the three-dimensional point cloud data; a second information acquisition module, configured to acquire information of pixels in the two-dimensional image data, wherein the two-dimensional image data and the three-dimensional point cloud data cover the same area; The alignment constraint loss determination module is configured to use the information of the point and the information of the pixel to determine Alignment constraint loss between the three-dimensional point cloud data and the two-dimensional image data; a classification loss determination module configured to determine the classification loss of the three-dimensional point cloud processing model using the classification information of the points in the three-dimensional point cloud data predicted by the three-dimensional point cloud processing model and the pre-labeled scene classification information; The three-dimensional point cloud processing model training module is configured to train the three-dimensional point cloud processing model using the alignment constraint loss and the classification loss.
22. A training device for a three-dimensional point cloud processing model, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the training method for the three-dimensional point cloud processing model according to any one of claims 1 to 19 based on instructions stored in the memory.
23. A training system for a semantic segmentation model, comprising: The training device for the three-dimensional point cloud processing model according to claim 21 or 22; A prediction module is configured to predict classification information in the three-dimensional point cloud data to be processed using a three-dimensional point cloud processing model; A generating module, configured to generate pseudo labels of points in the three-dimensional point cloud data to be processed according to the classification information; The semantic segmentation model training module is configured to train a semantic segmentation model of a three-dimensional point cloud using the pseudo labels.
24. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a three-dimensional point cloud processing model according to any one of claims 1 to 19.
25. A computer program comprising: Instructions, when executed by a processor, cause the processor to execute the training method for a three-dimensional point cloud processing model according to any one of claims 1 to 19.
Citation Information
Patent Citations
Three-dimensional point cloud processing method and three-dimensional point cloud processing model training method and device
CN111754636A
Three-dimensional laser radar road point cloud segmentation method based on YOLACT
CN114155265A
Three-dimensional point cloud processing method and system
CN114298168A
Three-dimensional point cloud processing model and semantic segmentation model training method and related equipment
CN116563666A
Semantic labeling of point clouds using images
US10650278B1
Cited By
Unmanned aerial vehicle automatic classification method and system based on power transmission line, and storage medium
CN120877008A
Three-dimensional model retrieval method based on double viewpoint sampling and modal constraint
CN121542454A
