A point cloud defect segmentation method and system based on weakly supervised structure residual learning

Abstract

The application discloses a kind of point cloud defect segmentation method and system based on weak supervision structure residual learning, it is related to three-dimensional point cloud processing, industrial defect detection and deep learning technical field, method includes the following steps: the original training point cloud file is preprocessed and weakly labeled by artificial to obtain the training point cloud set after processing and weakly labeled area mark;With obtaining the structure residual of each processing point is calculated;And three-state weak supervision pseudo label judgment is carried out to obtain three-state weak supervision pseudo label;Then training point cloud is input to PointNet++ defect segmentation network model and is trained twice to obtain defect segmentation network model, and then according to the obtained defect segmentation network model, point cloud defect segmentation processing is carried out to the point cloud file to be detected, and point cloud defect segmentation result is obtained.The point cloud defect segmentation method and system based on weak supervision structure residual learning disclosed in the application have the advantages of reducing the labeling cost of point cloud defect segmentation, improving the training reliability, enhancing the recognition ability of point cloud defect area and the like.

Description

Technical Field

[0001] This invention relates to the fields of 3D point cloud processing, industrial defect detection, and deep learning technology, and in particular to a point cloud defect segmentation method and system based on weakly supervised structural residual learning. Background Technology

[0002] 3D point clouds can directly represent the geometric shape of an object's surface or spatial structure, and have been widely used in scenarios such as industrial component inspection, pipeline inner wall inspection, tunnel lining inspection, wall structure inspection, and surface quality inspection of complex structures. In these scenarios, the goal of point cloud defect segmentation is to identify cracks, spalling, depressions, holes, damage, protrusions, or local abnormal regions from a large-scale 3D point set, and output the point-level segmentation results corresponding to the defect regions.

[0003] Existing point cloud defect detection methods mainly include geometric rule-based methods and deep learning-based methods. Geometric rule-based methods typically use manual features such as curvature, normal vector changes, point density, local plane fitting errors, or height differences to identify abnormal regions. These methods are simple to implement, but they are weakly adaptable to scanning noise, point cloud density variations, structural surface undulations, and different types of defects. They also exhibit significant threshold dependence, making it difficult to consistently obtain fine defect segmentation results in complex industrial scenarios.

[0004] Deep learning-based methods typically employ point cloud networks such as PointNet, PointNet++, DGCNN, and Point Transformer to learn features from point clouds and output point-level classification or segmentation results. While these methods can improve feature representation in complex point cloud scenarios, they usually rely on large amounts of precisely labeled point-level data. In real-world industrial inspection scenarios, point clouds are large, defect boundaries are irregular, and defect areas are small, making point-by-point labeling costly, time-consuming, and susceptible to human experience. Therefore, supervised learning methods that rely entirely on precise point-level labeling are difficult to implement quickly with low-cost data.

[0005] To reduce annotation costs, some methods employ weakly supervised annotation methods such as bounding box selection, region selection, or image-level labeling. However, weakly labeled bounding boxes typically only indicate the presence of defects within the box, not that all points within the box are defect points. If all points within the weakly labeled bounding box are directly used as defect points for model training, a large number of mislabeled points will be introduced, leading to blurred defect boundaries, false detections of normal regions, and unstable model training. Furthermore, small-area defects account for a relatively small proportion of point clouds, and the farthest point sampling in traditional PointNet++ primarily focuses on spatial coverage, easily missing local high-anomaly regions during the sampling stage, further affecting defect segmentation accuracy.

[0006] Therefore, existing technologies still have the following problems: First, point-level precise annotation is costly and it is difficult to build large-scale supervised training data; second, traditional geometric rule methods are sensitive to noise and scene changes and have insufficient generalization ability; third, the labels inside weakly labeled boxes are impure and will cause supervision noise if used directly for training; fourth, conventional point cloud networks lack explicit modeling of the differences between local normal and abnormal structures, which easily leads to missed detection of small-area defects or false detection of structural boundary regions.

[0007] Based on the above problems, it is necessary to provide a defect segmentation method that can utilize weak bounding boxes for training and combine local structural residuals, three-state pseudo-labels, and structural residual-aware point cloud networks, so as to improve the accuracy and stability of point cloud defect region segmentation while reducing the cost of point-level annotation. Summary of the Invention

[0008] This invention addresses the problems existing in the prior art by proposing a point cloud defect segmentation method and system based on weakly supervised structural residual learning.

[0009] The technical means employed in this invention are as follows: A point cloud defect segmentation method based on weakly supervised structural residual learning includes the following steps: Step 1: Obtain the original training point cloud file for training, parse the original training point cloud file to obtain the original training point cloud set, and preprocess the original training point cloud set to obtain the processed training point cloud set. Step 2: Perform manual weak annotation box operation on the defect candidate regions in the obtained original training point cloud set to obtain weak annotation boxes, and normalize the weak annotation boxes to generate normalized weak annotation boxes. Map the normalized weak annotation boxes to the processed training point cloud set, and generate weak annotation region labels for each processing point according to the coordinate relationship between the normalized weak annotation boxes and each processing point in the processed training point cloud set. Step 3: Calculate the structural residual for each processed point in the processed training point cloud set to obtain the structural residual for each processed point; Step 4: For each processed point in the processed training point cloud set, construct a point cloud connectivity graph; filter candidate defect points within each normalized weakly labeled bounding box; perform connectivity analysis on each candidate defect point based on the point cloud connectivity graph to obtain the number of points in the connectivity region to which each candidate defect point belongs; and perform three-state weak supervision pseudo-label judgment on each processed point based on the weakly labeled region marker, the structural residual, and the number of points in the connectivity region to which the candidate defect point belongs to obtain a three-state weak supervision pseudo-label. Step 5: Divide the processed training point cloud set into one or more training point cloud blocks, and construct processing point input features for each processing point in each training point cloud block; construct a PointNet++ defect segmentation network model for weak supervision of point cloud structure residual perception, and set the sampling strategy, grouping strategy, and three-state probability output layer of the PointNet++ defect segmentation network model; wherein, the output of the three-state probability output layer is set as the three-state probabilities of the processing point: high confidence normal, high confidence defect, and uncertain state; Step 6: Using the tri-state weakly supervised pseudo-labels, structural residuals, 16 nearest neighbor relationships, and weakly labeled boxes as supervision information for each processing point, construct the tri-state classification loss function, box-level defect existence loss function, residual consistency loss function, and local smoothing loss function for the PointNet++ defect segmentation network model. Step 7: Set the number of training iterations, training data volume, learning rate, and weight decay coefficient for the PointNet++ defect segmentation network model. Input the input features of the processing points in the set number of training point cloud blocks into the PointNet++ defect segmentation network model. Train the PointNet++ defect segmentation network model using the tri-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function, and update the network parameters to obtain the first-stage PointNet++ defect segmentation network model and the tri-state probability of each processing point in the training point cloud block. Step 8: Re-execute forward inference on all training point cloud blocks using the PointNet++ defect segmentation network model of the first stage. Based on the three-state probabilities output by the PointNet++ defect segmentation network model of the first stage, and combined with the weak labeling region markers and structural residuals, update the three-state weak supervision pseudo-labels of all training processing points to obtain the updated three-state weak supervision pseudo-labels. The original three-state weakly supervised pseudo-label in the three-state classification loss function is replaced with the updated three-state weakly supervised pseudo-label, and the network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The number of training times, the amount of training data, the learning rate, and the weight decay coefficient are set for the second-stage training. The input features of the processing points in the training point cloud block are input into the first-stage PointNet++ defect segmentation network model. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function are used to train and update the network parameters to obtain the second-stage PointNet++ defect segmentation network model. Step 9: Obtain the point cloud file to be detected, parse the point cloud file to be detected to obtain the point cloud set to be detected, and preprocess the point cloud set to be detected to obtain the processed point cloud set to be detected. Set the weak labeling region flag of each processing point in the point cloud set to be detected to 0; Structural residuals are calculated for each processed point in the processed point cloud set to be detected to obtain the structural residuals of each processed point. The processed point cloud set to be detected is divided to obtain one or more point cloud blocks to be detected, and a processing point input feature is constructed for each processing point in each point cloud block to be detected. The input features of each processing point in each of the point cloud blocks to be detected are input into the second-stage PointNet++ defect segmentation network model to perform point cloud defect segmentation processing, and the point cloud defect segmentation result is obtained.

[0010] Furthermore, the original training point cloud set is preprocessed, including normalization and voxel downsampling. The normalization process is as follows: The original training point cloud set is obtained by parsing the original training point cloud file. :

[0011] in, For the first One original point; For the first The three-dimensional coordinate vector of the original point; This is the intensity value; The original number of points; when the intensity field is not present in the original training point cloud file, it will be... Set to 0; Calculate the geometric center of the original training point cloud The calculation formula is:

[0012] The formula for calculating the scale normalization factor is:

[0013] in, The maximum Euclidean distance from each original point in the original training point cloud to the geometric center of the point cloud, i.e., the scale normalization factor; when Issue an abnormal signal for point cloud data and stop processing; Normalize the 3D coordinates of each original point in the original training point cloud set:

[0014] in, Indicates the first The three-dimensional coordinate vector obtained by translating and scaling the original points; The voxel downsampling process obtains the processed training point cloud set. as follows:

[0015] in, For the processed training point cloud set, the first One processing point; For the first Normalized three-dimensional coordinate vectors of each processing point; For the first Intensity values ​​at each treatment point; The number of points processed in the processed training point cloud set.

[0016] Furthermore, the process of normalizing the weak bounding boxes to generate normalized weak bounding boxes is as follows:

[0017]

[0018] in, This indicates the coordinate normalization transformation used; Represents any three-dimensional coordinate vector in the original coordinate system; Represents the first in the original coordinate system A weak annotation box; Represents the first in the normalized coordinate system A weak annotation box; Represents the coordinate vector The coordinate vector obtained by performing a normalization transformation; Represents coordinate vector Located in the Within the original weak annotation box; Index for weak bounding boxes; The process of generating a weakly labeled region marker for each processing point based on the coordinate relationship between the normalized weakly labeled bounding box and each processing point in the processed training point cloud set is as follows:

[0019]

[0020] in, Indicates the first element in the processed training point cloud set. Each processing point is located within at least one weakly labeled box; Indicates the first element in the processed training point cloud set. Each processing point is located outside all weak annotation boxes.

[0021] Furthermore, the structural residual calculation is performed on each processed point in the processed training point cloud set to obtain the structural residual of each processed point. The process is as follows: For the processed training point cloud set Each processing point in it, with its The nearest neighbor processing points constitute a local neighborhood. And calculate the local neighborhood geometric center and covariance matrix:

[0022]

[0023] in, For the local neighborhood geometric center, Let be the local neighborhood covariance matrix; for the local neighborhood covariance matrix Perform eigenvalue decomposition to obtain the desired result. The three eigenvalues ​​are used, and the unit eigenvector corresponding to the smallest eigenvalue is taken as the local normal vector. And calculate the original local curvature. :

[0024] Wherein, stability constant Set as ;by and Construct the first The local normal structural datum plane of each treatment point is calculated, and the original distance residual, original normal variation residual, local point density, and original point density variation residual are calculated respectively:

[0025]

[0026]

[0027]

[0028] in, Indicates the first The original distance residuals of each processing point relative to the local normal structural datum plane; Represents the local normal vector With the The displacement vector of each processing point relative to the local neighborhood geometric center The inner product, and the outer absolute value, represent taking the absolute value of the inner product; Represents the residual of the original normal change; Represents the local normal vector transpose; Represents the local neighborhood of the first Local unit normal vector of each processing point; Indicates the first The processing point and its number Euclidean distance between nearest neighbors; Indicates the first The first processing point The three-dimensional coordinate vectors of the nearest neighbor points; Represents the Euclidean norm; Represents local point density; This represents the volume coefficient per unit sphere, with a fixed value of 4.1887902. This represents the residual of the original density change; Indicates the first The median of the local density of each neighboring point within the local neighborhood of each processing point; For the original distance residual Original local curvature Original normal change residual and the original density change residual Perform median absolute deviation normalization separately; Represents any one of the four types of primitive features mentioned above, and ,calculate:

[0029]

[0030]

[0031] in, Representing original features The median in the currently processed point cloud; This represents the absolute deviation of the median of the corresponding original feature; This indicates that the calculation result will be truncated to an interval. ; This represents the corresponding feature after normalization; Obtain the normalized distance residual Normalized local curvature Normalized normal variation residual and normalized density change residual Normalize the local point density into network input features. :

[0032] in, Indicates the first Normalized local point density of each processing point; and This shows the minimum and maximum local point densities in the currently processed point cloud; based on normalized distance residuals. Normalized local curvature Normalized normal variation residual and normalized density change residual Generate structural residuals:

[0033]

[0034] in, For the first The degree of deviation of each treatment point from the local normal structure This represents the structural residual field.

[0035] Further, the specific process of step 4 is as follows: For each processed point in the processed training point cloud set, a point cloud connectivity graph is constructed based on the 16 nearest neighbor relationships of each processed point; within each normalized weakly labeled bounding box, a selection is made... The processing points are selected as candidate defect points. Based on the point cloud connectivity graph, a connectivity analysis is performed on each candidate defect point to obtain the number of point cloud nodes in the connectivity region to which each candidate defect point belongs. Not satisfied and The processing point will Set to 0; Based on the weakly labeled region markers, the structural residuals, and the number of point clouds in the connected domains to which the candidate defect points belong, a three-state weakly supervised pseudo-label is determined for each processing point to obtain the following three-state weakly supervised pseudo-labels:

[0036]

[0037]

[0038] in, Indicates a high-confidence defect; Indicates the high confidence normal point; Indicates an uncertain point.

[0039] Furthermore, the processed training point cloud set is divided into one or more training point cloud blocks, and the processing point input features are constructed for each processing point in each training point cloud block as follows: The processed training point cloud set is divided into one or more fixed-size groups. For training point cloud blocks with a number of processing points, when the number of processing points is not less than 8192, the farthest point sampling is used to sequentially select the center of the point cloud block, and for each block center, the nearest 8192 processing points are selected to form a point cloud block, until all processing points are covered by at least one point cloud block; adjacent point cloud blocks are allowed to overlap; when the number of processing points is less than 8192, all processing points are added to the point cloud block, and the sampling is repeated according to the farthest point sampling order until 8192 points are reached; the processing point input features of each processing point are constructed. for:

[0040] in, For the first Local normal vector of each processing point In the normalized point cloud coordinate system axis, shaft and Components along the axial direction; The sampling strategy of the PointNet++ defect segmentation network model is hybrid sampling, as follows: The set of center points for each layer of the PointNet++ defect segmentation network model consists of a spatial coverage sampling set, a high-structure residual sampling set, and a weakly labeled region sampling set.

[0041] Of all input points in the current layer, Selected according to the sampling order of the farthest points; Select according to structural residuals from largest to smallest; The weakly labeled regions were selected based on their sampling scores, from highest to lowest; the sampling scores for the weakly labeled regions are as follows:

[0042] The center sampling point sets of the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are as follows: The first layer selects 512 spatial coverage sampling points, 256 high-structure residual sampling points, and 256 weakly labeled region sampling points from 8192 input points, resulting in 1024 center points; the second layer selects 128 spatial coverage sampling points, 64 high-structure residual sampling points, and 64 weakly labeled region sampling points from the 1024 center points of the first layer, resulting in 256 center points; the third layer selects 32 spatial coverage sampling points, 16 high-structure residual sampling points, and 16 weakly labeled region sampling points from the 256 center points of the second layer, resulting in 64 center points; each layer first merges the three types of sampling points and removes duplicate points, and then supplements the unselected points to the set number of center points according to the sampling order of the farthest point of that layer; The grouping strategy of the PointNet++ defect segmentation network model is as follows: The query radii of the sampling point set of the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are set to 0.08, 0.16, and 0.32 respectively, and the number of neighboring points is set to 32, 64, and 128 respectively for processing point queries. When the number of points within the query radius exceeds the set number, the nearest corresponding number of processing points are retained according to Euclidean distance from near to far. When the number of points within the radius is insufficient, processing points are supplemented from outside the radius according to Euclidean distance from near to far. For each processing point set obtained from the query, the neighboring points are sorted in ascending order of structural residual. The top half of the sorted points are set as normal supporting neighbors, and the bottom half are set as abnormal candidate neighbors.

[0043] Furthermore, the process of constructing the three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function for the PointNet++ defect segmentation network model is as follows: The tri-state classification loss uses cross-entropy with class weights:

[0044] in, , , , representing the loss weights for the high-confidence normal category, the high-confidence defect category, and the uncertain category, respectively; Represents the three-state classification loss; This indicates the number of processing points contained in the current training point cloud block; Indicates the relationship with the first Three-state weak supervision pseudo-label of each processing point Corresponding category weights; This indicates that the network will be the first Each processing point is predicted to be a false label. The probability of the referred category, where the superscript Use category indexes instead of exponentiation; , and The first The probability that each processing point belongs to a high-confidence normal, high-confidence defective, or uncertain state, the sum of which is 1; There is a loss function for box-level defects:

[0045] in, This indicates that there is a loss due to a frame-level defect; This represents the number of valid weak bounding boxes within the current point cloud block; when At that time, Set to 0; Indicates the first The probability of a valid weakly labeled bounding box having a box-level defect is calculated as follows:

[0046] in, Indicates the first A set of indexes of processing points within a valid weakly labeled bounding box; Indicates from Defect probability selected highest A set of indexes for each processing point; Represents a set The number of processing points in the process; Let the set of points within the high residual frame and the set of normal points in the low residual frame be respectively:

[0047] The residual consistency loss function is:

[0048] in, This indicates the residual consistency loss; This represents the set of high residual point indices where the structural residual within the weak annotation box is not less than 0.60. This represents the set of indices of low residual normal points with structural residuals not exceeding 0.25. and These represent the number of processing points in the corresponding sets; The local smoothing loss function is:

[0049] in, This represents the loss due to local smoothing. Indicates the first The processing point and its number Similarity weights between adjacent points are determined based on spatial distance and normal vector differences; Indicates the first A set of 16 nearest neighbor indexes for each processing point; The calculation is as follows:

[0050] in, Represented by natural constant An exponential function with base 1; the distance term and the normal vector difference term are both expressed as the square of the Euclidean norm; The total training loss function is:

[0051] in, This represents the total training loss function.

[0052] Furthermore, step 8 also includes averaging the three-state probabilities of a processing point if the processing point is located in one of the training point cloud blocks using the following formula:

[0053] in, Indicates the first The processing point belongs to the first Average cross-block prediction probability of class state; Indicates containing the first A set of training point cloud block indexes for each processing point; Represents a set The number of training point cloud blocks in the data; Indicates the training point cloud block index; Indicates the first The processing point is at the... Among the training point cloud blocks, the one predicted to be the first... The probability of class state; if the first If a processing point is located in only one training point cloud block, then the three-state probability output by that training point cloud block is directly used. The tri-state weak supervision pseudo-label is updated based on the tri-state probability of each processing point, combined with the weak labeling region marker and structural residual. Processing points located in multiple training point cloud blocks are predicted using their cross-block average probability, while processing points located in a single training point cloud block are predicted using the probability output by that training point cloud block.

[0054]

[0055]

[0056] The original tri-state weak supervision pseudo-label in the tri-state classification loss function is replaced with the updated tri-state weak supervision pseudo-label. The network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The training point cloud blocks are then trained in the second stage using the tri-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function. After updating the network parameters, the second-stage PointNet++ defect segmentation network model is obtained.

[0057] Furthermore, step 1 also includes searching for the nearest neighbor processing point in the processed training point cloud set for each original training point in the normalized original training point cloud set, and recording the mapping index:

[0058] in, For the first The nearest neighbor processing point index corresponding to each original point; Step 9 further includes taking the average of the three-state probabilities of a processing point in a point cloud block to be detected if the processing point is located in multiple point cloud blocks to be detected:

[0059] in, Indicates containing the first A set of indexes of point cloud blocks to be detected for each processing point; Represents a set The number of point cloud blocks to be detected in the data; Indicates the first The processing point is at the... Among the point cloud blocks to be detected, the one predicted to be the first... The probability of a class state; Indicates the fused first Class state prediction probability; An initial defect mask is generated based on the obtained tri-state probability mean and structural residual of the processing point:

[0060]

[0061] When constructing a point cloud connectivity graph based on the 16 nearest neighbor relationships of each processing point, for those satisfying... Perform connected component analysis on the processing points, delete defective connected components with fewer than 20 points, and obtain the final processing point defect mask. ; Reconstruct the original point cloud defect mask based on the nearest neighbor processing point index recorded in step 1:

[0062] in, Indicates the first Defect mask for each original point; Indicates the relationship with the first The final defect mask of the processing points with nearest neighbor mapping relationship of the original points; Output a point cloud file with a defect label field:

[0063]

[0064] in, This represents the output point cloud set containing the original 3D coordinates, original intensity values, and defect mask fields. This indicates the number of defect points in the original point cloud.

[0065] A point cloud defect segmentation system based on weakly supervised structural residual learning is provided for executing the point cloud defect segmentation method based on weakly supervised structural residual learning described in this invention.

[0066] Compared with existing technologies, the point cloud defect segmentation method and system based on weakly supervised structural residual learning disclosed in this invention have the following beneficial effects: (1) Reduce the labeling cost of point cloud defect segmentation: This application uses manual weak label boxes as the source of training supervision, which eliminates the need for precise labeling of each point cloud and reduces the amount of manual labeling work in the process of constructing industrial point cloud defect datasets.

[0067] (2) Improve training reliability under weak supervision: This application combines structural residuals, weakly labeled regions and connected domain constraints to generate three-state pseudo-labels of high-confidence defect points, high-confidence normal points and uncertain points, avoiding directly taking all points in the weakly labeled box as defect points, thereby reducing the impact of incorrect labels on model training.

[0068] (3) Enhance the ability to identify point cloud defect regions: This application constructs a structural residual field by varying the local normal structural base surface, curvature, normal and density, and introduces structural residual sensing sampling and normal-abnormal contrast neighborhood grouping in the PointNet++ feature extraction process, so that the model can learn more fully the deviation features of the defect region relative to the normal structure.

[0069] (4) Improve the stability and practicality of point cloud defect segmentation results: This application continuously optimizes model parameters through structural residual consistency pseudo-label iteration and manual review closed-loop update mechanism, and outputs point cloud files with defect label fields, defect area visualization results and defect point number statistics results, which facilitate subsequent defect review, quality assessment and operation and maintenance decisions. Attached Figure Description

[0070] Figure 1 This is a flowchart of the point cloud defect segmentation method based on weakly supervised structural residual learning disclosed in this invention; Figure 2 This is a schematic diagram of the process for generating structural residual fields and generating three-state weakly supervised pseudo-labels in this invention; Figure 3 This is a schematic diagram of the structure of the PointNet++ defect segmentation network with weak supervision and structural residual awareness disclosed in this invention. Figure 4 This is a schematic diagram of a point cloud defect segmentation system based on weakly supervised structural residual learning disclosed in this invention. Detailed Implementation

[0071] like Figure 1 and Figure 2As shown, the point cloud defect segmentation method based on weakly supervised structural residual learning disclosed in this invention includes the following steps: Step 1: Obtain the original training point cloud file for training, parse the original training point cloud file to obtain the original training point cloud set, and preprocess the original training point cloud set to obtain the processed training point cloud set. Step 2: Perform manual weak annotation box operation on the defect candidate regions in the obtained original training point cloud set to obtain weak annotation boxes, and normalize the weak annotation boxes to generate normalized weak annotation boxes. Map the normalized weak annotation boxes to the processed training point cloud set, and generate weak annotation region labels for each processing point according to the coordinate relationship between the normalized weak annotation boxes and each processing point in the processed training point cloud set. Step 3: Calculate the structural residual for each processed point in the processed training point cloud set to obtain the structural residual for each processed point; Step 4: For each processed point in the processed training point cloud set, construct a point cloud connectivity graph; filter candidate defect points within each normalized weakly labeled bounding box; perform connectivity analysis on each candidate defect point based on the point cloud connectivity graph to obtain the number of points in the connectivity region to which each candidate defect point belongs; and perform three-state weak supervision pseudo-label judgment on each processed point based on the weakly labeled region marker, the structural residual, and the number of points in the connectivity region to which the candidate defect point belongs to obtain a three-state weak supervision pseudo-label. Step 5: Divide the processed training point cloud set into one or more training point cloud blocks, and construct processing point input features for each processing point in each training point cloud block; construct a PointNet++ defect segmentation network model for weak supervision of point cloud structure residual perception, and set the sampling strategy, grouping strategy, and three-state probability output layer of the PointNet++ defect segmentation network model; wherein, the output of the three-state probability output layer is set as the three-state probabilities of the processing point: high confidence normal, high confidence defect, and uncertain state; Step 6: Using the tri-state weakly supervised pseudo-labels, structural residuals, 16 nearest neighbor relationships, and weakly labeled boxes as supervision information for each processing point, construct the tri-state classification loss function, box-level defect existence loss function, residual consistency loss function, and local smoothing loss function for the PointNet++ defect segmentation network model. Step 7: Set the number of training iterations, training data volume, learning rate, and weight decay coefficient for the PointNet++ defect segmentation network model. Input the input features of the processing points in the set number of training point cloud blocks into the PointNet++ defect segmentation network model. Train the PointNet++ defect segmentation network model using the tri-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function, and update the network parameters to obtain the first-stage PointNet++ defect segmentation network model and the tri-state probability of each processing point in the training point cloud block. Step 8: Re-execute forward inference on all training point cloud blocks using the PointNet++ defect segmentation network model of the first stage. Based on the three-state probabilities output by the PointNet++ defect segmentation network model of the first stage, and combined with the weak labeling region markers and structural residuals, update the three-state weak supervision pseudo-labels of all training processing points to obtain the updated three-state weak supervision pseudo-labels. The original three-state weakly supervised pseudo-label in the three-state classification loss function is replaced with the updated three-state weakly supervised pseudo-label, and the network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The number of training times, the amount of training data, the learning rate, and the weight decay coefficient are set for the second-stage training. The input features of the processing points in the training point cloud block are input into the first-stage PointNet++ defect segmentation network model. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function are used to train and update the network parameters to obtain the second-stage PointNet++ defect segmentation network model. Step 9: Obtain the point cloud file to be detected, parse the point cloud file to be detected to obtain the point cloud set to be detected, and preprocess the point cloud set to be detected to obtain the processed point cloud set to be detected. Set the weak labeling region flag of each processing point in the point cloud set to be detected to 0; Structural residuals are calculated for each processed point in the processed point cloud set to be detected to obtain the structural residuals of each processed point. The processed point cloud set to be detected is divided to obtain one or more point cloud blocks to be detected, and a processing point input feature is constructed for each processing point in each point cloud block to be detected. The input features of each processing point in each of the point cloud blocks to be detected are input into the second-stage PointNet++ defect segmentation network model to perform point cloud defect segmentation processing, and the point cloud defect segmentation result is obtained.

[0072] The point cloud defect segmentation method based on weakly supervised structural residual learning disclosed in this invention has the following beneficial effects: (1) Reduce the labeling cost of point cloud defect segmentation: This application uses manual weakly labeled boxes as the source of training supervision, which eliminates the need for precise labeling of point clouds point by point, thereby reducing the amount of manual labeling work in the process of constructing industrial point cloud defect datasets.

[0073] (2) Improve training reliability under weak supervision: This application combines structural residuals, weakly labeled regions and connected domain constraints to generate three-state pseudo-labels of high-confidence defect points, high-confidence normal points and uncertain points, avoiding directly taking all points in the weakly labeled box as defect points, thereby reducing the impact of incorrect labels on model training.

[0074] (3) Enhance the ability to identify point cloud defect regions: This application constructs a structural residual field by varying the local normal structural base surface, curvature, normal and density, and introduces structural residual sensing sampling and normal-abnormal contrast neighborhood grouping in the PointNet++ feature extraction process, so that the model can learn more fully the deviation features of the defect region relative to the normal structure.

[0075] (4) Improve the stability and practicality of point cloud defect segmentation results: This application continuously optimizes model parameters through structural residual consistency pseudo-label iteration and manual review closed-loop update mechanism, and outputs point cloud files with defect label fields, defect area visualization results and defect point number statistics results, which facilitate subsequent defect review, quality assessment and operation and maintenance decisions.

[0076] Furthermore, the original training point cloud set is preprocessed, including normalization and voxel downsampling. Specifically, the normalization process is as follows: In this embodiment, a custom point cloud data file containing three-dimensional coordinate fields and optional intensity fields is denoted as a .pcb format point cloud file. The point cloud file is read and used as the original training point cloud file for training the network model. The original training point cloud set is obtained by parsing the original training point cloud file. :

[0077] in, For the first One original point; For the first The three-dimensional coordinate vector of the original point; This is the strength value; The original number of points; when the intensity field is not present in the original training point cloud file, it will be... Set to 0; Calculate the geometric center of the original training point cloud The calculation formula is:

[0078] The formula for calculating the scale normalization factor is:

[0079] in, The maximum Euclidean distance from each original point in the original training point cloud to the geometric center of the point cloud, i.e., the scale normalization factor; when Issue an abnormal signal for point cloud data and stop processing; Normalize the 3D coordinates of each original point in the original training point cloud set:

[0080] in, Indicates the first The three-dimensional coordinate vector obtained by translating and scaling the original points; The voxel downsampling process obtains the processed training point cloud set. as follows:

[0081] in, For the processed training point cloud set, the first One processing point; For the first Normalized three-dimensional coordinate vectors of each processing point; For the first Intensity values ​​at each treatment point; The number of points processed in the processed training point cloud set.

[0082] Furthermore, the process of normalizing the weak bounding boxes to generate normalized weak bounding boxes is as follows:

[0083]

[0084] in, This indicates the coordinate normalization transformation used; Represents any three-dimensional coordinate vector in the original coordinate system; Represents the first in the original coordinate system A weak annotation box; Represents the first in the normalized coordinate system A weak annotation box; Represents the coordinate vector The coordinate vector obtained by performing a normalization transformation; Represents coordinate vector Located in the Within the original weak annotation box; Index for weak bounding boxes; The process of generating a weakly labeled region marker for each processing point based on the coordinate relationship between the normalized weakly labeled bounding box and each processing point in the processed training point cloud set is as follows;

[0085]

[0086] in, Indicates the first point in the processed training point cloud set. Each processing point is located within at least one weakly labeled box; Indicates the first point in the processed training point cloud set. Each processing point is located outside all weakly labeled bounding boxes. That is, during the model training data preparation phase, annotators view each original training point cloud through the point cloud annotation interface; when cracks, depressions, holes, peeling, or other suspected defect areas are found, a 3D axis-aligned bounding box covering the candidate area is generated in the original point cloud coordinate system. If multiple separate defect candidate areas exist in the same point cloud, multiple weakly labeled boxes are generated for each; normal point clouds without defects do not generate weakly labeled boxes. A weakly labeled box only indicates the presence of a defect within the box, not that all points within the box are defect points.

[0087] Furthermore, the structural residual calculation is performed on each processed point in the processed training point cloud set to obtain the structural residual of each processed point. The process is as follows: For the processed training point cloud set Each processing point in it, with its The nearest neighbor processing points constitute a local neighborhood. And calculate the local neighborhood geometric center and covariance matrix:

[0088]

[0089] in, For the local neighborhood geometric center, Let be the local neighborhood covariance matrix; for the local neighborhood covariance matrix Perform eigenvalue decomposition to obtain the desired result. The three eigenvalues ​​are used, and the unit eigenvector corresponding to the smallest eigenvalue is taken as the local normal vector. And calculate the original local curvature. :

[0090] Wherein, stability constant Set as ;by and Construct the first The local normal structural datum plane of each treatment point is calculated, and the original distance residual, original normal variation residual, local point density, and original point density variation residual are calculated respectively:

[0091]

[0092]

[0093]

[0094] in, Indicates the first The original distance residuals of each processing point relative to the local normal structural datum plane; Represents the local normal vector With the The displacement vector of each processing point relative to the local neighborhood geometric center The inner product, and the outer absolute value, represent taking the absolute value of the inner product; Represents the residual of the original normal change; Represents the local normal vector Transpose of; Represents the local neighborhood of the first Local unit normal vector of each processing point; Indicates the first The processing point and its number Euclidean distance between nearest neighbors; Indicates the first The first processing point The three-dimensional coordinate vectors of the nearest neighbor points; Represents the Euclidean norm; Represents local point density; This represents the volume coefficient per unit sphere, with a fixed value of 4.1887902. This represents the residual of the original density change; Indicates the first The median of the local density of each neighboring point within the local neighborhood of each processing point; For the original distance residual Original local curvature Original normal change residual and the original density change residual Perform median absolute deviation normalization separately; Represents any one of the four types of primitive features mentioned above, and ,calculate:

[0095]

[0096]

[0097] in, Representing original features The median in the currently processed point cloud; This represents the absolute deviation of the median of the corresponding original feature; This indicates that the calculation result will be truncated to an interval. ; This represents the corresponding feature after normalization; Obtain the normalized distance residual Normalized local curvature Normalized normal variation residual and normalized density change residual Normalize the local point density into network input features. :

[0098] in, Indicates the first Normalized local point density of each processing point; and This shows the minimum and maximum local point densities in the currently processed point cloud; based on the normalized distance residual. Normalized local curvature Normalized normal variation residual and normalized density change residual Generate structural residuals:

[0099]

[0100] in, For the first The degree of deviation of each treatment point from the local normal structure The structural residual field is used for the generation of three-state weakly supervised pseudo-labels in step 4, the hybrid sampling in step 5, the division of normal support neighborhoods and abnormal candidate neighborhoods in the grouping strategy, and the online defect determination in step 9.

[0101] Further, the specific process of step 4 is as follows: For each processed point in the processed training point cloud set, a point cloud connectivity graph is constructed based on the 16 nearest neighbor relationships of each processed point; within each normalized weakly labeled bounding box, a selection is made... The processing points are selected as candidate defect points. Based on the point cloud connectivity graph, a connectivity analysis is performed on each candidate defect point to obtain the number of point cloud nodes in the connectivity region to which each candidate defect point belongs. Not satisfied and The processing point will Set to 0; Based on the weakly labeled region markers, the structural residuals, and the number of point clouds in the connected domains to which the candidate defect points belong, a three-state weakly supervised pseudo-label is determined for each processing point to obtain the following three-state weakly supervised pseudo-labels:

[0102]

[0103]

[0104] in, Indicates a high-confidence defect; Indicates the high confidence normal point; This rule indicates uncertain points. Only points with high structural residuals and local connectivity within weakly labeled boxes are designated as high-confidence defect points, while points with blurred boundaries, scan noise, and residual conflict points are retained as uncertain points.

[0105] Furthermore, the processed training point cloud set is divided into one or more training point cloud blocks, and the processing point input features are constructed for each processing point in each training point cloud block as follows: The processed training point cloud set is divided into one or more fixed-size groups. For training point cloud blocks with a number of processing points, when the number of processing points is not less than 8192, the farthest point sampling is used to sequentially select the center of the point cloud block, and for each block center, the nearest 8192 processing points are selected to form a point cloud block, until all processing points are covered by at least one point cloud block; adjacent point cloud blocks are allowed to overlap; when the number of processing points is less than 8192, all processing points are added to the point cloud block, and the sampling is repeated according to the farthest point sampling order until 8192 points are reached; the processing point input features of each processing point are constructed. for:

[0106] in, For the first Local normal vector of each processing point In the normalized point cloud coordinate system axis, shaft and Components along the axial direction; The sampling strategy of the PointNet++ defect segmentation network model is hybrid sampling, as follows: The set of center points for each layer of the PointNet++ defect segmentation network model consists of a spatial coverage sampling set, a high-structure residual sampling set, and a weakly labeled region sampling set.

[0107] Of all input points in the current layer, Selected according to the sampling order of the farthest points; Select according to structural residuals from largest to smallest; The weakly labeled regions were selected based on their sampling scores, from highest to lowest; the sampling scores for the weakly labeled regions are as follows:

[0108] The center sampling point sets of the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are as follows: The first layer selects 512 spatial coverage sampling points, 256 high-structure residual sampling points, and 256 weakly labeled region sampling points from 8192 input points, resulting in 1024 center points; the second layer selects 128 spatial coverage sampling points, 64 high-structure residual sampling points, and 64 weakly labeled region sampling points from the 1024 center points of the first layer, resulting in 256 center points; the third layer selects 32 spatial coverage sampling points, 16 high-structure residual sampling points, and 16 weakly labeled region sampling points from the 256 center points of the second layer, resulting in 64 center points; each layer first merges the three types of sampling points and removes duplicate points, and then supplements the unselected points to the set number of center points according to the sampling order of the farthest point of that layer; The grouping strategy of the PointNet++ defect segmentation network model is as follows: The query radii of the sampling point sets in the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are set to 0.08, 0.16, and 0.32, respectively, and the number of neighboring points is set to 32, 64, and 128, respectively, for processing point queries. When the number of points within the query radius exceeds the set number, the nearest corresponding number of processing points are retained according to Euclidean distance from nearest to farthest. When the number of points within the radius is insufficient, processing points are supplemented from outside the radius according to Euclidean distance from nearest to farthest. For each set of processing points obtained from the query, the neighboring points are sorted in ascending order of structural residual. The top half of the sorted points are set as normal support neighborhoods, and the bottom half are set as anomalous candidate neighborhoods. The specific process is as follows: For each center point, the system sorts the neighboring points in ascending order of structural residual. Since the number of neighboring points in each layer is even, the top half of the sorted points constitutes the normal support neighborhood, and the bottom half constitutes the anomalous candidate neighborhood. Features of the two neighborhoods are extracted using a parameter-independent multilayer perceptron and max pooling, respectively.

[0109]

[0110]

[0111]

[0112] in, It is formed by concatenating the relative coordinates of neighboring points with the center point and the input features of this layer; To support the normal neighborhood features, These are features of the candidate neighborhood of anomalies. The structural deviation features are obtained by element-wise absolute difference. The features are output for the center point of this layer. Thus, each ensemble abstraction layer completes the encoding in the order of "hybrid center point sampling - neighborhood query - normal / abnormal grouping - feature extraction and fusion".

[0113] That is, in this application, if Figure 3 As shown, the PointNet++ defect segmentation network model includes an encoder, a feature propagation decoder, and a three-state decoder. The first layer, the ensemble abstraction layer, takes 8192 points and their 11-dimensional features as input and outputs 1024 center points and 64-dimensional features. The multilayer perceptron channels for both normal supporting neighborhoods and abnormal candidate neighborhoods are [32, 32, 64]. The second layer takes 1024 center points and 64-dimensional features as input and outputs 256 center points and 128-dimensional features. The multilayer perceptron channels are [64, 64, 128]. The third layer takes 256 center points and 128-dimensional features as input and outputs 64 center points and 256-dimensional features. The multilayer perceptron channels are [128, 128, 256]. Each layer of the multilayer perceptron employs one-dimensional convolution, batch normalization, and ReLU activation. The feature propagation decoder uses three-nearest neighbor inverse distance interpolation to propagate the high-level features of 64 center points sequentially to 256 center points, 1024 center points, and 8192 processing points of the current point cloud block. The interpolated features of each layer are concatenated with the low-level features of the corresponding layer in the encoding stage and then processed by the multilayer perceptron. The three feature propagation channels are [256, 256], [256, 128], and [128, 128, 64], respectively. After propagation, each processing point obtains 64-dimensional point-level features. The three-state weakly supervised decoder head consists of two hidden fully connected layers with dimensions of 64 and 32, respectively, and an output layer with dimension 3. The output layer is processed by a softmax function to obtain the three-state probabilities.

[0114] in, , and The first The probability that a processing point belongs to a high-confidence normal state, a high-confidence defective state, or an uncertain state, is summed to 1.

[0115] Furthermore, the process of constructing the three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function for the PointNet++ defect segmentation network model is as follows: The tri-state classification loss uses cross-entropy with class weights:

[0116] in, , , , representing the loss weights for the high-confidence normal category, the high-confidence defect category, and the uncertain category, respectively; Represents the three-state classification loss; This indicates the number of processing points contained in the current training point cloud block; Indicates the relationship with the first Three-state weak supervision pseudo-label of each processing point Corresponding category weights; This indicates that the network will be the first Each processing point is predicted to be a false label. The probability of the referred category, where the superscript Use category indexes instead of exponentiation; , and The first The probability that each processing point belongs to a high-confidence normal, high-confidence defective, or uncertain state, the sum of which is 1; There is a loss function for box-level defects:

[0117] in, This indicates that there is a loss due to a frame-level defect; This represents the number of valid weak bounding boxes within the current point cloud block; when At that time, Set to 0; Indicates the first The probability of a valid weakly labeled bounding box having a box-level defect is calculated as follows:

[0118] in, Indicates the first A set of indexes of processing points within a valid weakly labeled bounding box; Indicates from Defect probability selected highest A set of indexes for each processing point; Represents a set The number of processing points in the process; Let the set of points within the high residual frame and the set of normal points in the low residual frame be respectively:

[0119] The residual consistency loss function is:

[0120] in, This indicates the residual consistency loss; This represents the set of high residual point indices where the structural residual within the weak annotation box is not less than 0.60. This represents the set of indices of low residual normal points with structural residuals not exceeding 0.25. and These represent the number of processing points in the corresponding sets; The local smoothing loss function is:

[0121] in, This represents the loss due to local smoothing. Indicates the first The processing point and its number Similarity weights between adjacent points are determined based on spatial distance and normal vector differences; Indicates the first A set of 16 nearest neighbor indexes for each processing point; The calculation is as follows:

[0122] in, Represented by natural constant An exponential function with base 1; the distance term and the normal vector difference term are both expressed as the square of the Euclidean norm; The total training loss function is:

[0123] in, This represents the total training loss function.

[0124] In this embodiment, backpropagation is performed after averaging the total training loss of each point cloud block within each training batch, and the AdamW optimizer is used to update the network parameters. The first stage of training consists of 80 epochs, with a batch size of 8 point cloud blocks, a learning rate of 0.001, and a weight decay coefficient of 0.0001; the model parameters are saved after the first stage.

[0125] Furthermore, step 8 also includes averaging the three-state probabilities of a processing point if the processing point is located in one of the training point cloud blocks using the following formula:

[0126] in, Indicates the first The processing point belongs to the first Average cross-block prediction probability of class state; Indicates containing the first A set of training point cloud block indexes for each processing point; Represents a set The number of training point cloud blocks in the data; Indicates the training point cloud block index; Indicates the first The processing point is at the... Among the training point cloud blocks, the one predicted to be the first... The probability of class state; if the first If a processing point is located in only one training point cloud block, then the three-state probability output by that training point cloud block is directly used. The tri-state weak supervision pseudo-label is updated based on the tri-state probability of each processing point, combined with the weak labeling region marker and structural residual. Processing points located in multiple training point cloud blocks are predicted using their cross-block average probability, while processing points located in a single training point cloud block are predicted using the probability output by that training point cloud block.

[0127]

[0128]

[0129] The original tri-state weakly supervised pseudo-labels in the tri-state classification loss function are replaced with updated tri-state weakly supervised pseudo-labels. Using the network parameters of the first-stage PointNet++ defect segmentation network model as initial parameters, the training point cloud blocks are trained in the second stage using the aforementioned tri-state classification loss function, box-level defect existence loss function, residual consistency loss function, and local smoothing loss function. After updating the network parameters, the second-stage PointNet++ defect segmentation network model is obtained. In this embodiment, the updated... Replace the original pseudo-labels in the tri-state classification loss in step S4, and use the parameters of the first-stage model as initial parameters to continue training for 40 epochs using the joint loss function from step S6. In the second stage, set the learning rate to 0.0005, maintain the batch size at 8, and keep the weight decay coefficient at 0.0001. After the second stage, solidify the final point cloud defect segmentation model parameters.

[0130] Furthermore, step 1 also includes searching for the nearest neighbor processing point in the processed training point cloud set for each original training point in the normalized original training point cloud set, and recording the mapping index:

[0131] in, For the first The nearest neighbor processing point index corresponding to each original point; Step 9 further includes taking the average of the three-state probabilities of a processing point in a point cloud block to be detected if the processing point is located in multiple point cloud blocks to be detected:

[0132] in, Indicates containing the first A set of indexes of point cloud blocks to be detected for each processing point; Represents a set The number of point cloud blocks to be detected in the data; Indicates the first The processing point is at the... Among the point cloud blocks to be detected, the one predicted to be the first... The probability of a class state; Indicates the fused first Class state prediction probability; An initial defect mask is generated based on the obtained tri-state probability mean and structural residual of the processing point:

[0133]

[0134] When constructing a point cloud connectivity graph based on the 16 nearest neighbor relationships of each processing point, for those satisfying... Perform connected component analysis on the processing points, delete defective connected components with fewer than 20 points, and obtain the final processing point defect mask. ; Reconstruct the original point cloud defect mask based on the nearest neighbor processing point index recorded in step 1:

[0135] in, Indicates the first Defect mask for each original point; Indicates the relationship with the first The final defect mask of the processing points with nearest neighbor mapping relationship of the original points; Output a point cloud file with a defect label field:

[0136]

[0137] in, This represents the output point cloud set containing the original 3D coordinates, original intensity values, and defect mask fields. This indicates the number of defect points in the original point cloud. This process verifies the point cloud defect segmentation, and in the closed-loop verification stage, the verifier can confirm or correct the output defect areas, forming a verification mask. Confirmed samples are added to the closed-loop training set; every 50 confirmed samples are reviewed, and the confirmed normal points and defect points are assigned accordingly. and Assign unconfirmed points And mark all weakly labeled areas. Set to 0. Incremental training continues with the joint loss function from step S6, where box-level defects incur a loss. Set the value to 0, use the current fixed model as the initial model, perform 20 rounds of incremental training with a learning rate of 0.0003, update the point cloud defect segmentation model parameters, and thus form a closed loop of "weak annotation training - online inference - manual review - sample return - model update".

[0138] A point cloud defect segmentation system based on weakly supervised structural residual learning is provided for executing the point cloud defect segmentation method based on weakly supervised structural residual learning described in this invention.

[0139] Specifically, such as Figure 4 As shown, the point cloud defect segmentation system based on weakly supervised structural residual learning disclosed in this invention includes: The preprocessing module is used to obtain the original training point cloud file for training, parse the original training point cloud file to obtain the original training point cloud set, and preprocess the original training point cloud set to obtain the processed training point cloud set. The weak labeling region marker module is used to perform manual weak labeling operations on the defect candidate regions in the obtained original training point cloud set to obtain weak labeling boxes, and to normalize the weak labeling boxes to generate normalized weak labeling boxes. The normalized weak labeling boxes are mapped to the processed training point cloud set, and weak labeling region markers for each processing point are generated according to the coordinate relationship between the normalized weak labeling boxes and each processing point in the processed training point cloud set. The structural residual calculation module is used to perform structural residual calculation on each processed point in the processed training point cloud set to obtain the structural residual of each processed point. The three-state weak supervision pseudo-label acquisition module is used to construct a point cloud connectivity graph for each processed point in the processed training point cloud set; filter candidate defect points within each normalized weak labeling box; perform connectivity analysis on each candidate defect point based on the point cloud connectivity graph to obtain the number of point clouds in the connectivity region to which each candidate defect point belongs; and perform three-state weak supervision pseudo-label judgment on each processed point based on the weak labeling region marker, the structural residual, and the number of point clouds in the connectivity region to which the candidate defect point belongs to obtain a three-state weak supervision pseudo-label. The point cloud block partitioning and network model construction module partitions the processed training point cloud set to obtain one or more training point cloud blocks, constructs processing point input features for each processing point in each training point cloud block, constructs a PointNet++ defect segmentation network model for weak supervision of point cloud structure residual perception, and sets the sampling strategy, grouping strategy, and three-state probability output layer of the PointNet++ defect segmentation network model; wherein, the output of the three-state probability output layer is set as the three-state probabilities of the processing point: high-confidence normal, high-confidence defect, and uncertain state; The loss function construction module is used to construct the three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function of the PointNet++ defect segmentation network model using the three-state weakly supervised pseudo-labels, structural residuals, 16-point nearest neighbor relationships, and weakly labeled boxes as supervision information for each processing point. The first-stage training module is used to set the number of training iterations, training data volume, learning rate, and weight decay coefficient of the PointNet++ defect segmentation network model. The input features of the processing points in the set number of training point cloud blocks are input into the PointNet++ defect segmentation network model. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function are used to train the PointNet++ defect segmentation network model and update the network parameters to obtain the first-stage PointNet++ defect segmentation network model and the three-state probability of each processing point in the training point cloud block. The second-stage training module is used to re-perform forward inference on all training point cloud blocks using the first-stage PointNet++ defect segmentation network model. Based on the three-state probabilities output by the first-stage PointNet++ defect segmentation network model, and combined with the weak labeling region markers and structural residuals, the three-state weak supervision pseudo-labels of all training processing points are updated to obtain the updated three-state weak supervision pseudo-labels. The original three-state weakly supervised pseudo-label in the three-state classification loss function is replaced with the updated three-state weakly supervised pseudo-label, and the network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The number of training times, the amount of training data, the learning rate, and the weight decay coefficient are set for the second-stage training. The input features of the processing points in the training point cloud block are input into the first-stage PointNet++ defect segmentation network model. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function are used to train and update the network parameters to obtain the second-stage PointNet++ defect segmentation network model. The online detection module is used to acquire a point cloud file to be detected, parse the point cloud file to be detected to obtain a set of point clouds to be detected, and preprocess the set of point clouds to be detected to obtain a processed set of point clouds to be detected. Set the weak labeling region flag of each processing point in the point cloud set to be detected to 0; Structural residuals are calculated for each processed point in the processed point cloud set to be detected to obtain the structural residuals of each processed point. The processed point cloud set to be detected is divided to obtain one or more point cloud blocks to be detected, and a processing point input feature is constructed for each processing point in each point cloud block to be detected. The input features of each processing point in each of the point cloud blocks to be detected are input into the second-stage PointNet++ defect segmentation network model to perform point cloud defect segmentation processing, and the point cloud defect segmentation result is obtained.

[0140] The specific execution content and method of each module in the point cloud defect segmentation system based on weakly supervised structural residual learning disclosed in this application are consistent and will not be described in detail here.

[0141] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A point cloud defect segmentation method based on weakly supervised structural residual learning, characterized in that, Includes the following steps: Step 1: Obtain the original training point cloud file for training, parse the original training point cloud file to obtain the original training point cloud set, and preprocess the original training point cloud set to obtain the processed training point cloud set. Step 2: Perform manual weak annotation box operation on the defect candidate regions in the obtained original training point cloud set to obtain weak annotation boxes, and normalize the weak annotation boxes to generate normalized weak annotation boxes. Map the normalized weak annotation boxes to the processed training point cloud set, and generate weak annotation region labels for each processing point according to the coordinate relationship between the normalized weak annotation boxes and each processing point in the processed training point cloud set. Step 3: Calculate the structural residual for each processed point in the processed training point cloud set to obtain the structural residual for each processed point; Step 4: For each processed point in the processed training point cloud set, construct a point cloud connectivity graph; Candidate defect points are selected within each normalized weakly labeled bounding box. Based on the point cloud connectivity graph, a connectivity analysis is performed on each candidate defect point to obtain the number of point clouds within the connectivity region to which each candidate defect point belongs. Based on the weak labeling region marker, the structural residual, and the number of point clouds in the connected domain to which the candidate defect point belongs, a three-state weak supervision pseudo-label is determined for each processing point to obtain a three-state weak supervision pseudo-label. Step 5: Divide the processed training point cloud set to obtain one or more training point cloud blocks, and construct processing point input features for each processing point in each training point cloud block; A PointNet++ defect segmentation network model for weak supervision of point cloud structure residual perception is constructed, and the sampling strategy, grouping strategy and tri-state probability output layer of the PointNet++ defect segmentation network model are set; wherein, the output of the tri-state probability output layer is set as the tri-state probability of the processing point in the states of high confidence normal, high confidence defect and uncertain. Step 6: Using the tri-state weakly supervised pseudo-labels, structural residuals, 16 nearest neighbor relationships, and weakly labeled boxes as supervision information for each processing point, construct the tri-state classification loss function, box-level defect existence loss function, residual consistency loss function, and local smoothing loss function for the PointNet++ defect segmentation network model. Step 7: Set the number of training iterations, training data volume, learning rate, and weight decay coefficient for the PointNet++ defect segmentation network model. Input the input features of the processing points in the set number of training point cloud blocks into the PointNet++ defect segmentation network model. Train the PointNet++ defect segmentation network model using the tri-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function, and update the network parameters to obtain the first-stage PointNet++ defect segmentation network model and the tri-state probability of each processing point in the training point cloud block. Step 8: Re-execute forward inference on all training point cloud blocks using the PointNet++ defect segmentation network model of the first stage. Based on the three-state probabilities output by the PointNet++ defect segmentation network model of the first stage, and combined with the weak labeling region markers and structural residuals, update the three-state weak supervision pseudo-labels of all training processing points to obtain the updated three-state weak supervision pseudo-labels. The original three-state weakly supervised pseudo-label in the three-state classification loss function is replaced with the updated three-state weakly supervised pseudo-label, and the network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The number of training times, the amount of training data, the learning rate, and the weight decay coefficient are set for the second-stage training. The input features of the processing points in the training point cloud block are input into the first-stage PointNet++ defect segmentation network model. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function are used to train and update the network parameters to obtain the second-stage PointNet++ defect segmentation network model. Step 9: Obtain the point cloud file to be detected, parse the point cloud file to be detected to obtain the point cloud set to be detected, and preprocess the point cloud set to be detected to obtain the processed point cloud set to be detected. Set the weak labeling region flag of each processing point in the point cloud set to be detected to 0; Structural residuals are calculated for each processed point in the processed point cloud set to be detected to obtain the structural residuals of each processed point. The processed point cloud set to be detected is divided to obtain one or more point cloud blocks to be detected, and a processing point input feature is constructed for each processing point in each point cloud block to be detected. The input features of each processing point in each of the point cloud blocks to be detected are input into the second-stage PointNet++ defect segmentation network model to perform point cloud defect segmentation processing, and the point cloud defect segmentation result is obtained.

2. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 1, characterized in that: The preprocessing of the original training point cloud set includes normalization and voxel downsampling. The normalization process is as follows: The original training point cloud set is obtained by parsing the original training point cloud file. : in, For the first One original point; For the first The three-dimensional coordinate vector of the original point; This is the intensity value; The original number of points; when the intensity field is not present in the original training point cloud file, it will be... Set to 0; Calculate the geometric center of the original training point cloud The calculation formula is: The formula for calculating the scale normalization factor is: in, The maximum Euclidean distance from each original point in the original training point cloud to the geometric center of the point cloud, i.e., the scale normalization factor; when Issue an abnormal signal for point cloud data and stop processing; Normalize the 3D coordinates of each original point in the original training point cloud set: in, Indicates the first The three-dimensional coordinate vector obtained by translating and scaling the original points; The voxel downsampling process yields the processed training point cloud set. as follows: in, For the processed training point cloud set, the first One processing point; For the first Normalized three-dimensional coordinate vectors of each processing point; For the first Intensity values ​​at each treatment point; The number of points processed in the processed training point cloud set.

3. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 2, characterized in that: The process of normalizing the weak bounding boxes to generate normalized weak bounding boxes is as follows: in, This indicates the coordinate normalization transformation used; Represents any three-dimensional coordinate vector in the original coordinate system; Represents the first in the original coordinate system A weak annotation box; Represents the first in the normalized coordinate system A weak annotation box; Represents the coordinate vector The coordinate vector obtained by performing a normalization transformation; Represents coordinate vector Located in the Within the original weak annotation box; Index for weak bounding boxes; The process of generating a weakly labeled region marker for each processing point based on the coordinate relationship between the normalized weakly labeled bounding box and each processing point in the processed training point cloud set is as follows; in, Indicates the first element in the processed training point cloud set. Each processing point is located within at least one weakly labeled box; Indicates the first element in the processed training point cloud set. Each processing point is located outside all weak annotation boxes.

4. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 3, characterized in that: The process of calculating the structural residual for each processed point in the processed training point cloud set is as follows: For the processed training point cloud set Each processing point in it, with its The nearest neighbor processing points constitute a local neighborhood. And calculate the local neighborhood geometric center and covariance matrix: in, For the local neighborhood geometric center, Let be the local neighborhood covariance matrix; for the local neighborhood covariance matrix Perform eigenvalue decomposition to obtain the desired result. The three eigenvalues ​​are used, and the unit eigenvector corresponding to the smallest eigenvalue is taken as the local normal vector. And calculate the original local curvature. : Wherein, stability constant Set as ;by and Construct the first The local normal structural datum plane of each treatment point is calculated, and the original distance residual, original normal variation residual, local point density, and original point density variation residual are calculated respectively: in, Indicates the first The original distance residuals of each processing point relative to the local normal structural datum plane; Represents the local normal vector With the The displacement vector of each processing point relative to the local neighborhood geometric center The inner product, and the outer absolute value, represent taking the absolute value of the inner product; Represents the residual of the original normal change; Represents the local normal vector transpose; Represents the local neighborhood of the first Local unit normal vector of each processing point; Indicates the first The processing point and its number Euclidean distance between nearest neighbors; Indicates the first The first processing point The three-dimensional coordinate vectors of the nearest neighbor points; Represents the Euclidean norm; Represents local point density; This represents the volume coefficient per unit sphere, with a fixed value of 4.1887902. This represents the residual of the original density change; Indicates the first The median of the local density of each neighboring point within the local neighborhood of each processing point; For the original distance residual Original local curvature Original normal change residual and the original density change residual Perform median absolute deviation normalization separately; Represents any one of the four types of primitive features mentioned above, and ,calculate: in, Representing original features The median in the currently processed point cloud; This represents the absolute deviation of the median of the corresponding original feature; This indicates that the calculation result will be truncated to the interval [ ; This represents the corresponding feature after normalization; Obtain the normalized distance residual Normalized local curvature Normalized normal variation residual and normalized density change residual Normalize the local point density into network input features. : in, Indicates the first Normalized local point density of each processing point; and This shows the minimum and maximum local point densities in the currently processed point cloud; based on normalized distance residuals. Normalized local curvature Normalized normal variation residual and normalized density change residual Generate structural residuals: in, For the first The degree of deviation of each treatment point from the local normal structure This represents the structural residual field.

5. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 4, characterized in that: The specific process of step 4 is as follows: For each processed point in the processed training point cloud set, a point cloud connectivity graph is constructed based on the 16 nearest neighbors of each processed point; and filtering is performed within each normalized weakly labeled bounding box. The processing points are selected as candidate defect points. Based on the point cloud connectivity graph, a connectivity analysis is performed on each candidate defect point to obtain the number of point cloud nodes in the connectivity region to which each candidate defect point belongs. Not satisfied and The processing point will Set to 0; Based on the weakly labeled region markers, the structural residuals, and the number of point clouds in the connected domains to which the candidate defect points belong, a three-state weakly supervised pseudo-label is determined for each processing point to obtain the following three-state weakly supervised pseudo-labels: in, Indicates a high-confidence defect; Indicates the high confidence normal point; Indicates an uncertain point.

6. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 5, characterized in that: The processed training point cloud set is divided into one or more training point cloud blocks. The specific steps for constructing processing point input features for each processing point in each training point cloud block are as follows: The processed training point cloud set is divided into one or more fixed-size groups. For training point cloud blocks with a number of processing points, when the number of processing points is not less than 8192, the farthest point sampling is used to sequentially select the center of the point cloud block, and for each block center, the nearest 8192 processing points are selected to form a point cloud block, until all processing points are covered by at least one point cloud block; adjacent point cloud blocks are allowed to overlap; when the number of processing points is less than 8192, all processing points are added to the point cloud block, and the sampling is repeated according to the farthest point sampling order until 8192 points are reached; the processing point input features of each processing point are constructed. for: in, For the first Local normal vector of each processing point In the normalized point cloud coordinate system axis, shaft and Components along the axial direction; The sampling strategy of the PointNet++ defect segmentation network model is hybrid sampling, as follows: The set of center points for each layer of the PointNet++ defect segmentation network model consists of a spatial coverage sampling set, a high-structure residual sampling set, and a weakly labeled region sampling set. Of all input points in the current layer, Selected according to the sampling order of the farthest points; Select according to structural residuals from largest to smallest; The weakly labeled regions were selected based on their sampling scores, from highest to lowest; the sampling scores for the weakly labeled regions are as follows: The center sampling point sets of the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are as follows: The first layer selects 512 spatial coverage sampling points, 256 high-structure residual sampling points, and 256 weakly labeled region sampling points from 8192 input points, resulting in 1024 center points; the second layer selects 128 spatial coverage sampling points, 64 high-structure residual sampling points, and 64 weakly labeled region sampling points from the 1024 center points of the first layer, resulting in 256 center points; the third layer selects 32 spatial coverage sampling points, 16 high-structure residual sampling points, and 16 weakly labeled region sampling points from the 256 center points of the second layer, resulting in 64 center points; each layer first merges the three types of sampling points and removes duplicate points, and then supplements the unselected points to the set number of center points according to the sampling order of the farthest point of that layer; The grouping strategy of the PointNet++ defect segmentation network model is as follows: The query radii of the sampling point set of the three-layer ensemble abstraction layer of the PointNet++ defect segmentation network model are set to 0.08, 0.16, and 0.32 respectively, and the number of neighboring points is set to 32, 64, and 128 respectively for processing point queries. When the number of points within the query radius exceeds the set number, the nearest corresponding number of processing points are retained according to Euclidean distance from near to far. When the number of points within the radius is insufficient, processing points are supplemented from outside the radius according to Euclidean distance from near to far. For each processing point set obtained from the query, the neighboring points are sorted in ascending order of structural residual. The top half of the sorted points are set as normal supporting neighbors, and the bottom half are set as abnormal candidate neighbors.

7. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 6, characterized in that: The process of constructing the three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function, and the local smoothing loss function of the PointNet++ defect segmentation network model is as follows: The tri-state classification loss uses cross-entropy with class weights: in, , , , representing the loss weights for the high-confidence normal category, the high-confidence defect category, and the uncertain category, respectively; This represents the three-state classification loss; This indicates the number of processing points contained in the current training point cloud block; Indicates the relationship with the first Three-state weak supervision pseudo-label of each processing point Corresponding category weights; This indicates that the network will be the first Each processing point is predicted to be a false label. The probability of the referred category, where the superscript Use category indexes instead of exponentiation; , and The first The probability that each processing point belongs to a high-confidence normal, high-confidence defective, or uncertain state, the sum of which is 1; There is a loss function for box-level defects: in, This indicates that there is a loss due to a frame-level defect; This represents the number of valid weak bounding boxes within the current point cloud block; when At that time, Set to 0; Indicates the first The probability of a valid weakly labeled bounding box having a box-level defect is calculated as follows: in, Indicates the first A set of indexes of processing points within a valid weakly labeled bounding box; Indicates from Defect probability selected highest A set of indexes for each processing point; Represents a set The number of processing points in the process; Let the set of points within the high residual frame and the set of normal points in the low residual frame be respectively: The residual consistency loss function is: in, This indicates the residual consistency loss; This represents the set of high residual point indices where the structural residual within the weak annotation box is not less than 0.

60. This represents the set of indices of low residual normal points with structural residuals not exceeding 0.

25. and These represent the number of processing points in the corresponding sets; The local smoothing loss function is: in, This represents the loss due to local smoothing. Indicates the first The processing point and its number Similarity weights between adjacent points are determined based on spatial distance and normal vector differences; Indicates the first A set of 16 nearest neighbor indexes for each processing point; The calculation is as follows: in, Represented by natural constant An exponential function with base 1; the distance term and the normal vector difference term are both expressed as the square of the Euclidean norm; The total training loss function is: in, This represents the total training loss function.

8. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 7, characterized in that: Step 8 further includes taking the average of the three-state probabilities of a processing point if the processing point is located in multiple training point cloud blocks, using the following formula: in, Indicates the first The processing point belongs to the first Average cross-block prediction probability of class states; Indicates containing the first A set of training point cloud block indexes for each processing point; Represents a set The number of training point cloud blocks in the data; Indicates the training point cloud block index; Indicates the first The processing point is at the... Among the training point cloud blocks, the one predicted to be the first... The probability of class state; if the first If a processing point is located in only one training point cloud block, then the three-state probability output by that training point cloud block is directly used. The tri-state weak supervision pseudo-label is updated based on the tri-state probability of each processing point, combined with the weak labeling region marker and structural residual. Processing points located in multiple training point cloud blocks are predicted using their cross-block average probability, while processing points located in a single training point cloud block are predicted using the probability output by that training point cloud block. The original three-state weak supervision pseudo-label in the three-state classification loss function is replaced with the updated three-state weak supervision pseudo-label. The network parameters of the first-stage PointNet++ defect segmentation network model are used as the initial parameters. The three-state classification loss function, the box-level defect existence loss function, the residual consistency loss function and the local smoothing loss function are used to train the training point cloud blocks for the second stage. After updating the network parameters, the second-stage PointNet++ defect segmentation network model is obtained.

9. The point cloud defect segmentation method based on weakly supervised structural residual learning according to claim 8, characterized in that: Step 1 further includes searching for the nearest neighbor processed point in the processed training point cloud set for each original training point in the normalized original training point cloud set, and recording the mapping index: in, For the first The nearest neighbor processing point index corresponding to each original point; Step 9 further includes taking the average of the three-state probabilities of a processing point in a point cloud block to be detected if the processing point is located in multiple point cloud blocks to be detected: in, Indicates containing the first A set of indexes of point cloud blocks to be detected for each processing point; Represents a set The number of point cloud blocks to be detected in the data; Indicates the first The processing point is at the... Among the point cloud blocks to be detected, the one predicted to be the first... The probability of a class state; Indicates the fused first Class state prediction probability; An initial defect mask is generated based on the obtained tri-state probability mean and structural residual of the processing point: When constructing a point cloud connectivity graph based on the 16 nearest neighbor relationships of each processing point, for those satisfying... Perform connected component analysis on the processing points, delete defective connected components with fewer than 20 points, and obtain the final processing point defect mask. ; Reconstruct the original point cloud defect mask based on the nearest neighbor processing point index recorded in step 1: in, Indicates the first Defect mask for each original point; Indicates the relationship with the first The final defect mask of the processing points with nearest neighbor mapping relationship of the original points; Output a point cloud file with a defect label field: in, This represents the output point cloud set containing the original 3D coordinates, original intensity values, and defect mask fields. This indicates the number of defect points in the original point cloud.

10. A point cloud defect segmentation system based on weakly supervised structural residual learning, characterized in that: Used to perform the point cloud defect segmentation method based on weakly supervised structural residual learning as described in any one of claims 1 to 9.

Images