A defect cluster identification method and system based on point cloud semantic segmentation

Abstract

The application provides a defect aggregation identification method and system based on point cloud semantic segmentation, and relates to the technical fields of intelligent manufacturing and artificial intelligence. The method flow is as follows: a defect aggregation data set is constructed; a point cloud semantic segmentation model based on a Transformer is constructed; the point cloud semantic segmentation model is trained using the defect aggregation data set to obtain a defect point semantic segmentation model; the defect point data of an industrial panel are subjected to semantic segmentation using the defect point semantic segmentation model to obtain point-level semantic labels, and the point-level semantic labels are subjected to defect aggregation identification to obtain a defect aggregation identification result. The application regards all defect points as a whole point cloud, uses the self-attention mechanism of the point cloud semantic segmentation model to capture the complex spatial relationship (including long-range and short-range dependence) between points, and combines relative position coding to perceive the geometric structure, so that semantic identification of the defect point data is realized.

Description

Technical Field

[0001] This invention relates to the fields of intelligent manufacturing and artificial intelligence technology, and more specifically, to a defect clustering identification method and system based on point cloud semantic segmentation. Background Technology

[0002] During the manufacturing process of industrial panels (such as displays, solar panels, or semiconductor wafers), various types of defects inevitably arise on their surfaces or within their interiors due to factors such as processes, equipment, and raw materials. If these defects are randomly or sparsely distributed, they may still fall within the acceptable quality range for the product. However, when defects in the same panel exhibit clustering patterns in specific areas (e.g., forming lines, clumps, or grids), it often indicates a systemic production problem, potentially leading to a decline in key performance indicators or even the scrapping of the entire batch. Therefore, accurately and efficiently identifying defect clustering patterns on panels is crucial for timely detection of potential production problems, optimization of process parameters, improvement of product yield, and reduction of production costs.

[0003] In existing technologies, defect clustering identification methods can be mainly divided into two categories:

[0004] (1) Traditional machine learning methods: such as density-based clustering algorithms and statistical analysis methods. The advantages of these methods are that they are relatively simple in principle, fast in calculation, and highly interpretable. In certain specific scenarios, they can achieve certain recognition effects through parameter adjustment. However, the disadvantages of these methods are also very obvious: ① Limited pattern recognition ability, making it difficult to effectively identify complex, non-convex, or clustering patterns with large density variations, such as sparse linear clusters or clusters of specific shapes. ② Strong parameter dependence, the clustering effect is highly dependent on manually set parameters, which often need to be repeatedly adjusted for defective data of different types and densities, lacking adaptability. ③ Insufficient generalization ability, parameters optimized for one dataset may perform poorly on new datasets with different characteristics.

[0005] (2) Image-based deep learning methods: For example, the defect distribution map is treated as an image, and then processed using convolutional neural networks, object detection networks, or image segmentation networks. The advantage of this type of method is that it can automatically extract complex features by learning from a large amount of data, and has a good recognition ability for some clustering patterns that are obvious in images. However, it also faces the following challenges: ① Difficulty in obtaining high-quality labeled data: Deep learning models usually require a large amount of high-quality labeled data for training. For defect clustering problems, it is not only necessary to label whether the samples are clustered or not, but ideally, it is also necessary to accurately label the clustered areas, types, and even the specific defect points involved. This process is time-consuming, labor-intensive, and costly. ② In most cases, data labeling remains at the sample level, making it difficult to provide fine-grained information accurate to the region or defect point. Even if there is region-level labeling, it is difficult to directly guide which specific defect points constitute the cluster. ③ Poor adaptability to changes in defect definition: Once the definition of the type, size, or clustering pattern of defects changes, it often means that a large amount of data needs to be collected and labeled again, resulting in poor flexibility. Summary of the Invention

[0006] This invention provides a defect clustering identification method and system based on point cloud semantic segmentation, which solves the problems existing in the current defect clustering identification methods.

[0007] In a first aspect, embodiments of the present invention provide a defect clustering identification method based on point cloud semantic segmentation, the method comprising the following steps:

[0008] Construct a defect clustering dataset, which includes defect point cloud data of different clustering types and their corresponding point-level semantic labels;

[0009] A point cloud semantic segmentation model based on Transformer is constructed. The point cloud semantic segmentation model includes a point embedding module, a Transformer Encoder module, and a classification output module.

[0010] The point embedding module embeds features into the defect point cloud data to obtain an initial feature vector; the Transformer Encoder module uses a self-attention mechanism with relative position encoding to encode contextual information and learn features from the initial feature vector to obtain a semantic feature vector; the classification output module processes the semantic feature vector for semantic categories to obtain point-level semantic labels.

[0011] The point cloud semantic segmentation model was trained using a defect clustering dataset to obtain a defect point semantic segmentation model.

[0012] The defect point semantic segmentation model is used to perform semantic segmentation on the defect point data of industrial panels to obtain point-level semantic labels, and then defect clustering identification is performed on the point-level semantic labels to obtain defect clustering identification results.

[0013] In the above embodiments, the present invention treats all defect points as a whole point cloud, utilizes the self-attention mechanism of the point cloud semantic segmentation model to capture the complex spatial relationships between points, and combines relative position encoding to perceive geometric structures to achieve semantic recognition of defect point data. Furthermore, it outputs a specific semantic label for each defect point (such as background point, linear cluster point, grid cluster point, etc.), which can accurately identify various complex clustering patterns and locate all relevant points constituting the clusters, providing refined information for subsequent clustering attribute analysis (such as quantity, type, geometric features), offering greater flexibility and scalability.

[0014] As some optional implementations of this application, the process for constructing the defect clustering dataset is as follows:

[0015] Defect point cloud data is generated by simulating using a parameterized algorithm. The clustering types of the defect point cloud data include random distribution, linear clustering, and grid clustering.

[0016] Semantic annotation is performed on defect point cloud data containing different clustering types to construct a defect clustering dataset.

[0017] In the above embodiments, the present invention adopts a parameterized data generation strategy to generate training data with precise point-level semantic labels at low cost and on a large scale, fundamentally solving the bottleneck of training data acquisition.

[0018] As some optional implementations of this application, the process of constructing a Transformer-based point cloud semantic segmentation model is as follows:

[0019] The defect point cloud data is represented as a point set of all defect points, where the defect points are represented using two-dimensional coordinates.

[0020] The point set of all defect points is input into the point embedding module, and the two-dimensional coordinates of the defect points are mapped from low dimension to high dimension through the point embedding module to obtain the initial feature vector;

[0021] Obtain the relative positions of any two defect points, and encode the relative positions of any two defect points to obtain the relative position feature vector;

[0022] The initial feature vector and the relative position feature vector are input into the Transformer Encoder module. The Transformer Encoder module uses a multi-head attention mechanism combined with relative position encoding to perform attention calculation and nonlinear transformation on the initial feature vector and the relative position feature vector to obtain the semantic feature vector.

[0023] The semantic feature vector is input into the classification output module, and the classification output module performs category prediction on the semantic feature vector to obtain point-level semantic labels.

[0024] In the above embodiments, the present invention directly treats the defect point set as a two-dimensional point cloud, adopts a point cloud semantic segmentation model based on Transformer, and introduces relative position encoding to enhance the model's perception of the spatial geometric relationship between points, focusing more on the characteristics of the point cloud data itself.

[0025] As some optional implementations of this application, the process of performing attention calculation and nonlinear transformation on the initial feature vector and the relative position feature vector using the Transformer Encoder module with a multi-head attention mechanism combined with relative position encoding is as follows:

[0026] A linear transformation is performed on the initial feature vectors to obtain a learnable weight matrix;

[0027] Attention scores are calculated based on learnable weight matrices, and the vector bias of the relative position feature vector is added to the attention scores to obtain the attention scores.

[0028] Attention weights are calculated based on attention scores, and the attention weights of all defect points are weighted and summed to obtain the attention output of the defect points.

[0029] The attention output of the defect point is subjected to linear transformation and nonlinear activation to obtain the linear transformation output of the defect point;

[0030] The attention output and linear transformation output of the defect points are residually connected and layer normalized to obtain the semantic feature vector of the defect points.

[0031] In the above embodiments, the present invention employs a point cloud semantic segmentation model incorporating relative position encoding to process defective point clouds, effectively capturing complex spatial dependencies between points. The point cloud semantic segmentation model, with its powerful attention mechanism, can capture both long-range and short-range dependencies between any two points in a point cloud, which is crucial for identifying clusters (such as sparse long straight lines) composed of spatially dispersed but pattern-related points. By introducing relative position encoding, the model can better understand the geometric arrangement of points, enhancing its perception of the spatial structure of the point cloud and thus improving the accuracy of recognizing various clustering patterns.

[0032] As one of the optional implementations of this application, the process of training a point cloud semantic segmentation model using a defect clustering dataset is as follows:

[0033] The pointwise cross-entropy loss function is used to calculate the loss of the defect clustered dataset;

[0034] We employ deep learning optimization and backpropagation to minimize the cross-entropy loss function and update the model parameters to obtain a defect semantic segmentation model.

[0035] As some optional implementations of this application, the process of semantically segmenting defect data of industrial panels using a defect semantic segmentation model is as follows:

[0036] The defect point data of industrial panels is preprocessed to obtain defect point cloud data;

[0037] The defect point cloud data is input into the defect point semantic segmentation model for model inference to obtain the predicted point-level semantic labels.

[0038] In the above embodiments, the present invention outputs point-level semantic tags for each defect point, which can be directly used to automatically determine the cluster type, locate the cluster range, and extract cluster attributes. Its information granularity and degree of automation are far superior to the existing technology.

[0039] As some optional implementations of this application, the process for defect clustering identification of point-level semantic tags is as follows:

[0040] Determine the point-level semantic tags corresponding to industrial panels;

[0041] If all defect clusters are randomly distributed, then the industrial panel is determined to not have a specific type of cluster.

[0042] If the clustering type of defect points is non-random distribution, then the industrial panel is determined to have the corresponding type of clustering.

[0043] In a second aspect, the present invention provides a defect clustering identification system based on point cloud semantic segmentation, the system comprising:

[0044] A data acquisition unit is used to construct a defect clustering dataset, which includes defect point cloud data of different clustering types and their corresponding point-level semantic labels.

[0045] The segmentation model unit is used to construct a point cloud semantic segmentation model based on Transformer. The point cloud semantic segmentation model includes a point embedding module, a Transformer Encoder module, and a classification output module.

[0046] The point embedding module embeds features into the defect point cloud data to obtain an initial feature vector; the Transformer Encoder module uses a self-attention mechanism with relative position encoding to encode contextual information and learn features from the initial feature vector to obtain a semantic feature vector; the classification output module processes the semantic feature vector for semantic categories to obtain point-level semantic labels.

[0047] The model training unit uses a defect aggregation dataset to train a point cloud semantic segmentation model to obtain a defect point semantic segmentation model.

[0048] The clustering identification unit uses a defect point semantic segmentation model to perform semantic segmentation on the defect point data of the industrial panel to obtain point-level semantic labels, and performs defect clustering identification on the point-level semantic labels to obtain defect clustering identification results.

[0049] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned defect clustering identification method based on point cloud semantic segmentation.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the defect clustering identification method based on point cloud semantic segmentation.

[0051] The beneficial effects of this invention are as follows:

[0052] 1. This invention employs a parameterized data generation strategy to replace traditional manual annotation, solving the bottleneck problem of acquiring training data for deep learning models. Since the generation rules for certain cluster types (such as linear and grid-like clusters) are far simpler than their judgment rules, a controllable generation algorithm can be designed to generate training data with precise point-level semantic labels at low cost and on a large scale. This significantly reduces the reliance on expensive and time-consuming manual annotation, especially for semantic segmentation tasks requiring fine-grained annotation. The parameters of the generated data (such as cluster density, length, orientation, and number of points) are adjustable, making the generated dataset diverse and controllable, capable of simulating defect clustering under various real-world conditions, and enhancing the robustness and generalization ability of the model.

[0053] 2. This invention models the defect clustering identification problem as a two-dimensional point cloud semantic segmentation task, achieving refined classification and localization of defect points. Unlike traditional clustering methods (which typically only provide fuzzy clusters) and image-level or coarse region-level visual methods, this scheme treats each defect point as an independent unit in the point cloud for semantic classification (e.g., background point, linear clustering point, grid-like clustering point, etc.). This point-level semantic output provides extremely rich and accurate basic information for subsequent clustering analysis (e.g., OK / NG judgment, clustering type identification, quantity statistics, location localization, geometric attribute extraction, etc.), enabling more flexible and in-depth subsequent applications.

[0054] 3. This invention employs a point cloud semantic segmentation model incorporating relative position encoding to process defective point clouds, effectively capturing complex spatial dependencies between points. The point cloud semantic segmentation model, with its powerful attention mechanism, can capture both long-range and short-range dependencies between any two points in a point cloud. This is crucial for identifying clusters (such as sparse long straight lines) composed of spatially dispersed but pattern-related points. By introducing relative position encoding, the model can better understand the geometric arrangement of points, enhancing its perception of the spatial structure of the point cloud and thus improving the accuracy of recognizing various clustering patterns.

[0055] 4. This invention achieves a combination of high flexibility and high precision, and possesses excellent scalability. Flexibility: Data generation rules can be adjusted according to needs, or new generation algorithms can be added as required; post-processing logic can be flexibly customized according to specific quality control standards. High Precision: The powerful fitting ability of the point cloud semantic segmentation model combined with the fine granularity of point cloud semantic segmentation is expected to achieve higher recognition accuracy than traditional methods and general image methods, especially for cluster types well covered by the data generation algorithm; Scalability: If new, parameterizable cluster types need to be identified in the future, only the corresponding clustering pattern algorithm needs to be added to the data generation module, and the model needs to be retrained, without the need for laborious data annotation or large-scale modifications to the core framework. Attached Figure Description

[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a flowchart of the defect clustering identification method described in the embodiments of the present invention;

[0058] Figure 2 This is an overall architecture diagram of the point cloud semantic segmentation model described in this embodiment of the invention;

[0059] Figure 3 This is a flowchart of the self-attention mechanism with relative position encoding as described in an embodiment of the present invention;

[0060] Figure 4 This is an example diagram of the background points described in an embodiment of the present invention;

[0061] Figure 5 This is an example diagram of the linear aggregation points described in an embodiment of the present invention;

[0062] Figure 6 This is an example diagram of the grid aggregation points described in an embodiment of the present invention;

[0063] Figure 7 This is an example diagram showing the change in the loss value of the training set as described in an embodiment of the present invention;

[0064] Figure 8 This is an example diagram showing the change in the loss value of the validation set as described in an embodiment of the present invention;

[0065] Figure 9 This is an example diagram illustrating the absence of obvious aggregation defects described in the embodiments of the present invention;

[0066] Figure 10 This is an example diagram illustrating the linear aggregated defects described in an embodiment of the present invention;

[0067] Figure 11 This is an example diagram illustrating the mesh aggregation defect described in an embodiment of the present invention;

[0068] Figure 12 This is an example diagram illustrating different types of aggregation as described in the embodiments of the present invention. Detailed Implementation

[0069] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0070] To address the problems existing in current defect clustering identification methods, this invention provides a defect clustering identification method based on point cloud semantic segmentation. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 The flowchart of the defect clustering identification method is as follows:

[0071] Step 1: Construct a defect cluster dataset.

[0072] Specifically, a parametric algorithm is used to simulate and generate two-dimensional defect point cloud data containing specific defect cluster types. Since the generation rules for some cluster types are much simpler than their judgment rules, a large amount of labeled data can be obtained efficiently and at low cost through parametric programming.

[0073] In this embodiment of the invention, the process of constructing the defect cluster dataset is as follows:

[0074] Step 1.1: Select several representative cluster types that are easy to generate parameters, and use a parameterized algorithm to simulate and generate defect point cloud data.

[0075] Specifically, the clustering types include, but are not limited to, random distribution, linear clustering, and grid clustering.

[0076] Random distribution: Within a preset panel area, a specified number of defect points are randomly generated according to a uniform or Poisson distribution. These defect points do not constitute any specific cluster type and are marked as background points.

[0077] Linear clustering: Parameters include the number of straight lines, the starting point, direction angle, and length of each straight line, the average density of defect points on the line, and the perturbation amplitude of the points (simulating the imperfection of the straight line); the straight line parameters are randomly generated according to the parameters or determined according to preset rules, and defect points are generated on and near the straight line according to the specified density and perturbation. These defect points are marked as linear clustering points.

[0078] Mesh clustering: Parameters include the number of horizontal lines, the number of vertical lines, the line spacing, and the line parameters (same as linear clustering). Based on the parameters, multiple linear clusters parallel to the horizontal and vertical axes are combined, and these defect points are marked as mesh cluster points.

[0079] It should be emphasized that this invention does not limit the specific clustering categories of defect points to be generated. As long as they can be generated through parameterization, this invention can be applied. Furthermore, the specific parameterization algorithm is not a core focus of this invention; users can set and select appropriate parameterization algorithms based on their understanding of various shapes.

[0080] Step 1.2: Perform semantic annotation on defect point cloud data containing different clustering types to construct a defect clustering dataset.

[0081] Specifically, each generated defect point cloud contains a mixture of different cluster types. For example, a defect point cloud might contain a linear cluster, a small clumped cluster, and some random background points. Importantly, during the generation process, each generated defect point is assigned a precise point-level semantic label ci∈{background, linear, clumped, grid-like}.

[0082] By changing the above parameters, large-scale and diverse defect point cloud data and their corresponding point-level labels are generated, forming a defect cluster dataset.

[0083] Step 2: Construct a point cloud semantic segmentation model based on Transformer.

[0084] Specifically, this invention designs a point cloud semantic segmentation model capable of effectively processing two-dimensional defect point cloud data and classifying each defect point. Due to its powerful sequence modeling capabilities and ability to capture contextual information, the point cloud semantic segmentation model is suitable for analyzing the spatial relationships between points in a point cloud. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a diagram of the overall architecture of the point cloud semantic segmentation model.

[0085] In this embodiment of the invention, the point cloud semantic segmentation model includes a point embedding module, a TransformerEncoder module, and a classification output module.

[0086] Specifically, the point embedding module performs feature embedding on the defect point cloud data, converting the defect point cloud data into high-dimensional features to obtain an initial feature vector.

[0087] Specifically, the Transformer Encoder module employs a self-attention mechanism with relative position encoding to encode contextual information and learn features from the initial feature vector to obtain a semantic feature vector; wherein, the relative position encoding mechanism is used to enhance the model's perception of spatial relationships between points.

[0088] Specifically, the classification output module performs semantic category processing on the semantic feature vector of each defect point through a fully connected layer to obtain point-level semantic labels.

[0089] In this embodiment of the invention, the process of constructing a point cloud semantic segmentation model based on Transformer is as follows:

[0090] Step 2.1: Represent the generated defect point cloud data and the defect point cloud data of the industrial panel as a point set about all defect points. , where each defect point P i Using two-dimensional coordinates It can be indicated that, in addition, other initial features can be included as needed, such as the size and brightness of defects, but two-dimensional coordinates are still the core data.

[0091] Step 2.2: Input the point set of all defect points into the point embedding module, and use the point embedding module to map the two-dimensional coordinates of the defect points from the low dimension to the high dimension to obtain the initial feature vector.

[0092] Specifically, for the original defect point coordinates (Possibly combined with other initial features), it is first mapped to a D-dimensional feature space through one or more fully connected layers to obtain the initial feature embedding for each defect point. : ;

[0093] Among them, MLP embed This represents the initial feature embedding process of the fully connected layer, which enables the model to learn a richer initial representation from the original low-dimensional coordinates.

[0094] Step 2.3: Obtain the relative positions of any two defect points, and perform position encoding on the relative positions of any two defect points to obtain the relative position feature vector.

[0095] Specifically, the standard Transformer model does not directly handle the order or spatial position of input elements. For unordered defect point cloud data, the relative spatial relationship between points is crucial. Therefore, this invention employs relative position encoding, where for any two defect points P... i and P j Its relative position vector It is converted into a relative position embedding r by a position encoder (e.g., a small fully connected layer or a predefined trigonometric function code). ij : ;

[0096] in, This represents the relative position embedding process of the position encoder. The relative position embedding process adjusts the attention weights between points in the self-attention mechanism, enabling the model to perceive and utilize the local and global geometry of the point cloud.

[0097] Step 2.4: Input the initial feature vector and the relative position feature vector into the Transformer Encoder module, and use a multi-head attention mechanism combined with relative position encoding to perform attention calculations and nonlinear transformations on the initial feature vector and the relative position feature vector to obtain the semantic feature vector. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 The flowchart is for the self-attention mechanism with relative position encoding.

[0098] Specifically, the core of the model consists of L identical Transformer Encoder layers stacked together. Each Transformer Encoder layer mainly contains two sub-layers: a Multi-Head Self-Attention (MHSA) mechanism and a Feed-Forward Network (FFN).

[0099] Specifically, the multi-head self-attention mechanism allows the model to compute the relationship weights between each defect point in the point cloud and all other defect points.

[0100] For the initial input feature vector: .

[0101] First, a linear transformation is used to obtain the query (Q), key (K), and value (V) matrix: ;

[0102] Among them, W Q W K W v It is a learnable weight matrix. The attention score (also called energy) is typically Q. i and K j The dot product, combined with the relative position embedding r ij Modulation is then performed. A common approach is to incorporate relative positional information as a bias term into the attention score.

[0103] More directly, embed the relative position into r ij Mapped bias b ij Add to attention score: ;

[0104] Among them, b ij Embedding r from relative position i The learned scalar or vector bias.

[0105] Then, the attention weights are calculated using the activation function: ;

[0106] Where, d k It is the dimension of the key (Key, K) and also a scaling factor, which can prevent the gradient from vanishing due to the dot product result being too large.

[0107] Furthermore, the attention weight α ij and the value V of all defect points j The defect point P is obtained by weighted summation. i Attention output: ;

[0108] Specifically, the multi-head attention mechanism divides Q, K, and V into H independent heads, computes the attention in parallel, then concatenates the results from each head and performs a linear transformation. ; ;in, It is the projection matrix of each head and output, and RPE represents the integration of relative position encoding.

[0109] Specifically, the output of the multi-head self-attention mechanism passes through a feedforward neural network, which typically consists of two linear transformation layers and a non-linear activation function, applied independently to the features of each point: ;

[0110] Where W1, b1, W2, and b2 are learnable parameters.

[0111] Specifically, following the multi-head self-attention mechanism and feedforward neural network, residual connections and layer normalization (LN) are used to stabilize the training process and make it possible to build deeper networks: ; Where F is the input of the sublayer, F out It is the output of the sub-layer, that is, the semantic feature vector.

[0112] Step 2.5: Input the semantic feature vector into the classification output module, and use the classification output module to predict the category of the semantic feature vector to obtain point-level semantic labels.

[0113] Specifically, after processing by L Transformer Encoder modules, each defect point P... i Each will result in a context-aware, high-dimensional semantic feature embedding. Finally, these high-dimensional semantic feature embeddings are fed into a classification head consisting of one or more fully connected layers (MLP) to predict which of C predefined semantic categories (such as background, linear clusters, clumped clusters, etc.) the point belongs to, and the output is the log odds of each category: ;

[0114] Furthermore, the log odds of each category are transformed into a probability distribution using an activation function: ;

[0115] in, , , Point P i The predicted probability of belonging to category k.

[0116] Step 3: Train the point cloud semantic segmentation model using the defect aggregation dataset to obtain the defect point semantic segmentation model.

[0117] In this embodiment of the invention, the process of training the point cloud semantic segmentation model using the defect aggregation dataset is as follows:

[0118] Step 3.1: Calculate the loss on the defect cluster dataset using the pointwise cross-entropy loss function;

[0119] Specifically, for a sample containing N points, the loss is calculated using the point-by-point cross-entropy loss function: ;

[0120] Where C is the total number of semantic categories, y ik It is a one-hot vector representing the defect point P. i The true label (if defect point P) i If it belongs to category k, then y ik =1, otherwise y ik =0), It is the model's predicted defect point P i The probability of belonging to category k.

[0121] Step 3.2: Minimize the cross-entropy loss function using deep learning optimization and backpropagation, and update the model parameters to obtain the defect semantic segmentation model.

[0122] Specifically, standard deep learning optimizers and backpropagation algorithms are used to minimize the loss function and update the model parameters.

[0123] Step 4: Use the defect point semantic segmentation model to perform semantic segmentation on the defect point data of the industrial panel to obtain point-level semantic labels, and perform defect clustering identification on the point-level semantic labels to obtain defect clustering identification results.

[0124] In this embodiment of the invention, the process of semantically segmenting defect data of industrial panels using a defect semantic segmentation model is as follows:

[0125] Step 4.1: Preprocess the defect point data of the industrial panel to obtain defect point cloud data;

[0126] Specifically, the preprocessing includes: acquiring defect point data of actual industrial panels, typically a series of defect point coordinates (x, y), and performing normalization processing to make it consistent with the scale of the training data.

[0127] Step 4.2: Input the defect point cloud data into the defect point semantic segmentation model for model inference to obtain the predicted point-level semantic labels.

[0128] Specifically, the model inference includes: inputting preprocessed defect point cloud data into a trained defect point semantic segmentation model, and the model outputs a predicted point-level semantic label (e.g., background point, linear cluster point, etc.) for each defect point.

[0129] In this embodiment of the invention, after obtaining the point-level semantic tags of the defect points, it is necessary to perform defect clustering identification on the point-level semantic tags; wherein, the defect clustering identification process is as follows:

[0130] Step 4.3: Determine the point-level semantic tags corresponding to the industrial panels.

[0131] Step 4.4: If all defect points are randomly distributed, then the industrial panel is determined to not have a specific type of cluster.

[0132] Step 4.5: If the clustering type of defect points is non-random distribution, then it is determined that the industrial panel has the corresponding type of clustering.

[0133] Specifically, if all defect points output by the model are background points, the industrial panel is determined to be OK (no specific clustering); if any defect points are marked as non-background clustering types (e.g., linear, clumpy), the industrial panel is determined to be NG.

[0134] In this embodiment of the invention, if the industrial panel is determined to be NG (Not Good), it is necessary to extract the clustering attributes of the defect points; the process for extracting the clustering attributes is as follows:

[0135] Linear clusters: The linear equations of the points that make up a linear cluster can be fitted, and its length, direction, and number of points contained can be calculated.

[0136] Clustered aggregates: their centroid, approximate extent (such as circumscribed rectangle or convex hull), area, number of points contained, point density, etc. can be calculated.

[0137] Mesh aggregation: It can identify the individual line segments that make up the mesh.

[0138] Specifically, after obtaining the clustering attributes of defect points, the results of defect points can be visualized, that is, the original defect points are colored and displayed according to the semantic labels predicted by the model, which intuitively shows the type, location and range of clustering.

[0139] In this embodiment of the invention, after obtaining the clustering attributes of defect points, custom post-processing rules can be defined. That is, based on the point-level semantic segmentation results, more complex post-processing rules can be flexibly defined according to requirements. For example, it can be set that only linear clusters containing at least k points are considered valid clusters.

[0140] In particular, since the output of defect cluster identification is a refined semantic category for each defect point, it can flexibly adapt to almost all subsequent defect cluster identification needs and statistical analysis tasks. The detailed steps of the embodiments of the present invention are described below using a combination of tables and diagrams, mainly including:

[0141] Step S10: Develop corresponding data generation algorithms based on different aggregation types to generate defect aggregation datasets.

[0142] The purpose of this step is to generate a large amount of two-dimensional defect point cloud data with precise point-level semantic labels using a parametric algorithm, for subsequent model training. In this embodiment, the default size of the industrial panel is set to 100x100 units, and samples containing background points, linear clustered defect points, and grid clustered defect points are mainly generated.

[0143] (1) Background Points Generation:

[0144] Within a predefined panel area, randomly distributed background points are simulated by randomly scattering points on a two-dimensional coordinate system. The number of points generated is determined based on a predefined density parameter (e.g., the number of points per unit area is randomly selected between 0.005 and 0.05). For example, for a 100x100 panel, 50 to 500 background points might be generated. All these points are labeled as background points (e.g., with a label value of 0). See [link to relevant documentation]. Figure 4 , Figure 4 Example image of the background points.

[0145] (2) Linear aggregation point generation: simulates the generation of defect aggregation in a straight line or approximately a straight line.

[0146] First, randomly determine the core parameters of the line: randomly select a starting point within the effective area of ​​the panel (e.g., an area with a certain margin from the edge of the panel, such as 80% of the panel size); randomly select the length of the line (e.g., between 20 and 80 units); and randomly select the direction angle of the line (e.g., between 0 and 2π radians).

[0147] Secondly, the processing depends on whether the line is a polyline: the number of segments can be set (e.g., 1 to 3 segments). If it is a polyline, the gaps are randomly distributed within the total length according to a certain proportion (e.g., the gaps account for 10% to 40% of the total length).

[0148] Then, defect points are generated on and near the (each) straight line segment: the number of generated points is calculated based on a preset point density on the line (e.g., 0.5 to 3.0 points per unit length). The points are initially distributed along the ideal line segment, then line width effects are added to them, i.e., a small random displacement is introduced in the direction perpendicular to the line segment (e.g., the displacement ranges from 0.5 to half the width of 2.0 units). Finally, a small two-dimensional Gaussian noise (e.g., with a standard deviation of 0.3 units) is superimposed on each point to simulate the imperfections of straight lines in the real world.

[0149] All generated linear clusters are labeled as linear clusters (e.g., label value 1). See [link to documentation]. Figure 5 , Figure 5 This is an example diagram of the linear aggregation points.

[0150] (3) Generation of grid cluster points: Simulate the generation of grid-like defect clusters composed of multiple approximately horizontal and vertical line segments intersecting or arranged in parallel.

[0151] First, determine the number of horizontal and vertical lines or their precise coordinates. For example, you can preset a fixed set of y-coordinates as the reference position for the horizontal lines and a fixed set of x-coordinates as the reference position for the vertical lines. Alternatively, you can randomly generate several horizontal and vertical lines (e.g., 2 to 5 each).

[0152] Secondly, for each potential horizontal or vertical line, the decision is made whether to generate the line based on a preset line existence probability (e.g., 0.7).

[0153] If a line exists, it is further divided into several segments: the number of segments inside the line is randomly determined (e.g., 1 to 4 segments), and the start and end points of each segment are determined by randomly selecting tangent points within the entire length of the line, while ensuring that the length of each segment is not less than a minimum proportion of the total length (e.g., 10%).

[0154] Then, on each generated line segment, points are generated in a manner similar to that used for generating linear cluster points: points are generated based on parameters such as point density (e.g., 0.4 to 2.0 points per unit length), line width (e.g., 0.3 to 1.2 units), and noise standard deviation (e.g., 0.2 units).

[0155] All generated mesh clusters are assigned a label (e.g., label value 2). See [link to relevant documentation]. Figure 6 , Figure 6 This is an example diagram of the grid aggregation points.

[0156] (4) Data sample synthesis and defect clustering dataset construction:

[0157] The different types of generated defect points are combined according to certain rules to generate training samples. When generating a training sample, background points are first generated (with a probability of 1.0, i.e., they always exist). Then, one or more linear clusters are attempted to be generated with a certain probability (e.g., each with a probability of 0.3), and then a grid cluster is attempted to be generated with a certain probability (e.g., 0.3). This can generate complex samples containing a single cluster type or a mixture of multiple cluster types and background points. Each generated point has its precise semantic label. By repeating this process extensively and continuously changing the parameters inside each generator, a training set (256,000 samples in this embodiment) and a validation set containing hundreds of thousands of samples are constructed.

[0158] The parameterized generation method described above can efficiently and cost-effectively obtain a large amount of training data with precise point-level semantic labels, overcoming the difficulties of manual annotation.

[0159] Step S20: Construct a point cloud semantic segmentation model based on Transformer.

[0160] In this embodiment, the model adopts a Transformer-based architecture, which is mainly implemented using the Python language and the PyTorch deep learning framework.

[0161] The overall architecture of the point cloud semantic segmentation model includes a point embedding module, a multi-layer Transformer Encoder module integrating relative position encoding, and a final classification output model. The relative position encoding mechanism is used to enhance the model's ability to perceive the spatial geometric relationships between defect points.

[0162] Step S30: Train the model using the generated defect cluster dataset to obtain a defect point semantic segmentation model.

[0163] This step uses the dataset generated in step S10 to train the model constructed in step S20.

[0164] (1) Data loading: Training and validation data are provided to the model in batches (128 samples per batch). The training data is randomly shuffled before the start of each batch, and the samples within a batch are organized to fill samples of different numbers of points with the same length and processed into a format acceptable to the model.

[0165] (2) Optimizer: The Adam optimizer was selected, and the initial learning rate was set to 0.001.

[0166] (3) Loss function: The pointwise cross-entropy loss function is used to calculate the difference between the semantic labels predicted by the model and the actual generated labels.

[0167] (4) Training process: After each training round, the model performance is evaluated on the validation set to monitor the training status and for model selection.

[0168] (5) Training Results: After training, the model's loss values ​​on the training and validation sets will gradually decrease and tend to stabilize. Please refer to [link / reference]. Figure 7 , Figure 7 This is an example graph showing the variation in loss values ​​for the training set. Please refer to [link / reference]. Figure 8 , Figure 8 This is an example graph showing the variation of the loss value of the validation set.

[0169] This step yields a model capable of performing point-level semantic classification on the input defect point cloud data.

[0170] Step S40: Use the defect point semantic segmentation model to perform semantic segmentation on the industrial panel defect data, and perform defect cluster identification based on the segmentation results.

[0171] The model trained in step S30 is applied to actual or newly generated test defect data to identify and analyze defect clusters.

[0172] (1) Data preparation and model inference: Obtain the defect point data to be analyzed (mainly the two-dimensional coordinates of each defect point), perform normalization processing on these coordinate data similar to that during training, input the processed point cloud data into the trained model, and the model will predict a semantic label (background point, linear cluster point or grid cluster point) for each input defect point.

[0173] (2) Results Analysis and Visualization (Example):

[0174] Example 1: Background Point Identification Only. Given a set of randomly distributed defect points, the model expects to identify all points as background points. Based on this result, it can be determined that the panel has no obvious clustered defects (OK). Please refer to [link to relevant documentation]. Figure 9 , Figure 9 The example image without obvious aggregation defects includes, from left to right, the background point input image, the category label image, and the recognition prediction image.

[0175] Example 2: Identification of Single Linear Clusters. Input a set of defect data containing a clear linear cluster and several background points. The model is expected to accurately identify the points constituting the linear cluster and label them as linear cluster points, while the remaining points are labeled as background points. Based on this, the panel can be determined to have a linear cluster defect (NG). Subsequent steps can be taken to fit lines based on the coordinates of these linear cluster points, calculate attributes such as length and direction; please refer to [link to relevant documentation]. Figure 10 , Figure 10The example image showing the linear clustering defect includes, from left to right, the input image of the linear clustering defect, the category label image, and the recognition prediction image.

[0176] Example 3: Identification of Mesh Clusters. Input a set of defect data containing mesh-like clusters (e.g., composed of multiple approximately orthogonal line segments) and several background points. The model is expected to identify the points on each line segment constituting the mesh and mark them as mesh cluster points. Based on this, the panel can be determined to have a mesh cluster defect (NG). See [link to relevant documentation]. Figure 11 , Figure 11 The example image showing the presence of mesh clustering defects includes, from left to right, the input image of the mesh clustering defect, the category label image, and the recognition prediction image.

[0177] Example 4: Complex Scene Recognition with Multiple Clustering Types. Input a set of complex defect data containing linear clusters, partially grid-like clusters, and background points. The model is expected to distinguish between different types of clusters and assign corresponding semantic labels to each point. Please refer to [link to relevant documentation]. Figure 12 , Figure 12 The example image showing different types of clusters includes, from left to right, a complex defect data input image, a category label image, and a recognition prediction image.

[0178] (3) Defect clustering determination and attribute extraction.

[0179]

[0180] Based on the point-level semantic labels output by the model (see Table 1), various post-processing operations can be performed. This approach does not impose any limitations; the following are just a few examples:

[0181] OK / NG Judgment: If all or most points are labeled as background points, then it is judged as OK; otherwise, if a certain number (a threshold can be set) of points are marked as linear cluster points or grid cluster points, then it is judged as NG.

[0182] Clustering type and location: By statistically analyzing the number and spatial distribution of points with different non-background labels, the clustering type (linear, grid-like) and its approximate location can be determined. Please refer to Table 2:

[0183]

[0184] Clustering attribute extraction: For identified linear clusters, algorithms such as Hough transform can be applied to fit the straight line parameters; for grid clusters, horizontal and vertical line segments can be extracted separately, as shown in Table 3.

[0185]

[0186] Custom rules: Based on point-level fine-grained labels, more complex judgment rules can be flexibly defined, such as: a linear cluster must contain at least N points to be considered a valid cluster, etc.

[0187] This embodiment achieves automated and refined identification of defect point clustering patterns on industrial panels through the above steps. By employing parametric data generation and point cloud semantic segmentation technology, this method not only overcomes the bottleneck of labeled data but also provides detailed point-level classification information, offering strong support for subsequent quality control and process improvement.

[0188] Furthermore, in one embodiment, based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a defect clustering identification system based on point cloud semantic segmentation, the system corresponding one-to-one with the method, the system comprising:

[0189] A data acquisition unit is used to construct a defect clustering dataset, which includes defect point cloud data of different clustering types and their corresponding point-level semantic labels.

[0190] The segmentation model unit is used to construct a point cloud semantic segmentation model based on Transformer. The point cloud semantic segmentation model includes a point embedding module, a Transformer Encoder module, and a classification output module.

[0191] The point embedding module embeds features into the defect point cloud data to obtain an initial feature vector; the Transformer Encoder module uses a self-attention mechanism with relative position encoding to encode contextual information and learn features from the initial feature vector to obtain a semantic feature vector; the classification output module processes the semantic feature vector for semantic categories to obtain point-level semantic labels.

[0192] The model training unit uses a defect aggregation dataset to train a point cloud semantic segmentation model to obtain a defect point semantic segmentation model.

[0193] The clustering identification unit uses a defect point semantic segmentation model to perform semantic segmentation on the defect point data of the industrial panel to obtain point-level semantic labels, and performs defect clustering identification on the point-level semantic labels to obtain defect clustering identification results.

[0194] It should be noted that each unit in the defect cluster identification system in this embodiment corresponds one-to-one with each step in the defect cluster identification method in the aforementioned embodiment. Therefore, the specific implementation method and the technical effects achieved in this embodiment can be referred to the implementation method of the aforementioned defect cluster identification method, and will not be repeated here.

[0195] Furthermore, in one embodiment, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the methods in the foregoing embodiments.

[0196] In addition, in one embodiment, this application also provides a computer storage medium storing a computer program, which is executed by a processor to implement the methods in the foregoing embodiments.

[0197] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a device including one or any combination of the above-mentioned memories. The computer may be a variety of computing devices, including smart terminals and servers.

[0198] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0199] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0200] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0201] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0202] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0203] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0204] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A defect cluster identification method based on point cloud semantic segmentation, characterized in that, The method comprises the following steps: constructing a defect aggregation data set comprising defect point cloud data of different aggregation types and corresponding point-level semantic labels; The process of constructing the defect aggregation data set is as follows: generating defect point cloud data by using a parameterized algorithm, wherein the aggregation types of the defect point cloud data include random distribution, linear aggregation, and grid aggregation; performing semantic labeling on the defect point cloud data of different aggregation types to construct the defect aggregation data set; during the generation of the defect point cloud data, each generated defect point is assigned an accurate point-level semantic label; constructing a point cloud semantic segmentation model based on a Transformer, which comprises a point embedding module, a Transformer Encoder module, and a classification output module; The point embedding module performs feature embedding on the defect point cloud data to obtain an initial feature vector; the Transformer Encoder module uses a self-attention mechanism with relative position encoding to encode the context information and learn the features of the initial feature vector to obtain a semantic feature vector; and the classification output module performs semantic category processing on the semantic feature vector to obtain a point-level semantic label; training the point cloud semantic segmentation model using the defect aggregation data set to obtain a defect point semantic segmentation model; performing semantic segmentation on the defect point data of the industrial panel using the defect point semantic segmentation model to obtain a point-level semantic label, and performing defect aggregation identification on the point-level semantic label to obtain a defect aggregation identification result.

2. The defect cluster identification method based on point cloud semantic segmentation according to claim 1, characterized in that, The process of constructing the point cloud semantic segmentation model based on the Transformer is as follows: representing the defect point cloud data as a point set of all defect points, wherein the defect points are represented by two-dimensional coordinates; inputting the point set of all defect points into the point embedding module, and mapping the two-dimensional coordinates of the defect points from a low dimension to a high dimension through the point embedding module to obtain an initial feature vector; obtaining the relative positions of any two defect points, and performing position encoding on the relative positions of any two defect points to obtain a relative position feature vector; inputting the initial feature vector and the relative position feature vector into the Transformer Encoder module, and performing attention calculation and nonlinear transformation on the initial feature vector and the relative position feature vector through the Transformer Encoder module by using a multi-head attention mechanism combined with relative position encoding to obtain a semantic feature vector; inputting the semantic feature vector into the classification output module, and performing category prediction on the semantic feature vector through the classification output module to obtain a point-level semantic label.

3. The defect cluster identification method based on point cloud semantic segmentation according to claim 2, characterized in that, The process of performing attention calculation and nonlinear transformation on the initial feature vector and the relative position feature vector by using a multi-head attention mechanism combined with relative position encoding is as follows: performing linear transformation on the initial feature vector to obtain a learnable weight matrix; The attention score is calculated based on a learnable weight matrix, and a vector bias of a relative position feature vector is added to the attention score to obtain an attention score; The attention weight is calculated based on the attention score, and the attention weight values of all defect points are weighted and added to obtain the attention output of the defect points; The attention output of the defect points is linearly transformed and nonlinearly activated to obtain the linear transformation output of the defect points; The attention output and the linear transformation output of the defect points are residual connected and layer normalized to obtain the semantic feature vector of the defect points.

4. The defect cluster identification method based on point cloud semantic segmentation according to claim 1, characterized in that, The process of training the point cloud semantic segmentation model using the defect aggregation dataset is as follows: The cross-entropy loss function is used to calculate the loss of the defect aggregation dataset; The cross-entropy loss function is minimized using deep learning optimization and back propagation, and the model parameters are updated to obtain the defect point semantic segmentation model.

5. The defect cluster identification method based on point cloud semantic segmentation according to claim 1, characterized in that, The process of using the defect point semantic segmentation model to perform semantic segmentation on the defect point data of the industrial panel is as follows: The defect point data of the industrial panel is preprocessed to obtain defect point cloud data; The defect point cloud data is input into the defect point semantic segmentation model for model inference to obtain the predicted point-level semantic label.

6. The defect cluster identification method based on point cloud semantic segmentation according to claim 1, characterized in that, The process of defect aggregation identification on the point-level semantic label is as follows: The point-level semantic label corresponding to the industrial panel is determined; If the aggregation type of all defect points is random distribution, it is determined that the industrial panel does not exist for a specific type of aggregation; If the aggregation type of the defect points is non-random distribution, it is determined that the industrial panel exists for the corresponding type of aggregation. 7.A defect cluster identification system based on point cloud semantic segmentation, characterized in that, The system comprises: A data acquisition unit for constructing a defect aggregation dataset, the defect aggregation dataset comprising defect point cloud data of different aggregation types and corresponding point-level semantic labels; Wherein, the process of constructing the defect aggregation dataset is as follows: Parameterized algorithms are used to simulate the generation of defect point cloud data, and the aggregation types of the defect point cloud data include random distribution, linear aggregation, and grid aggregation; The defect point cloud data containing different aggregation types is semantically annotated to construct the defect aggregation dataset; the semantic annotation is that during the generation of the defect point cloud data, each generated defect point is assigned an accurate point-level semantic label; A segmentation model unit for constructing a Transformer-based point cloud semantic segmentation model, the point cloud semantic segmentation model comprising a point embedding module, a Transformer Encoder module, and a classification output module; The point embedding module performs feature embedding on the defect point cloud data to obtain an initial feature vector; the Transformer Encoder module uses a self-attention mechanism with relative position encoding to encode the context information and learn the features of the initial feature vector to obtain a semantic feature vector; and the classification output module performs semantic class processing on the semantic feature vector to obtain a point-level semantic label. a model training unit configured to train the point cloud semantic segmentation model using the defect aggregation dataset to obtain a defect point semantic segmentation model; an aggregation identification unit configured to perform semantic segmentation on defect point data of the industrial panel using the defect point semantic segmentation model to obtain point-level semantic labels, and perform defect aggregation identification on the point-level semantic labels to obtain a defect aggregation identification result.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor implements the point cloud semantic segmentation based defect aggregation identification method in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the computer program, and the computer program, when executed by the processor, implements the point cloud semantic segmentation based defect aggregation identification method in any one of claims 1-6. The computer readable storage medium stores the computer program, and the computer program, when executed by the processor, implements the point cloud semantic segmentation based defect aggregation identification method in any one of claims 1-6.

Images