Three-dimensional defect detection method based on global-local feature fusion and frequency selectivity enhancement

Abstract

The invention provides a three-dimensional defect detection method based on global-local feature fusion and frequency selectivity enhancement. According to the method, two-dimensional image features and three-dimensional point cloud features are combined. Firstly, a three-dimensional point cloud is projected to a two-dimensional plane, and feature extraction is performed on multiple views by using a two-dimensional feature extraction technology; according to the method, a Transform framework and Kolmogorov-ArnoldNetworks (KAN) are innovatively combined, so that the accuracy of two-dimensional feature extraction is remarkably improved, and the model reasoning speed is remarkably increased. In an up-sampling stage, a frequency selection enhancement module is introduced to enhance the consistency of features and the definition of boundaries. In three-dimensional point cloud feature extraction, global feature extraction is added, complementation of global and local features is achieved, and the point cloud anomaly detection performance is improved. And finally, fusing all the features to construct a memory bank, accurately positioning defect points by matching the memory bank during testing, ensuring that defect positions are accurately extracted, and providing an efficient and accurate solution for three-dimensional defect detection.

Description

Technical Field

[0001] This invention belongs to the field of point cloud processing technology, specifically relating to a method for detecting defects in three-dimensional point clouds. Background Technology

[0002] Surface defects in industrial products refer to any incomplete or abnormal parts on the product surface compared to a standard sample, often stemming from production equipment malfunctions or human error. These defects not only impair product functionality and normal use but also pose safety hazards. Given the inefficiency and high cost of traditional manual inspection methods, there is an urgent need to develop machine vision inspection technologies that meet industry requirements to address these issues.

[0003] The general steps for 3D defect detection are to first extract features from the data, compare the features from training and testing, and segment areas with anomalies. However, most of these methods only extract single features, such as global or local features, and the completeness of feature extraction needs further improvement. Furthermore, industrial 3D data contains various types of defects, and detecting different types of defects requires setting numerous parameters; therefore, detecting point clouds with abnormal structures remains challenging. Summary of the Invention

[0004] This invention provides a method for three-dimensional defect detection, comprising the following steps:

[0005] Step 1: Project the 3D point cloud data onto a 2D plane to create a multi-view view, and use the Kolmogorov-Arnold (KAN) + Transformer module to extract 2D features. The extracted features are then upsampled using a frequency selection enhancement module.

[0006] Step 2: Extract local geometric features from the original 3D point cloud using feature descriptors and extract global features using Point-NN based on clustering methods, and then fuse these extracted features.

[0007] Step 3: During training, all extracted features are added to the memory bank. During testing, features are extracted from the data using the same method and compared with the information in the memory bank to identify the defective parts of the data.

[0008] Optionally, the 3D point cloud can be projected onto multiple 2D images at different angles by designing different rotation matrices to reduce the loss of point cloud information.

[0009] Preferably, in order to better preserve previously learned information when processing sequence data, the KAN module is used to replace the MLP layer in the original Transformer.

[0010] Preferably, the KAN module employs a learnable activation function, replacing each weight parameter with a parameterized univariate function.

[0011] Optionally, any number of the adaptive low-pass filter generator, offset generator, and adaptive high-pass filter generator in the frequency module can be used.

[0012] Preferably, the Local Geometric Feature Descriptor (FPFH) constructs a histogram (SPFH) by calculating three geometric relationship angles between the query point and each point in the k-neighborhood, in order to statistically analyze the distribution of these geometric relationships, and ultimately achieves an accurate characterization of the local geometric features of the point cloud.

[0013] Preferably, Point-NN, based on clustering methods, is used for global feature extraction. For a given initial point cloud P containing n points, Point-NN first extracts features for each point using Positional Encoding (PosE), forming a combination of point and feature. After feature extraction, a k-NN search is used to select the center point, ensuring that the center points are evenly distributed in space and preserving the geometric structure of the dataset to the greatest extent. After determining the center point, the neighboring points around each center point are defined as its neighborhood. The relative position of each point in the neighborhood to the center point is calculated, and this relative position information is input into PosE to extract new features. Then, the extracted new features are combined with the features of the point itself and the features of the center point to form a new feature representation of the point containing spatial position information. The features of all points are aggregated to form the feature representation of each neighborhood.

[0014] Preferably, the features extracted by the local geometric feature descriptor FPFH and Point-NN are aggregated into a new point cloud feature and added to the memory.

[0015] Preferably, during segmentation, a memory M containing features is constructed.

[0016] Optionally, before incorporating new features, a Coreset sampling technique can be used to construct a "core set" by randomly selecting some points from the original dataset. These points are representative samples from the original dataset, thus reducing computational cost.

[0017] Preferably, the RANSAC algorithm is used to register the test 3D object. After registration, if at least one point cloud is determined to be an anomaly, the test 3D object is predicted to be an anomaly, and its point-level anomaly segmentation is further calculated.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] This invention proposes a 3D point cloud defect detection method (FKF-Net) based on a frequency-selective enhanced Kolmogorov-Arnold Transformer architecture and the fusion of global and local information. FKF-Net first projects the 3D point cloud onto a multi-view image, then combines the Transformer framework and Kolmogorov-Arnold Networks (KAN) for efficient 2D feature extraction. During the upsampling stage, a frequency-selective enhancement module is designed to improve feature consistency and boundary clarity. For 3D point cloud feature extraction, this study combines global feature extraction methods with geometric feature extraction methods based on local feature descriptors, thus achieving complementarity between global semantic information and local geometric features. Finally, feature fusion forms a comprehensive feature representation for 3D defect detection. Extensive experimental results show that, across all categories of MVTec 3D-AD, FKF-Net achieves an average P-ROC of 98.1%, a P-AUPR of 44.8%, and a P-PRO of 93%, all of which outperform other related methods. The ablation experiments demonstrated the effectiveness of each module, and the comparative experiments demonstrated FKF-Net's ability to handle 3D defect detection.

[0020] In addition to the objectives, features, and advantages described above, the present invention has other objectives, features, and advantages. The invention will now be described in further detail with reference to the figures. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0022] Figure 1 This is a schematic diagram of the overall architecture for feature extraction from two-dimensional images and three-dimensional point clouds during the training and testing process in this embodiment of the invention;

[0023] Figure 2 This is a schematic diagram of the structure for extracting two-dimensional features using KAN+Transformer in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the upsampling frequency selection enhancement module structure in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the point cloud global feature extraction method in an embodiment of the present invention;

[0026] Figure 5This is a schematic diagram comparing the detection results obtained by the method provided by the present invention with the detection results obtained by the method provided in the prior art. Detailed Implementation

[0027] To make the above-mentioned objectives, features, and advantages of the present invention clearer and easier to understand, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the accompanying drawings of the present invention are all in a simplified form and use non-precise proportions, and are only used to facilitate and clearly assist in illustrating the implementation of the present invention; the "several" mentioned in the present invention are not limited to the specific number shown in the examples in the accompanying drawings; the orientations or positional relationships indicated by terms such as "front," "middle," "rear," "left," "right," "up," "down," "top," "bottom," and "center" mentioned in the present invention are all based on the orientations or positional relationships shown in the accompanying drawings of the present invention, and do not indicate or imply that the device or component referred to must have a specific orientation, nor should they be construed as limitations on the present invention.

[0028] This example demonstrates:

[0029] The present invention provides a method for three-dimensional defect detection, comprising the following steps:

[0030] Step 1: Project the 3D point cloud data onto a 2D plane to create a multi-view view, and use the KAN+Transformer module to extract 2D features. The extracted features are then upsampled using the frequency module.

[0031] Step 2: Extract local geometric features from the original 3D point cloud using feature descriptors and extract global features using Point-NN based on clustering methods, and then fuse these extracted features.

[0032] Step 3: During training, all extracted features are added to the memory bank. Features are also extracted from the data for which features need to be found, and compared with the information in the memory bank to identify the defective parts of the data.

[0033] Optionally, the 3D point cloud is projected onto multiple 2D images at different angles by designing different rotation matrices to reduce the loss of point cloud information, where V represents a set of multiple viewpoints and represents the image rendered from viewpoint v.

[0034] I v =Project(P, v), v∈V (1)

[0035] Optionally, to better preserve previously learned information when processing sequential data, a KAN module is used to replace the MLP layer in the original Transformer. Compared to the traditional MLP module, the activation function of the KAN module is located on the edges between connections in each layer rather than on the neurons, meaning its activation function is learnable. Based on this, this invention uses a non-linear kernel function to replace the linear function on the MLP edges, while setting fine-grained nodes to approximate accuracy.

[0036] Optionally, the KAN module employs a learnable activation function, replacing each weight parameter with a parameterized univariate function, such as... Figure 2 As shown.

[0037] Preferably, the feature map input to the frequency selection enhancement module is first processed by a pointwise convolutional layer, and then upsampled to obtain a feature map of size C×8H×8W. These feature maps are then fed into an adaptive Hamming window filter (AHWF). Inside the AHWF, a mask is generated using a kernel normalizer and a Hamming window. The resulting mask and the processed original feature map are then fed into a CARAFE function for upsampling. The output of this module is the final upsampling result.

[0038] Optionally, the frequency selection enhancement module can be used multiple times.

[0039] Preferably, the Local Geometric Feature Descriptor (FPFH) constructs a histogram (SPFH) by calculating three geometric relationship angles between the query point and each point in its k-neighborhood to statistically analyze the distribution of these geometric relationships, ultimately achieving an accurate characterization of the local geometric features of the point cloud. Specifically, the extraction method described above is as follows:

[0040] ① First, calculate the query point p. q With each point p in the k-neighborhood k Based on the three geometric relationships between them, a histogram (SPFH) is constructed to statistically analyze the distribution of these geometric relationships.

[0041] ② For p k Generate the corresponding SPFH using the same method.

[0042] ③ Perform weighted statistics on each SPFH within the neighborhood to generate p. q FPFH is used to achieve accurate characterization of local geometric features of point clouds.

[0043]

[0044] ④ The local features of each point are represented by F. FPFH To indicate:

[0045] F FPFH=SPFH1,SPFH2,...,SPFH 33 (3)

[0046] Preferably, Point-NN is used for global feature extraction. For a given initial point cloud P containing n points, Point-NN first uses positional encoding (PosE) to extract features for each point, forming a combination of point and feature (p... i ,f i After feature extraction is complete, the center point (p) of the KNN search is selected using the FPS strategy. c ,f c This ensures that the center points are uniformly distributed in space, preserving the geometric structure of the dataset to the greatest extent possible. After determining the center points, each center point's adjacent points are defined as its neighborhood G. Within each neighborhood, the relative position Δp of each point in the neighborhood to the center point is calculated. i And this relative position information is input into PosE to extract new features f(Δp). i Then, the extracted new features are compared with the features f of the point itself. i and the characteristics f of the center point ci By combining these elements, a new feature representation of a point containing spatial location information is formed, where “×” represents element-wise multiplication.

[0047]

[0048] The features of all points are aggregated to form the feature representation of each neighborhood. Finally, the features of all neighborhoods are aggregated using the same pooling method to obtain the feature representation of the entire point cloud. The entire process requires no parameters or training.

[0049]

[0050] Preferably, the features extracted by the local geometric feature descriptor FPFH and Point-NN are aggregated into a new point cloud feature and added to the memory.

[0051] Preferably, during segmentation, a memory M containing features is constructed.

[0052]

[0053] Preferably, before incorporating new features, the Coreset sampling technique is used to build a memory bank by randomly selecting some points from the original dataset, which are representative samples in the original dataset, in order to reduce the amount of computation.

[0054] Preferably, the RANSAC algorithm is used to register the test 3D object. After registration, if at least one point cloud is determined to be an anomaly, the test 3D object is predicted to be an anomaly, and its point-level anomaly segmentation is further calculated.

[0055] Preferably, when calculating the outlier score, for each point p in the test point cloud... test Find the least dissimilar point m among its nearest neighbors in the memory. test,* And the most similar point m * The former is used for anomaly detection, while the latter is used to map points in the test point cloud to points in the memory database. Then, m is calculated. test,* and m * The Euclidean distance is used to measure the difference *s* between the test point and its nearest neighbor. These distances are then reweighted to enhance the model's robustness, yielding an anomaly score *s* for each point, thus more accurately identifying anomalies or defects. Aggregating the scores of each point yields the overall point cloud anomaly score, used for calculating subsequent evaluation metrics. After obtaining the anomaly score *s*, a threshold is applied to determine whether a point is an anomaly, and then the anomaly is added to a set.

[0056]

[0057] m * = arg min m∈M ||m test -m||2 (8)

[0058] s * =||m test,* -m * ||2 (9)

[0059]

[0060] Example:

[0061] In this application, the MVTec 3D-AD dataset was used, comprising a total of 4300 high-resolution 3D coordinate scan data points covering 10 different item categories. To eliminate the influence of the background, the data was preprocessed. First, all noise was removed using PCD, and then the RANSAC and DB-Scan algorithms from the Open 3D library were used to perform planar fitting on the data to filter out background information.

[0062] Comparative example:

[0063] FKF-Net was compared with other state-of-the-art methods to further validate the effectiveness of the proposed method. The comparison methods used in this study were derived from advanced approaches including 3D-ST, BTF, M3DM, CPMF, and AST. In comparison, FKF-Net significantly outperformed existing techniques, achieving an impressive P-ROC of 98.1%. Specifically, FKF-Net achieved the highest P-ROC in 7 out of 10 categories and ranked second in the remaining 3. Table 1 effectively demonstrates the comprehensive superiority of FKF-Net across various scenarios. Similarly, the P-AUPR results in Table 2 also surpassed the selected methods.

[0064] Table 1. P-ROC results of different methods

[0065]

[0066] Table 2. P-AUPR results using different methods

[0067]

[0068] Table 3. P-AUPR results of different methods

[0069]

[0070] Table 3 presents the P-PRO experimental results for different methods, with FKF-Net achieving a better average than the others. Similarly, in terms of P-PRO, previous work by BTF showed that handcrafted descriptors are highly effective, achieving a significant P-PRO of 92.43% using FPFH features. However, FKF-Net not only reaches this performance level but also surpasses it with a P-PRO of 93.02%. This improvement highlights FKF-Net's enhanced ability to accurately locate anomalies, which is crucial for practical applications across various fields.

[0071] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A three-dimensional defect detection method based on global-local feature fusion and frequency-selective enhancement, characterized in that, Includes the following steps: (1) Preprocess the dataset: Point cloud data generally falls into two categories: depth maps and unordered point clouds with coordinates (x, y, z). This invention requires two-dimensional image data, and the projection methods for the two types of data differ: for depth maps, the depth values ​​of each pixel are directly extracted and projected into an image; for unordered point clouds, downsampling and structuring are performed first, and then projection into an image is performed. Simultaneously, both types of data must retain the original (x, y, z) values ​​of each point as input for step (3). (2) Constructing a feature extraction method on a two-dimensional plane: The method consists of two steps: First, the input image is encoded and compressed into image data of the original size of 1 / 8×1 / 8, which is then fed into a deep learning model to obtain a feature map; then, the low-dimensional features are restored to the original image resolution through an upsampling model to ensure that the subsequent registration task can be performed in a unified dimension. (3) Constructing a feature extraction method in three-dimensional space: The method consists of three steps: First, geometric features are calculated independently point by point, preserving local details; then, the scattered point features are abstracted into a compact global representation through clustering; finally, the two parts of features are spliced ​​together according to the channel dimension to form a feature that combines local and global information. (4) By constructing a memory bank: The training data obtained in step (1) is put into step (2) to obtain the features of the image on the two-dimensional plane; at the same time, the above data is put into step (3) to obtain the features of the point cloud in the three-dimensional space; the above two features are integrated to form the features of all training data in multiple dimensions, and this part of the data is used as the detection basis for the subsequent step (5). (5) Three-dimensional defect detection: The test data obtained in step (1) is put into step (2) to obtain the features of the image on the two-dimensional plane; then the above data is put into step (3) to obtain the features of the point cloud in the three-dimensional space; the above two features are integrated to form the features of a single test data in multiple dimensions; the obtained features need to be aligned with the database in step (4); After alignment, the features of each part need to be compared with the features already in the memory to calculate the anomaly score; based on the anomaly score, the abnormal region can be detected and segmented.

2. The method for three-dimensional defect detection according to claim 1, characterized in that, In step 1, different rotation matrices are designed when the 3D point cloud is projected onto the 2D plane to generate images at multiple different angles.

3. The method for three-dimensional defect detection according to claim 1, characterized in that, In step 2, the deep learning model used is the vision-based Transformer model, and the existing MLP is modified based on it to adapt to the specific circumstances of this problem. The specific process is as follows: ① The vision-based Transformer model is ViT. The input image will be divided into multiple unordered small parts, and then these small images will be fed into the Transformer to extract features. ② To better preserve previously learned information when processing sequence data, a KAN module with learnable activation functions is used to replace the MLP layer in the original ViT encoder. Compared to the traditional MLP module, the activation function of the KAN module is located on the edges between connections in each layer rather than on the neural nodes, meaning its activation function is learnable. Based on this, this invention uses a non-linear kernel function to replace the linear function on the MLP edges, while setting fine-grained nodes to approximate accuracy. ③ The input data for the entire model is 27×3×224×224, where 27 represents the number of all 2D images projected from a point cloud. This data is fed into convolutional and flattening layers to obtain a 27×784×64 dataset. Subsequently, position embedding, dropout, and layer normalization are applied to the data before it is input into the encoder. The encoder then outputs the feature map obtained by the deep learning model. The shape of this output feature map is [27×64×28×28]×8.

4. The encoder according to claim 2, characterized in that, It consists of two residual connection modules, four layer normalization modules, a KAN module, and a multi-head attention module. The KAN module is characterized by being composed of two KAT_Group modules, two dropout modules, and two fully connected layers. The input and output of the KAN module are both 27×784×64. KAT_Group encapsulates the activation function and linear transformation into a lightweight unit and shares activation weights across the entire group of neurons, thereby significantly compressing the number of parameters and computational overhead while maintaining the original model performance, making the overall architecture more scalable.

5. The method for three-dimensional defect detection according to claim 1, characterized in that, The upsampling portion in step 2 uses an innovative frequency selection enhancement module that includes pointwise convolution, adaptive Hamming window filters, and CARAFE upsampling. The input feature map mentioned in claim 3 is first processed through a pointwise convolutional layer, followed by upsampling to obtain a feature map of size C×8H×8W. These feature maps are then fed into an adaptive Hamming window filter. Inside this filter, a mask is generated using a kernel normalizer and a Hamming window. The generated mask and the processed original feature map are then fed into a CARAFE function for upsampling.

6. The method for three-dimensional defect detection according to claim 1, characterized in that, The local feature extraction part in step 3 uses the local feature descriptor FPFH, and the specific process is as follows: ① Calculate the query point p q p with each point in the k-neighborhood k From the three geometric relationship perspectives, a histogram (SPFH) is constructed to statistically analyze the distribution of these geometric relationships for p. k Generate the corresponding SPFH using the same method. ② Perform weighted statistics on each SPFH within the neighborhood to generate p. q FPFH is used to achieve accurate characterization of local geometric features of point clouds. 。 7. The method for three-dimensional defect detection according to claim 1, characterized in that, The global feature extraction part in step 3 uses a non-parametric network, which consists of farthest point sampling, k-nearest neighbor search, feature aggregation, and pooling operations. The clustering method mentioned is KNN, and the specific process is as follows: ① First, use positional encoding to extract features for each point in the point cloud, forming a combination of point and feature (p i ,f i ). ② Select the center point of KNN search using the FPS strategy (p c ,f c This ensures that the center point is uniformly distributed in space, preserving the geometric structure of the dataset to the greatest extent possible. Within each neighborhood, the relative position Δp of each point in the neighborhood to the center point is calculated. i And this relative positional information is input into the positional encoding in step ① to extract new features f(Δp). i ). ③ Compare the extracted new features with the features f of the point itself. i and the characteristics f of the center point ci By combining these elements, a new feature representation of a point that includes spatial location information is formed. The features of all points are aggregated to form the feature representation of each neighborhood. Finally, the same pooling method is used to aggregate the features of all neighborhoods to obtain the feature representation of the entire point cloud. 。 8. The method for three-dimensional defect detection according to claim 1, characterized in that, In step 4, when incorporating new features into the memory, the Coreset sampling technique is used to randomly select some representative samples from the original dataset to construct a "core set" in order to reduce the amount of computation during testing.

9. The method for three-dimensional defect detection according to claim 1, characterized in that, In step 5, the registration method is RANSAC.

10. The method for three-dimensional defect detection according to claim 1, characterized in that, The method for calculating the anomaly score in step 5 is as follows: for each point p in the test point cloud... test Find the least dissimilar point m among its nearest neighbors in the memory. test,* And the most similar point m * The former is used for anomaly detection, while the latter is used to map points in the test point cloud to points in the memory database. Then, the Euclidean distance between these two points is calculated to measure the degree of difference (s*) between the test point and its nearest neighbor. m * = arg min m∈M ||m test -m||2 (5) s * =||m test,* -m * ||2 (6)。

Images