Industrial defect detection method based on point cloud and RGB image fusion

Abstract

The invention provides an industrial defect detection method based on point cloud and RGB image fusion, and belongs to the field of industrial defect detection, and a 3D point cloud and RGB image feature fusion defect detection model mainly comprises a feature fusion module and a final decision module. RGB image features and 3D point cloud features are obtained through the feature extraction module, then the 3D point cloud features are converted into 2D plane features through the feature alignment module, and the 2D plane features are preliminarily aligned with the RGB image features. And the feature fusion module adopts a multi-layer sensor and a full connection layer, so that information correspondence of different modal features is promoted, and the defect detection performance is improved. And the final decision-making module adopts a single-classification support vector machine to make decisions on the defect scores and the segmentation images so as to improve the detection accuracy. According to the method, the accuracy and robustness of industrial defect detection are effectively improved, and the method has important significance for improving the product quality and the production efficiency.

Description

Technical Field

[0001] The present invention relates to the field of industrial defect detection, and in particular to an industrial defect detection method based on point cloud and RGB image fusion. Background Art

[0002] As industrial production scale and automation levels increase, identifying and solving defects becomes increasingly important, and the requirements for defect detection are also increasing. Industrial defects may lead to problems such as product quality degradation, production line shutdowns, and equipment damage, so early detection and handling of defects is crucial to maintaining stable production operations.

[0003] Traditional defect detection methods are mainly based on RGB images and sensor data for analysis. However, these traditional methods may show limitations when facing large-scale and complex production environments. Simply using RGB images may not be able to capture the three-dimensional spatial shape of an object, and using point cloud data alone may not be able to obtain visual information on the surface of an object. Therefore, fusing point cloud and RGB images and making comprehensive use of their respective advantages can improve the accuracy and robustness of industrial anomaly detection. Point cloud data can provide rich three-dimensional structural information, and RGB image data provides color and texture information. Effectively fusing these two types of data can achieve higher accuracy and lower missed detection rates in defect detection. However, the fusion of point cloud and RGB image data still faces some challenges in practical applications, including how to effectively fuse information from two different sources, how to process large-scale point cloud data, and how to effectively extract and utilize three-dimensional feature information.

[0004] The development of computer vision technologies such as deep learning has provided new possibilities for industrial product defect detection methods. The present invention aims to solve the limitations of the prior art mentioned above and provide a new industrial defect detection method based on point cloud and RGB image fusion, which can effectively improve the efficiency and accuracy of defect detection in industrial production. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above background and propose a method for industrial defect detection by fusing RGB image and point cloud data features to improve detection efficiency and accuracy. In order to achieve the above purpose, the technical solution proposed by the present invention is: an industrial defect detection method based on point cloud and RGB image fusion, the method comprising:

[0006] RGB image data and 3D point cloud data of corresponding viewing angles are obtained as training samples, and a defect detection architecture integrating 3D point cloud and RGB image is constructed. The architecture includes a feature extraction module, a feature alignment module, a feature fusion module and a final decision module.

[0007] A set of RGB image data and 3D point cloud data of corresponding viewing angles in the training samples are input into the feature extraction module for feature extraction to obtain RGB image features and 3D point cloud features.

[0008] The feature alignment module includes projection and average pooling. The projection operation projects the extracted point cloud features and parameters onto a 2D plane, and the average pooling operation is used to obtain patch features on the 2D plane feature map. Through point feature projection and average pooling operations, 3D point cloud features are effectively converted into 2D plane features, and the features of the two modalities are preliminarily aligned to improve the efficiency and accuracy of the subsequent feature fusion module.

[0009] The feature fusion module includes multiple multi-layer perceptron layers and a single fully connected layer. This module concatenates 3D point cloud features and RGB image features with contrast loss. This operation can promote more information correspondence between different modal features at the same position, while reducing information correspondence between features at different positions. In this way, not only can the inherent relationship in the training data be fully utilized, but also the features of different modalities can be effectively integrated, thereby improving the performance of defect detection.

[0010] The final decision module includes three storage repositories and two learnable layers. The original color features, position features and fusion features are stored in three different storage repositories respectively, and are used to predict defect scores and segmentation maps. The two learnable layers use single-classification support vector machines to make final decisions on defect scores and segmentation maps. This module can effectively avoid the information loss problem that may occur in the above-mentioned feature fusion module.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] The present invention improves the detection accuracy and robustness of industrial defect detection by cross-modal feature fusion of RGB images and 3D point clouds, and utilizes contrast loss method and final decision method to improve the efficiency and accuracy of feature fusion. Point cloud and RGB images each have unique functions and advantages. Only by effectively fusing the features of the two modalities can better performance be achieved. In the process of designing a data fusion model, it is crucial to fully consider the correlation between different information sources, so as to ensure that the fused system achieves a higher level of optimization in detection performance.

[0013] The industrial defect detection technology based on RGB images and 3D point clouds in the present invention can more comprehensively detect various defects in industrial products, overcoming the limitations of single module detection. Through the fusion of spatial information provided by point cloud data and color and texture information presented by RGB images, the system exhibits higher recognition accuracy and robustness in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of a specific implementation of the industrial defect detection method based on point cloud and RGB image fusion of the present invention;

[0015] Figure 2 is a structural diagram of a feature alignment module of the present invention;

[0016] Figure 3 It is a multimodal feature fusion structure diagram based on contrast loss in the present invention;

[0017] Figure 4 It is the structural diagram of the final decision module in the present invention. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0019] Figure 1 1 is a flowchart of a specific implementation of the industrial defect detection method based on point cloud and RGB image fusion of the present invention. Figure 1 As shown, the specific implementation of the industrial defect detection method based on point cloud and RGB image fusion of the present invention includes:

[0020] S1: Extract 3D point cloud features and RGB image features of corresponding viewpoints and perform preliminary alignment;

[0021] First, a set of RGB image data and 3D point cloud data of corresponding viewpoints in the training sample are input into the feature extraction module for feature extraction to obtain RGB image features and 3D point cloud features. The feature extraction module used for point cloud features is Point Transformer, which divides the point cloud into M groups through farthest point sampling, each group contains S points, and encodes the points in each group into a feature vector and inputs it into Point Transformer, outputting M point features g i , where the point features contained in each point feature group can be regarded as the center point feature c i . Interpolate the feature back to the original point cloud, using the inverse distance weight to interpolate the feature to each point p in the input point cloud j (j∈{1,2,…,N}). This process can be described as:

[0022]

[0023] Then, p′ is transformed into j Project it onto the 2D plane and represent the projected point as p^. If the 2D plane position does not match any point, simply set the position to 0. The feature map obtained by projection is represented as ( is a 2D plane area of ​​the RGB image), and its size is the same as the input RGB image.

[0024] Finally, the average pooling operation is used to obtain the patch features on the 2D plane feature map. This effectively converts the 3D features into 2D plane features and performs preliminary alignment of the features. The feature extraction and alignment structure is as follows: Figure 2 shown.

[0025] S2: Construct 3D point cloud and RGB image fusion model;

[0026] The 3D point cloud and RGB image fusion model is the core component of the industrial defect detection method based on point cloud and RGB image fusion of the present invention, including the feature fusion process and the final decision process.

[0027] In the feature fusion process, a patch-based contrastive loss is used to train the feature fusion module. The goal of this process is to encourage different modal features at the same position to have more corresponding information, while features at different positions have less corresponding information.

[0028] Given RGB feature f rgb And point cloud features f pt , the patch feature is represented as where i is the index of the training sample and j is the index of the patch. Using a multi-layer perceptron (MLP) layer { rgb , χ pt} to extract the interaction information between the two modalities, and use the fully connected layer {σ rgb , σ pt}Map the processed features to the query or key vector. We represent the mapped features as Then we adopt InfoNCE loss for contrastive learning:

[0029]

[0030] Among them, N b is the batch size, N p is the number of non-zero patches.

[0031] The RGB features and 3D features are processed separately using a multi-layer perceptron (MLP), and their outputs are concatenated as the fused patch features, expressed as

[0032]

[0033] Through this process, not only can the inherent relationships in the training data be fully utilized, but also the features of different modalities can be effectively integrated to improve the performance of defect detection. Figure 3 shown.

[0034] In the final decision process, three different repositories are used to store the original color features, position features, and fusion features, respectively. Denote the three repositories, each for predicting defect scores and segmentation maps. Two learnable one-class support vector machines (OCSVM) Da and Ds are then used to make final decisions on the defect score a and segmentation map S. This process can be described as:

[0035]

[0036] Among them, φ, ψ are scoring functions, expressed as:

[0037]

[0038] in, and η is a reweighting parameter.

[0039] Through this process, the information loss problem that may occur in the feature fusion process can be effectively avoided, and the accuracy of defect detection can be improved. The final decision structure is as follows Figure 4 shown.

[0040] S3: Train the model on the industrial defect detection dataset;

[0041] In the process of network training, the industrial defect detection dataset is used to input 3D point cloud features together with RGB image features, guide the feature fusion process to generate fused features and store them in the repository, and finally predict the defect score and segmentation map to obtain the defect detection results. The loss function converges the model by comparing the InfoNCE function added in the loss process. Through iterative training, the network parameters are optimized to accurately predict and segment defects.

[0042] S4: Defect detection result generation;

[0043] In the inference stage, the output features are represented as fused features through a multi-layer perceptron, and the segmentation map of industrial product defects is finally predicted based on the decision of the repository and the two learnable modules.

[0044] S5: Performance evaluation;

[0045] In order to comprehensively evaluate the performance of the model, multiple industrial defect detection evaluation indicators are introduced, including the improved area under the receiver operating characteristic curve (I-AUROC) and the area under the precision-recall operating characteristic curve (AUPRO).

[0046] I-AUROC: Traditional AUROC evaluates the performance of a binary classification model by drawing an ROC curve and calculating the area under the curve. The ROC curve shows the True Positive Rate (TPR) and False Positive Rate (FPR) at different thresholds. The larger the area, the better the model performance. However, in the case of class imbalance, since negative class samples account for the majority, the FPR changes little, which may lead to insufficient AUROC evaluation of the ability to distinguish positive class samples. I-AUROC introduces a weighted method to assign different weights to positive and negative samples to balance the impact of class imbalance.

[0047] AUPRO: AUPRO uses the precision-recall (PR) curve to reflect the performance of the model at different thresholds, emphasizing the ability to accurately predict positive samples. The traditional PR curve evaluates the performance of the classification model by plotting the precision and recall at different thresholds. The area under it, namely AUPR (Area Under the Precision-Recall Curve), is a comprehensive indicator of model performance. Traditional AUPR may have high precision but low recall, or high recall but low precision when dealing with category imbalance. AUPRO optimizes the target to make the comprehensive performance of precision and recall better. By adjusting the threshold, the model can achieve the best precision-recall balance in specific application scenarios.

[0048] In order to better illustrate the technical effect of the present invention, Table 1 shows the experimental results comparison of the present invention with other industrial defect detection network algorithms, including FPFH, PADiM, PachCore and 3D-ST, and their performance is evaluated using I-AUROC and AUPRO.

[0049] Table 1 Comparison of experimental results of the present invention and other industrial defect detection algorithms

[0050] FPF PADiM PatchCore 3D-ST The network of the present invention I-AUROC 0.918 0.975 0.876 0.862 0.994 AUPRO 0.952 0.980 0.901 0.950 0.988

[0051] It can be observed from Table 2 that in terms of I-AUROC and AUPRO indicators, the industrial defect detection network of the present invention has achieved better results and can effectively improve the precision and accuracy of industrial product defect detection.

[0052] In summary, the method of the present invention can not only overcome the limitations of single-mode industrial defect detection, but also improve the efficiency and accuracy of defect detection in industrial production. In terms of data fusion, it can effectively solve the inevitable information loss problem and achieve higher-precision industrial defect detection. The method of the present invention has far-reaching significance for improving the quality of industrial production in my country, reducing production costs, and promoting the development of industrial manufacturing technology.

Claims

1. An industrial defect detection method based on point cloud and RGB image fusion, characterized in that: The steps include: S1: Extract 3D point cloud features and RGB image features from the corresponding perspective and use the projection method for preliminary alignment; S2: Construct a 3D point cloud and RGB image fusion model, which includes a feature fusion process and a final decision process, adopts a patch-based contrast loss function, and optimizes the fusion result through multiple repositories and two learnable modules; S3: training the fusion model on the data set, and optimizing the network parameters through iterative training to accurately predict and segment defects; S4: In the inference stage, the output features are represented as fused features through a multi-layer perceptron, and the segmentation map of industrial product defects is finally predicted based on the decision of the repository and the two learnable modules; S5: Comprehensively evaluate the model performance, using the improved area under the receiver operating characteristic curve (I-AUROC) and the area under the precision-recall operating characteristic curve (AUPRO) as evaluation indicators.

2. The industrial defect detection method based on point cloud and RGB image fusion according to claim 1, characterized in that: The feature extraction and alignment include farthest point sampling, feature interpolation and feature projection, which are used to extract features of 3D point cloud and RGB image and perform preliminary alignment, providing a solid foundation for effective feature fusion of the two modalities.

3. The industrial defect detection method based on point cloud and RGB image fusion according to claim 1, characterized in that: The feature fusion module includes a multi-layer perceptron, a fully connected layer and a contrast loss, which ensures that different modal features at the same position have more corresponding information, while features at different positions have less corresponding information, thereby optimizing the model's ability to fuse different modal features.

4. The industrial defect detection method based on point cloud and RGB image fusion according to claim 1, characterized in that: The two learnable modules make final decisions on the defect score and the segmentation map through a single-classification support vector machine.

5. The industrial defect detection method based on point cloud and RGB image fusion according to claim 4, characterized in that: The final decision module includes three repositories and two learnable modules: three different repositories are used to store the original color features, position features and fusion features respectively, and each repository is used to predict the defect score and segmentation map respectively; then two learnable single-classification support vector machines are used to make decisions to obtain the final defect detection results, ensuring that the model can effectively avoid the problem of information loss in the fusion detection process.

6. The industrial defect detection method of point cloud and RGB image fusion according to claim 1, a computer program product for executing the method according to any one of claims 1 to 4, characterized in that: The product includes a storage medium and computer executable instructions stored on the medium, the instructions being configured to implement the steps of the method when executed.

Images