A method, system, equipment, medium, and product for three-dimensional anomaly detection of industrial products.

Abstract

This application discloses a method, system, device, medium, and product for 3D anomaly detection of industrial products, relating to the industrial manufacturing field. The method includes: preprocessing 3D point cloud data and 2D RGB images of the industrial product; extracting 3D point cloud features based on the preprocessed 3D point cloud data using a 3D point cloud feature extraction model constructed with stacked Point Transformer V3 as its backbone; extracting 2D image features based on the preprocessed 2D RGB image data using an image feature extraction model; performing multimodal feature fusion on the 3D point cloud features and 2D image features; determining an anomaly score based on the fused features and a preset mean of the fused features; and determining the anomaly detection result based on the relationship between the anomaly score and a set threshold. This application can efficiently process 3D point clouds, accurately extract 2D features, and achieve collaborative multimodal fusion.

Description

Technical Field

[0001] This application relates to the field of industrial manufacturing, and in particular to a method, system, equipment, medium, and product for three-dimensional anomaly detection of industrial products. Background Technology

[0002] With the trends of mass customization and agile manufacturing, industrial production lines are increasingly demanding high-precision and real-time product quality inspection. As a core component of quality control, industrial anomaly detection must promptly identify anomalies such as geometric deformation, surface contamination, and structural defects to prevent production accidents and economic losses. However, existing inspection methods suffer from the following key bottlenecks: (1) Limitations of single-modal detection: Traditional 2D image detection methods rely on texture and color information. Due to the loss of 3D geometric information, they cannot identify three-dimensional defects such as depressions and protrusions. Although 3D point cloud detection methods can obtain spatial geometric features, they are affected by the irregularity and sparsity of point clouds. Traditional processing methods (such as K-nearest neighbor algorithm neighborhood search and voxelization) have problems of low computational efficiency and large memory overhead, which are difficult to meet the real-time response requirements of agile production lines.

[0003] (2) Labeling cost and sample scarcity: There are very few abnormal samples in industrial scenarios, and manual labeling (especially 3D point cloud labeling) is inefficient and costly, making it difficult for detection models that rely on supervised learning to adapt to flexible production with frequent product iterations.

[0004] (3) Poor multimodal fusion synergy: Existing multimodal (2D+3D) detection methods mostly extract features separately and then simply stitch them together, without considering spatial misalignment and high-dimensional redundancy interference between modalities. This results in weak discriminative power of fused features, slow inference speed, and an inability to balance detection accuracy and real-time performance.

[0005] While deep learning techniques (such as generative models and self-supervised learning) have provided insights into unsupervised anomaly detection, existing 3D point cloud feature extraction methods (such as PointNet and VoxelNet) still suffer from efficiency bottlenecks. Multimodal fusion methods (such as the traditional Transformer) are not optimized for industrial scenarios and struggle to meet the dual demands of high precision and real-time performance in agile manufacturing. Therefore, there is an urgent need in this field for an unsupervised detection method that can efficiently process 3D point clouds, accurately extract 2D features, and achieve collaborative multimodal fusion. Summary of the Invention

[0006] The purpose of this application is to provide a method, system, equipment, medium and product for three-dimensional anomaly detection of industrial products, which can efficiently process 3D point clouds, accurately extract 2D features and achieve collaborative multimodal fusion.

[0007] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for three-dimensional anomaly detection of industrial products, including: Acquire 3D point cloud data and 2D RGB images of industrial products, and preprocess the 3D point cloud data and 2D RGB images of industrial products to obtain 3D point cloud preprocessed data and 2D RGB image preprocessed data. A 3D point cloud feature extraction model is constructed using stacked Point Transformer V3 as the backbone; The 3D point cloud feature extraction model is used to extract 3D point cloud features based on the preprocessed 3D point cloud data. An image feature extraction model is constructed, and the image feature extraction model is used to extract 2D image features based on the 2D RGB image preprocessing data; Multimodal feature fusion is performed on the 3D point cloud features and the 2D image features to obtain fused features; An anomaly score is determined based on the fusion features and the mean of the preset fusion features, and an anomaly detection result is determined based on the relationship between the anomaly score and a set threshold; the anomaly detection result includes anomaly determination result and anomaly region location result.

[0008] Secondly, this application provides a three-dimensional anomaly detection system for industrial products, comprising: The data acquisition module is used to acquire 3D point cloud data and 2D RGB images of industrial products; The data processing module, connected to the data acquisition module, is used to preprocess the 3D point cloud data and 2D RGB image of the industrial product to obtain 3D point cloud preprocessed data and 2D RGB image preprocessed data. The PFE feature extraction module is connected to the data processing module. It is used to construct a 3D point cloud feature extraction model with stacked Point TransformerV3 as the backbone, and to extract 3D point cloud features based on the 3D point cloud preprocessed data using the 3D point cloud feature extraction model. The IFE feature extraction module is connected to the data processing module and is used to construct an image feature extraction model and extract 2D image features based on the 2D RGB image preprocessing data using the image feature extraction model. A multimodal fusion module is connected to the PFE feature extraction module and the IFE feature extraction module respectively, and is used to perform multimodal feature fusion on the 3D point cloud features and the 2D image features to obtain fused features; An anomaly detection module, connected to the multimodal fusion module, is used to determine an anomaly score based on the fusion features and the mean of preset fusion features, and to determine the anomaly detection result based on the relationship between the anomaly score and a set threshold. The result output module is connected to the anomaly detection module and is used to visualize the anomaly detection results; the anomaly detection results include anomaly determination results and anomaly area location results.

[0009] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for detecting three-dimensional anomalies in industrial products.

[0010] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method for detecting three-dimensional anomalies in industrial products.

[0011] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described industrial product three-dimensional anomaly detection method.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, system, device, medium, and product for 3D anomaly detection of industrial products. By adopting a 3D point cloud feature extraction model constructed with stacked Point Transformer V3 as the backbone, it can efficiently process 3D point clouds. By adopting the constructed image feature extraction model, it can accurately extract 2D features. By performing multimodal feature fusion of 3D point cloud features and 2D image features, it can achieve collaborative multimodal fusion, thereby solving the problems of geometric information loss, efficiency bottleneck of 3D point cloud methods, and poor multimodal fusion collaboration in traditional 2D anomaly detection methods in agile industrial manufacturing scenarios, achieving a balance between high precision and high real-time performance. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating a three-dimensional anomaly detection method for industrial products provided in an embodiment of this application; Figure 2 This is a schematic diagram of a point feature extraction process provided in an embodiment of this application; Figure 3 This is a schematic diagram of an image feature extraction process provided in an embodiment of this application; Figure 4 This is a schematic diagram of the array of detection results on the MVTec3D-AD dataset provided in an embodiment of this application; Figure 5 This is a schematic diagram of the comparison array of ROC curves for each category on the MVTec3D-AD dataset provided in an embodiment of this application; Figure 6 This is a schematic diagram of the comparison array of PR curves for each category on the MVTec3D-AD dataset provided in an embodiment of this application; Figure 7 A functional module diagram of an industrial product three-dimensional anomaly detection system provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] In one exemplary embodiment, this application provides a method for three-dimensional anomaly detection of industrial products. This method is executed by a computer device, specifically a terminal or server, or both. In this embodiment, the method is described using a server as an example. Figure 1 As shown, the method includes: Step 100: Acquire 3D point cloud data and 2D RGB images of industrial products, and preprocess the 3D point cloud data and 2D RGB images of industrial products to obtain 3D point cloud preprocessed data and 2D RGB image preprocessed data.

[0018] Step 101: Construct a 3D point cloud feature extraction model using stacked Point Transformer V3 as the backbone.

[0019] Step 102: Use a 3D point cloud feature extraction model to extract 3D point cloud features based on the 3D point cloud preprocessing data.

[0020] Step 103: Construct an image feature extraction model and use the image feature extraction model to extract 2D image features based on 2D RGB image preprocessing data.

[0021] Step 104: Perform multimodal feature fusion on 3D point cloud features and 2D image features to obtain fused features.

[0022] Step 105: Determine the anomaly score based on the fusion features and the preset mean of the fusion features, and determine the anomaly detection result based on the relationship between the anomaly score and a set threshold. The anomaly detection result includes the anomaly determination result and the anomaly region location result.

[0023] By implementing steps 100-105 above, this application constructs an unsupervised dual-stream collaborative network architecture (i.e., 3D point cloud feature extraction model + image feature extraction model + multimodal feature fusion), which can meet the real-time detection requirements of agile manufacturing and can be applied to 3D defect detection of various products such as industrial parts.

[0024] In another exemplary embodiment of this application, in step 100 above, a processing device can control a structured light sensor (or depth camera) to acquire 3D point cloud data of an industrial product through an image acquisition interface, while simultaneously controlling an RGB camera to acquire 2D RGB images of the same industrial product. Spatial alignment between the 3D point cloud data and the 2D RGB image is achieved through camera intrinsic parameter calibration, ensuring that the same physical location corresponds consistently in both types of data.

[0025] Furthermore, to provide standardized input for the 3D point cloud feature extraction model (hereinafter referred to as the PFE module) and the image feature extraction model (hereinafter referred to as the IFE module), avoid noise and scale inconsistencies affecting feature learning accuracy, and ensure the stability of subsequent processing, the preprocessing method in step 100 above can be to perform denoising and coordinate normalization on the 3D point cloud data, and to perform size unification and grayscale normalization on the 2D RGB image. Based on this, the implementation process of step 100 above can include: Step 100-1, 3D Point Cloud Denoising: Statistical filtering algorithms are used to denoise the 3D point cloud data to obtain the denoised point cloud coordinates. For example: calculate the average distance of each point's K nearest neighbors (e.g., K=20), remove outliers whose distance mean exceeds twice the standard deviation, and eliminate false points caused by environmental noise (e.g., dust, light reflection).

[0026] Step 100-2, 3D coordinate normalization: Map the denoised point cloud coordinates to the [-1,1] interval to obtain preprocessed 3D point cloud data. This process can eliminate coordinate scale interference caused by differences in the size of different industrial products.

[0027] Step 100-3, 2D Image Size Unification: Use bilinear interpolation to unify the pixel values ​​of the 2D RGB image. For example, use bilinear interpolation to transform the 2D RGB image to 224×224 pixels. This step ensures consistent resolution of the input images, preventing resolution differences from affecting the feature extraction scale.

[0028] Step 100-4, 2D grayscale normalization: Normalize the unified pixel values ​​to the [0,1] range to obtain 2D RGB image preprocessing data. For example, normalize the pixel values ​​(0-255) of the image after size unification to the [0,1] range to reduce the interference of pixel value magnitude on model training.

[0029] In another exemplary embodiment of this application, in order to fuse local subtle defect features with global structural features, and to balance small-size defect detection with overall structural anomaly identification, accurate 3D geometric information is provided for multimodal fusion. In this embodiment, in step 102 above, 3D point cloud preprocessing data can be processed by a PFE module to obtain layered 3D geometric features. The PFE module (such as...) Figure 2 As shown, using stacked Point Transformer V3 (PTv3) as the backbone, spatial filling curves are employed to serialize the disordered point cloud. Enhanced Conditional Position Encoding (xCPE) is combined to enhance spatial semantics. A two-stage strategy of in-patch feature extraction and inter-patch global pooling is used to capture local-global geometric features. This extraction process is as follows: Figure 2 As shown. Based on this, the implementation process of step 102 provided above in this application may include: Step 102-1: Using a specific curve (such as a Z-order space-filling curve or a Hilbert curve), the 3D point cloud preprocessing data is divided according to the grid size. (like Discretize the data into a grid index to obtain discrete coordinates. The discrete coordinates are represented as follows: .

[0030] In the formula, For discrete coordinates, ( , , ) represents the point cloud coordinates of the 3D point cloud preprocessing data.

[0031] Step 102-2, Point Cloud Serialization: Convert discrete coordinates into binary code (e.g., ... Bitwise operations are used for interleaving. , , The binary code is used to generate an ordered serialized code. This ordered serialized code is then sorted in ascending order to obtain an ordered point cloud sequence. The ordered point cloud sequence is represented as follows: , These are discrete coordinates converted to binary code.

[0032] Converting disordered point clouds into ordered sequences can solve the problem of low computational efficiency caused by disorder in traditional point cloud processing. At the same time, by using the local preservation property of the space-filling curve, it can ensure that geometrically adjacent points remain adjacent in the sequence, which is convenient for subsequent local feature learning.

[0033] Step 102-3, Enhanced Location Encoding: Divide the ordered point cloud sequence into... For each non-overlapping patch (e.g., each patch contains 32 points), xCPE is applied, and positional encoding is calculated through sparse convolutional layers with skip connections. The formula for calculating the positional encoding is: .

[0034] In the formula, For position encoding, For learnable sparse convolution weights, For layer normalization (to avoid gradient vanishing). It is a sparse convolutional layer.

[0035] Based on this step, the traditional relative position encoding can be replaced. The computation is reduced by sparse convolution, while the spatial semantic information of the point cloud is preserved, the discriminative power of geometric features is enhanced, and it is easier to distinguish subtle geometric defects (such as the tiny indentation of a pin).

[0036] Step 102-4: Generate a query (Q), key (K), and value (V) matrix based on each location code. Wherein, , , and All of these are learnable weights.

[0037] Step 102-5, Patch Attention Calculation: Based on the query, key, and value matrix, multi-head attention is applied to determine local associations within the patch, resulting in: .

[0038] In the formula, For local association, The dimension of the key matrix, This is the output weight. , and The first i The query, key, and value matrix corresponding to each location code. For the softmax function, For the concat function, This is the transpose of the matrix.

[0039] By modeling the local geometric relationships between points within a patch, local features of subtle defects (such as surface cracks) can be captured, avoiding the loss of local information.

[0040] Step 102-6, Hierarchical Feature Aggregation: A U-shaped encoder-decoder structure (block depth [2,2,6,2], number of channels [64,128,256,512]) is adopted. Max pooling is performed on the local associations within each patch to obtain the pooling result. The encoder reduces the feature map size and increases the semantic depth through downsampling, while the decoder restores the feature resolution through upsampling.

[0041] In-patch max pooling: for each local association Perform max pooling to preserve the most discriminative defect features within the patch (such as feature values ​​of raised areas).

[0042] Global average pooling between patches: Calculate the average value of the pooling results of all patches, and aggregate global structural information (such as the overall shape of the product) to obtain the pooling result.

[0043] Step 102-7: Determine the average value of the pooling results for all patches and aggregate the global structural information to obtain 3D geometric features. Use these 3D geometric features as 3D point cloud features. The 3D geometric features (i.e., 3D point cloud features) are represented as follows: : .

[0044] In the formula, This is for max pooling.

[0045] In another exemplary embodiment of this application, overfitting can be avoided by using ridge regression, and defect-related features can be enhanced by adaptive weighting to provide accurate 2D visual information for multimodal fusion. In this embodiment, the implementation of step 103 above can be described as follows: 2D RGB image preprocessing data is processed by the IFE module to obtain semantically enhanced 2D visual features. The IFE module can be built based on a width learning system, introducing an information entropy-driven adaptive weighting mechanism to dynamically adjust the feature channel weights to enhance discriminative regions and suppress redundant backgrounds. Based on this, as... Figure 3 As shown, the IFE module includes a feature layer, an enhancement layer, and an output layer.

[0046] In practical applications, the implementation process of step 103 can be achieved by the device's processor calling the IFE module, which, based on the width learning system, processes the 2D RGB image preprocessing data according to steps 103-1 to 103-9 to obtain 2D image features.

[0047] Step 103-1: Flatten the 2D RGB image preprocessing data into a matrix. , For the sample size, This represents the total number of pixels.

[0048] Step 103-2, Basic Feature Node Generation: Based on a matrix, basic feature nodes are generated through linear transformation and ReLU activation. The group of basic feature nodes includes: , For the first i One basic feature node, , For feature dimension, Randomly initialized in the range [-0.1, 0.1]. This is the ReLU activation function.

[0049] Step 103-3: Concatenate all basic feature nodes to obtain the basic matrix, which is: , .

[0050] Through steps 103-1 to 103-3 above, the basic visual patterns of 2D images (such as edges, textures, and color gradients) can be captured, providing a foundation for subsequent feature enhancement.

[0051] Step 103-4, Enhanced Node Generation: Perform a non-linear mapping (such as sigmoid activation) on the fundamental matrix to generate... Group enhancement nodes include: , For the first j One enhanced node, and Randomly initialized in the range [-0.01, 0.01]. It is the sigmoid activation function.

[0052] Step 103-5: Concatenate all enhancement nodes to obtain the enhancement matrix, which is: , .

[0053] Steps 103-4 and 103-5 can expand feature diversity to cover visual patterns of complex surface defects (such as color anomalies of surface contamination and irregular patterns of texture damage).

[0054] Step 103-6, Entropy-driven adaptive weighting: Determine the feature residuals based on the fundamental matrix and the enhancement matrix, with: , For characteristic residuals, For feature concatenation matrix, As initial random weights, These are reference features for normal samples.

[0055] Step 103-7: Determine the information entropy of the feature residuals through kernel density estimation, and generate the sample weight vector. , .in: .

[0056] .

[0057] In the formula, The standard deviation of the residuals. This refers to the KDE bandwidth. Regions with higher information entropy (such as defective regions) are considered higher. The larger.

[0058] Through steps 103-6 and 103-7 above, we can dynamically focus on regions with high semantic value (such as surface contamination and texture anomalies), suppress redundant information in uniform backgrounds (such as smooth product surfaces), and improve feature discriminability.

[0059] Step 103-8, Weighted Feature Output: Through ridge regression with weight penalty, the output weights are obtained based on the sample weight vector, as follows: , To output weights, The regularization coefficient is . It is an identity matrix.

[0060] Step 103-9: Based on the output weights, combine the fundamental matrix and the enhancement matrix to obtain semantically enhanced 2D visual features, and use these semantically enhanced 2D visual features as 2D image features. The semantically enhanced 2D visual features are represented as follows: .

[0061] In the formula, These are semantically enhanced 2D visual features, i.e., 2D image features.

[0062] In another exemplary embodiment of this application, in order to sequentially achieve low-dimensional spatial alignment, mid-dimensional semantic filtering, and high-dimensional feature aggregation, and to force intermodal interactions to only exchange key complementary information, this application can set a hierarchical attention bottleneck in the Multimodal Bottleneck Transformer (MBT) module to perform bidirectional collaborative fusion of hierarchical 3D geometric features (i.e., 3D point cloud features) and semantically enhanced 2D visual features (i.e., 2D image features) to obtain cross-modal fused features. Based on this, the implementation process of step 104 provided above in this application can be achieved by the processor of the processing device calling the MBT module and implementing the 3D geometric features according to steps 104-1 to 104-4. With 2D visual features The two-way synergistic integration yields fusion characteristics.

[0063] Step 104-1, Feature Projection Alignment: Employing a Modality-Specific Projection Function 3D point cloud features (dimensions) ) and 2D image features (dimensions) Mapped to the same dimension The formula is: .

[0064] In the formula, All are learnable embedding matrices, projection functions For linear transformation ( ). , These are the mapped 3D point cloud features and 2D image features, respectively.

[0065] This step can eliminate the dimensional differences between 3D and 2D modalities, provide a unified feature space for cross-modal interaction, and avoid fusion interference caused by dimensional mismatch.

[0066] Step 104-2, Single-modal feature refinement: Apply the Transformer model (containing 2 layers) to obtain single-modal 3D point cloud features and single-modal 2D image features based on the mapped 3D point cloud features and the mapped 2D image features, respectively. The formula is as follows: .

[0067] .

[0068] In the formula, , The parameters (including attention weights and feedforward network weights) of the modality-specific Transformer model are retained by the residual connections, which preserve the original feature information. , These are single-modal 3D point cloud features and single-modal 2D image features, respectively.

[0069] This step can enhance discriminative features within a single modality (such as 3D geometric convexities and 2D texture anomalies), reduce intramodal redundancy, and lay a high-quality feature foundation for cross-modal fusion.

[0070] Step 104-3, Layered Bottleneck Interaction: Set multiple (e.g.) Bottleneck Token (dimension) As a cross-modal interaction hub, the feature sequence structure is as follows: Multi-level bottleneck interactions were performed with single-modal 3D point cloud features and single-modal 2D image features respectively to obtain 3D features and 2D features.

[0071] In practical applications, multi-layered bottleneck interactions are achieved through bidirectional attention interactions, where 3D features interact with bottleneck tokens, and 2D features interact with bottleneck tokens, using the following formula: .

[0072] .

[0073] In the formula, , For attention parameters, This is for average pooling. Indicates the first Layer for the first Bottleneck token features for each modality.

[0074] Low-level bottlenecks focus on spatial alignment (matching 3D points with 2D pixel positions), mid-level focus on semantic filtering (preserving complementary information), and high-level focus on feature association (associating geometric and visual defects).

[0075] Based on the above description, bottleneck tokens can be used to limit the scope of cross-modal interactions, avoid computational explosion and interference caused by direct interaction of high-dimensional features, and only transmit key complementary information (such as 3D geometry + 2D texture).

[0076] Step 104-4, Cross-modal fusion output: Concatenate 3D features, 2D features, and bottleneck tokens to obtain fused features. For example, iterate... After layer bottleneck interactions, the final 3D features are stitched together. 2D features With bottleneck tokens , obtain fusion features .

[0077] By aggregating complementary information from 3D geometry and 2D vision, a fusion feature with both spatial integrity and semantic discriminativeness can be generated, providing a comprehensive basis for anomaly detection.

[0078] In another exemplary embodiment of this application, in order to achieve the dual functions of anomaly detection and area localization, and to provide production line operators with intuitive defect location information for subsequent rework or rejection, the anomaly detection and output method of this application may be as follows: input the fused features into the anomaly scoring head, calculate the deviation between the fused features and the distribution of normal sample features (i.e., preset fused features) to obtain an anomaly score, determine whether the product is abnormal based on a preset threshold, and output the anomaly area localization result. Based on this, the implementation process of step 105 above in this application can be performed by calling the anomaly scoring head through the processor of the processing device, according to steps 105-1 to 105-6.

[0079] Step 105-1, Anomaly Score Calculation: The deviation between the fused features and the preset fused feature mean distribution is measured using Mahalanobis distance and used as the anomaly score. The anomaly score is expressed as... : .

[0080] In the formula, The mean of the normal sample fusion features learned during the training phase (i.e., the preset mean of the fusion features). This is the covariance matrix (calculated from normal samples).

[0081] This step quantifies the difference between the current sample and the normal pattern; the greater the difference, the better. The larger the value, the more likely it is to be an anomalous sample.

[0082] Step 105-2, Anomaly Determination and Location: Determine if the anomaly score is greater than a set threshold. If the anomaly score is greater than the set threshold, the anomaly determination result is determined to be an industrial product anomaly. An anomaly region mask is generated based on the anomaly response from the fused features through feature back projection. The anomaly region location result is obtained based on the anomaly region mask. If the anomaly score is not greater than the set threshold, the anomaly determination result is determined to be that the industrial product is normal, and no action is taken.

[0083] In practical applications, the optimal threshold can be determined by the ROC curve of the validation set. The comparison results of ROC curves for each category are as follows: Figure 5 As shown. If If the condition is met, the industrial product is deemed abnormal; otherwise, it is deemed normal. Figure 5 In the dataset, the horizontal and vertical axes of parts a through j are exactly the same. Part a represents the ROC curve of the cable connector on the MVTec3D-AD dataset, part b represents the ROC curve of the pin on the MVTec3D-AD dataset, part c represents the ROC curve of the foam on the MVTec3D-AD dataset, part d represents the ROC curve of the rope on the MVTec3D-AD dataset, part e represents the ROC curve of the tire on the MVTec3D-AD dataset, part f represents the ROC curve of the bagel on the MVTec3D-AD dataset, part g represents the ROC curve of the carrot on the MVTec3D-AD dataset, part h represents the ROC curve of the cookie on the MVTec3D-AD dataset, part i represents the ROC curve of the peach on the MVTec3D-AD dataset, and part j represents the ROC curve of the potato on the MVTec3D-AD dataset.

[0084] Anomaly region localization: Anomaly region masks are generated by feature backprojection (mapping the anomaly response of the fused features back to the original 3D point cloud coordinates and 2D image pixels) (the 3D point cloud mask marks the anomaly points, and the 2D image mask marks the anomaly pixels).

[0085] Output results: The processing device displays the normal / abnormal judgment result and the abnormal area mask on the display through the output module. If it is judged to be abnormal, the alarm device (such as audible and visual alarm) is triggered.

[0086] In another exemplary embodiment of this application, the advantages of this application over the prior art are illustrated by means of simulation comparison.

[0087] 1. A breakthrough in both 3D point cloud processing efficiency and geometric feature extraction accuracy.

[0088] The PFE module in this application completely eliminates the high computational overhead of KNN neighborhood search in traditional 3D point cloud processing through Z-order space-filling curve serialization and enhanced conditional location coding (xCPE), achieving simultaneous improvement in efficiency and accuracy. In the MVTec3D-AD dataset test, the 3D modal inference speed adopted in this application reaches 70.12 FPS, which is 67.2% higher than the existing mainstream method AST (41.94 FPS). At the same time, the 3D modal image-level AUROC (detection accuracy index) reaches 0.843, ranking first among all compared methods, significantly outperforming methods such as PatchCore+FPFH (0.782) and VoxelVM (0.699), as shown in Table 1. For 3D geometric defect detection of rigid industrial parts, the PFE module of this application retains key discriminative features through a patch-based max pooling + inter-patch global pooling strategy. The 3D modal AUROC for pins reaches 0.863, a 27% improvement over AST (0.679), and far exceeds the performance of VoxelVM (0.823). This allows for accurate capture of subtle structural anomalies such as dents and deformations that are easily missed by traditional methods (see attached diagram). Figure 4 (Results of pin defect inspection).

[0089] Table 1. Comparison of the I-AUROC scores of the method provided in this application with other existing methods on the MVTec3D-AD dataset.

[0090] 2. The discriminative power of 2D visual features and the accuracy of defect localization are significantly improved.

[0091] The information entropy-driven adaptive weighting mechanism of the IFE module in this application can dynamically focus on high semantic value regions such as surface contamination and texture anomalies, while effectively suppressing redundant background interference. In RGB modal testing, the image-level AUROC of this application reaches 0.858, which is better than classic 2D anomaly detection methods such as PADiM (0.746) and PatchCore (0.770), as shown in Table 1 above. The pixel-level localization index AUPRO reaches 0.943, which is 8%-10% higher than CS-Flow (0.871) and PatchCore (0.876). For surface scratch defects of cable joints, the RGB modal detection AUROC of this method reaches 0.927, second only to CS-Flow (0.930) and AST (0.928), and the IoU (Intersection over Union) between the localization mask and the real defect area exceeds 0.85, which can clearly distinguish between effective defects and background noise. The detection performance is outstanding in 2D defect detection of industrial parts (corresponding to Figure 4 (Comparison of RGB modal prediction results and true values ​​for cable connectors). The comparison results of the relationship curve between precision and recall (i.e., the PR curve) are as follows: Figure 6 As shown. Figure 6 In the dataset, the horizontal and vertical axis scales of parts a-j are exactly the same. Part a represents the PR curve of the cable connector on the MVTec3D-AD dataset, part b represents the PR curve of the pin on the MVTec3D-AD dataset, part c represents the PR curve of the foam on the MVTec3D-AD dataset, part d represents the PR curve of the rope on the MVTec3D-AD dataset, part e represents the PR curve of the tire on the MVTec3D-AD dataset, part f represents the PR curve of the bagel on the MVTec3D-AD dataset, part g represents the PR curve of the carrot on the MVTec3D-AD dataset, part h represents the PR curve of the cookie on the MVTec3D-AD dataset, part i represents the PR curve of the peach on the MVTec3D-AD dataset, and part j represents the PR curve of the potato on the MVTec3D-AD dataset.

[0092] 3. Industry-leading multimodal fusion efficiency and cross-modal collaboration.

[0093] The hierarchical bottleneck design of the MBT module in this application reduces the computational complexity of cross-modal fusion from that of traditional Transformers. ( For 3D feature numbers, (for 2D feature numbers) reduced to ( The bottleneck token count significantly reduces computational costs while maintaining fusion accuracy. In multimodal (RGB-3D) testing, the image-level AUROC of this application reaches 0.934, ranking first among all compared methods, approaching the accuracy level of AST (0.937), but the inference speed is 1.67 times that of AST, as shown in Table 1 above. The pixel-level AUPRO reaches 0.962, which is better than PatchCore+FPFH (0.959), achieving the optimal balance between high accuracy and high real-time performance. This fusion advantage is particularly significant in the detection of flexible materials and food products. Among them, the AUROC of foam defects reaches 0.914, rope reaches 0.977, bagel reaches 0.988, and carrot reaches 0.977, all far exceeding existing similar methods. It can simultaneously utilize 3D geometric information (such as foam depressions) and 2D texture information (such as mold on the surface of carrots), completely solving the blind zone problem of single-modal detection (corresponding to Figure 4 (Multimodal detection results of Chinese bagel and carrot).

[0094] 4. Excellent adaptability and generalization ability to industrial scenarios in unsupervised learning This application requires only normal samples (without anomaly annotations) for training, perfectly addressing the core pain points in industrial scenarios where anomaly samples are scarce and manual annotation costs are high (3D point cloud annotation efficiency is only 1 / 5 of 2D image annotation efficiency). In the 10 product categories covered by MVTec3D-AD (5 industrial parts + 5 food products), the multimodal AUROC of all categories exceeds 0.85, as shown in Table 1 above. The industrial parts group has an average AUROC of 0.921, with stable detection performance for key categories such as cable connectors (0.879), pins (0.974), and tires (0.859). The food group has an average AUROC of 0.953, with detection accuracy for categories such as bagels (0.988), peaches (0.918), and potatoes (0.917) at an industry-leading level. The method provided in this application can be directly deployed on agile manufacturing production lines such as automotive parts assembly lines and food sorting lines, without requiring re-annotation of samples for product iterations, reducing annotation costs by more than 90%.

[0095] In another exemplary embodiment of this application, the space-filling curve serialization processing method in the PFE module described above can be replaced by a voxelization + sorting method to achieve point cloud ordering. That is, the point cloud is voxelized into a 3D mesh, and the points within each voxel are sorted according to their x / y / z coordinates to generate an ordered sequence. Alternatively, a Morton curve can be used instead of a Z-order curve. The binary interleaving rules of the Morton curve are similar to those of the Z-order curve, with only slight differences in local preservation in high-dimensional space.

[0096] In another exemplary embodiment of this application, the BLS architecture in the above-mentioned IFE module can be replaced by a lightweight CNN (such as MobileNetV2, SqueezeNet), which reduces the amount of computation through depthwise separable convolutions, and adds an SE attention module (Squeeze-and-Excitation) after the convolutional layer of the CNN to achieve adaptive adjustment of channel weights (similar to the entropy-driven weighting of IFE).

[0097] In another exemplary embodiment of this application, the hierarchical bottleneck fusion in the MBT module of this application can be replaced by a method of multimodal fusion using feature dimensionality reduction + cross attention. That is, the 3D / 2D features are first reduced to a low dimension (e.g., d=128) through PCA (principal component analysis), and then traditional cross attention is applied to perform modal interaction, thereby reducing the computational load of high-dimensional features.

[0098] Based on the same inventive concept, this application also provides an industrial product three-dimensional anomaly detection system for implementing the above-mentioned industrial product three-dimensional anomaly detection method. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the industrial product three-dimensional anomaly detection system provided below can be found in the limitations of the industrial product three-dimensional anomaly detection method described above, and will not be repeated here.

[0099] In one exemplary embodiment, such as Figure 7 As shown, a three-dimensional anomaly detection system for industrial products is provided, comprising: a data acquisition module 1, a data processing module 2, a PFE feature extraction module 3, an IFE feature extraction module 4, a multimodal fusion module 5, an anomaly detection module 6, and a result output module 7. The data processing module 2 is connected to the data acquisition module 1. The PFE feature extraction module 3 is connected to the data processing module 2. The IFE feature extraction module 4 is connected to the data processing module 2. The multimodal fusion module 5 is connected to both the PFE feature extraction module 3 and the IFE feature extraction module 4. The anomaly detection module 6 is connected to the multimodal fusion module 5. The result output module 7 is connected to the anomaly detection module 6.

[0100] The data acquisition module 1 acquires 3D point cloud data and 2D RGB images of industrial products. The data processing module 2 preprocesses the 3D point cloud data and 2D RGB images of industrial products to obtain preprocessed 3D point cloud data and preprocessed 2D RGB image data. The PFE feature extraction module 3 constructs a 3D point cloud feature extraction model based on a stacked Point Transformer V3 framework and extracts 3D point cloud features from the preprocessed data using this model. The IFE feature extraction module 4 constructs an image feature extraction model and extracts 2D image features from the preprocessed 2D RGB image data using this model. The multimodal fusion module 5 fuses the 3D point cloud features and 2D image features to obtain fused features. The anomaly detection module 6 determines anomaly scores based on the fused features and a preset mean value of the fused features, and determines the anomaly detection result based on the relationship between the anomaly score and a set threshold. The result output module 7 visualizes the anomaly detection results. The anomaly detection results include anomaly determination results and anomaly region location results.

[0101] As an optional implementation, the hardware of data acquisition module 1 includes: a structured light sensor or LiDAR, an industrial RGB camera, and a calibration board. The sensor and the industrial RGB camera are connected to the network port of an industrial computer via a GigE Ethernet interface, and the calibration board is used for camera intrinsic parameter calibration. With this hardware setup, 3D point cloud data and 2D RGB images of industrial products can be acquired simultaneously, and spatial alignment of the two types of data can be achieved using OpenCV calibration tools, with an alignment error of less than 1 pixel.

[0102] As an optional implementation, the hardware configuration of the data preprocessing module 2 includes: an industrial computer (CPU: Intel Core i9-13900K, 64GB DDR5 memory, 2TB SSD storage). It communicates with the data acquisition module 1 via a PCIe 4.0 interface to receive raw 3D / 2D data (i.e., 3D point cloud data and 2D RGB images). This hardware configuration allows for the execution of preprocessing algorithms such as statistical filtering and bilinear interpolation to denoise and normalize the 3D point cloud, and to unify the size and normalize the grayscale of the 2D image, with a preprocessing latency of less than 10ms / frame.

[0103] As an optional implementation, the hardware configuration of the PFE feature extraction module 3 includes: an industrial computer CPU shared with the data preprocessing module 2, equipped with the PTv3 algorithm acceleration library. Preprocessed 3D point cloud data is acquired through memory sharing (DDR5 high-speed memory). This hardware structure enables point cloud serialization, xCPE position encoding, patch attention calculation, and hierarchical feature aggregation, outputting 512-dimensional hierarchical 3D geometric features at a feature extraction speed of 60 frames per second.

[0104] As an optional implementation, the hardware configuration of the IFE feature extraction module 4 includes: sharing an industrial computer CPU with the data preprocessing module 2, and equipped with a dedicated acceleration engine for the BLS algorithm. Preprocessed 2D RGB image data is acquired through memory sharing. This hardware structure allows for the generation of BLS feature nodes and enhancement nodes, and the output of 1024-dimensional semantically enhanced 2D visual features through entropy-driven adaptive weighting, achieving a feature extraction speed of 80 frames per second.

[0105] As an optional implementation, the hardware configuration of the multimodal fusion module 5 includes: an FPGA chip, which embeds the MBT module inference logic. It receives 3D features from the PFE feature extraction module 3 and 2D features from the IFE feature extraction module 4 via a PCIe 4.0 interface, achieving a data transmission rate of 32Gbps. This hardware structure enables feature projection alignment, single-modal refinement, hierarchical bottleneck interaction, and cross-modal fusion, outputting fused features with a fusion latency of less than 5ms / frame, supporting real-time processing at 100 frames / second.

[0106] As an optional implementation, the anomaly detection module 6 comprises a GPU (NVIDIA RTX 4090) equipped with an anomaly scoring head and a feature back-projection algorithm. It receives fused features output from the FPGA chip via an NVLink interface. This hardware configuration enables the calculation of Mahalanobis distance anomaly scores, determination of anomaly states based on optimal thresholds, generation of anomaly region masks through feature back-projection, and a detection latency of less than 8ms / frame.

[0107] As an optional implementation, the hardware components of the result output module 7 include: an industrial touchscreen, an audible and visual alarm (model: LTE-1101J), and a printer. The industrial touchscreen communicates with the GPU via an RS485 interface, the audible and visual alarm is connected to the industrial computer via a GPIO interface, and the printer is connected to the touchscreen via a USB 3.0 interface. With this hardware configuration, the touchscreen displays the detection results, abnormal area masks, and confidence levels in real time. An audible and visual alarm is triggered when an abnormality occurs. It supports the generation and printing of PDF format detection reports, which include key information such as product information, detection time, abnormality type, and defect location.

[0108] Based on the system provided in this application, the overall data flow is as follows: data acquisition module → data preprocessing module → PFE / IFE feature extraction module (parallel) → multimodal fusion module → anomaly detection module → result output module. Each module is connected through standardized interfaces, supporting modular replacement and expansion to adapt to the detection needs of different industrial scenarios.

[0109] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores 3D anomaly detection data for industrial products. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D anomaly detection method for industrial products.

[0110] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0111] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0112] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0113] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0115] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (RRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0116] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0117] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting three-dimensional anomalies in industrial products, characterized in that, include: Acquire 3D point cloud data and 2D RGB images of industrial products, and preprocess the 3D point cloud data and 2D RGB images of industrial products to obtain 3D point cloud preprocessed data and 2D RGB image preprocessed data. A 3D point cloud feature extraction model is constructed using stacked Point Transformer V3 as the backbone; The 3D point cloud feature extraction model is used to extract 3D point cloud features based on the preprocessed 3D point cloud data. An image feature extraction model is constructed, and the image feature extraction model is used to extract 2D image features based on the 2D RGB image preprocessing data; Multimodal feature fusion is performed on the 3D point cloud features and the 2D image features to obtain fused features; An anomaly score is determined based on the fusion features and the mean of the preset fusion features, and an anomaly detection result is determined based on the relationship between the anomaly score and a set threshold; the anomaly detection result includes anomaly determination result and anomaly region location result.

2. The method for three-dimensional anomaly detection of industrial products according to claim 1, characterized in that, Preprocessing is performed on the 3D point cloud data and 2D RGB images of industrial products to obtain preprocessed 3D point cloud data and preprocessed 2D RGB image data, including: The 3D point cloud data is denoised using a statistical filtering algorithm to obtain the denoised point cloud coordinates; The denoised point cloud coordinates are mapped to the [-1,1] interval to obtain the 3D point cloud preprocessed data; The pixel values ​​of the 2D RGB image are unified using bilinear interpolation. The unified pixel values ​​are normalized to the [0,1] range to obtain the 2D RGB image preprocessing data.

3. The method for three-dimensional anomaly detection of industrial products according to claim 1, characterized in that, Using the aforementioned 3D point cloud feature extraction model, 3D point cloud features are extracted based on the preprocessed 3D point cloud data, including: Using a specific curve, the 3D point cloud preprocessed data is discretized into grid indices according to the grid size to obtain discrete coordinates; The discrete coordinates are converted into binary code, and the binary code is interleaved through bit operations to generate an ordered serialized code. The ordered serialized code is then arranged in ascending order to obtain an ordered point cloud sequence. The ordered point cloud sequence is divided into Each non-overlapping patch is applied with xCPE and position encoding is computed through a sparse convolutional layer with skip connections. Generate a query, key, and value matrix based on each of the aforementioned location codes; Based on the query, key, and value matrix, multi-head attention is applied to determine local associations within the patch. A U-shaped encoder-decoder structure is used to perform max pooling on the local correlations within each patch to obtain the pooling result; The average value of the pooling results of all patches is determined, and the global structural information is aggregated to obtain 3D geometric features; the 3D geometric features are used as the 3D point cloud features.

4. The method for three-dimensional anomaly detection of industrial products according to claim 1, characterized in that, The image feature extraction model includes: a feature layer, an enhancement layer, and an output layer; Using the image feature extraction model, 2D image features are extracted based on the 2D RGB image preprocessing data, including: Flatten the 2D RGB image preprocessing data into a matrix; Based on the matrix, it is generated through linear transformation and ReLU activation. Group basic feature nodes; Concatenate all the basic feature nodes to obtain the basic matrix; Perform a nonlinear mapping on the fundamental matrix to generate Group enhancement nodes; Concatenate all the enhancement nodes to obtain the enhancement matrix; The feature residuals are determined based on the base matrix and the enhancement matrix; The information entropy of the feature residuals is determined by kernel density estimation, and a sample weight vector is generated. The output weights are obtained based on the sample weight vector using ridge regression with weight penalty. Based on the output weights, semantically enhanced 2D visual features are obtained by combining the base matrix and the enhancement matrix, and these semantically enhanced 2D visual features are used as the 2D image features.

5. The method for three-dimensional anomaly detection of industrial products according to claim 1, characterized in that, Multimodal feature fusion is performed on the 3D point cloud features and the 2D image features to obtain fused features, including: A modality-specific projection function is used to map the 3D point cloud features and the 2D image features to the same dimension; By applying the Transformer model, single-modal 3D point cloud features and single-modal 2D image features are obtained based on the mapped 3D point cloud features and the mapped 2D image features, respectively. Multiple bottleneck tokens are set up as cross-modal interaction hubs, and multi-layer bottleneck interactions are performed with the single-modal 3D point cloud features and the single-modal 2D image features respectively to obtain 3D features and 2D features; The fused feature is obtained by concatenating the 3D feature, the 2D feature, and the bottleneck token.

6. The method for three-dimensional anomaly detection of industrial products according to claim 1, characterized in that, An anomaly score is determined based on the fusion features and the preset mean of the fusion features, and an anomaly detection result is determined based on the relationship between the anomaly score and a set threshold, including: The deviation between the fused feature and the preset fused feature mean distribution is measured using Mahalanobis distance, and this deviation is used as the anomaly score. Determine whether the abnormal score is greater than the set threshold; When the anomaly score is greater than the set threshold, the anomaly determination result is determined to be an industrial product anomaly. An anomaly region mask is generated based on the anomaly response of the fused features through feature back projection. The abnormal region location result is obtained based on the abnormal region mask; When the abnormal score is not greater than the set threshold, the abnormal judgment result is determined to be that the industrial product is normal and no action is taken.

7. A three-dimensional anomaly detection system for industrial products, characterized in that, include: The data acquisition module is used to acquire 3D point cloud data and 2D RGB images of industrial products; The data processing module, connected to the data acquisition module, is used to preprocess the 3D point cloud data and 2D RGB image of the industrial product to obtain 3D point cloud preprocessed data and 2D RGB image preprocessed data. The PFE feature extraction module is connected to the data processing module. It is used to construct a 3D point cloud feature extraction model with stacked Point Transformer V3 as the backbone, and to extract 3D point cloud features based on the 3D point cloud preprocessed data using the 3D point cloud feature extraction model. The IFE feature extraction module is connected to the data processing module and is used to construct an image feature extraction model and extract 2D image features based on the 2D RGB image preprocessing data using the image feature extraction model. A multimodal fusion module is connected to the PFE feature extraction module and the IFE feature extraction module respectively, and is used to perform multimodal feature fusion on the 3D point cloud features and the 2D image features to obtain fused features; An anomaly detection module, connected to the multimodal fusion module, is used to determine an anomaly score based on the fusion features and the mean of preset fusion features, and to determine the anomaly detection result based on the relationship between the anomaly score and a set threshold. The result output module is connected to the anomaly detection module and is used to visualize the anomaly detection results; the anomaly detection results include anomaly determination results and anomaly area location results.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the three-dimensional anomaly detection method for industrial products according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the three-dimensional anomaly detection method for industrial products as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the three-dimensional anomaly detection method for industrial products as described in any one of claims 1-6.

Images