Multi-modal industrial anomaly detection method and system based on mask reconstruction mechanism

Abstract

The invention discloses a multi-modal industrial anomaly detection method and system based on a mask reconstruction mechanism, and the method abandons a pre-training model and memory bank method for solving the problems that in industrial production, a high-quality annotation data set is difficult to obtain, and 3D multi-modal data is difficult to fuse and utilize, and employs a reconstruction-based mode to process point cloud and RGB images. Therefore, computing resource consumption is reduced, and real-time performance is enhanced. The method comprises the following steps: respectively applying a mask reconstruction mechanism in an RGB image and 3D point cloud data, accurately extracting 2D and 3D data features, and assisting in improving anomaly detection performance; gPS geometric perception sampling is introduced in 3D point cloud information extraction, a graph attention mechanism is added in a joint encoder, and 2D-3D feature interaction and cross-modal understanding ability is enhanced. According to the method, a large number of experiments are carried out on MVTec3D-AD and Eyescandia data sets, the effectiveness of the method is fully verified, powerful support is provided for industrial anomaly detection to reach a brand new performance height, and a new thought is provided for guaranteeing the quality of industrial products.

Description

Technical Field

[0001] The present invention belongs to the intersection of industrial manufacturing and computer vision, and specifically relates to a multimodal industrial anomaly detection method based on a mask reconstruction mechanism. It can be used to perform high-precision anomaly detection on various industrial products in industrial production scenarios to ensure product quality. Background Art

[0002] In today's rapidly developing industrial manufacturing sector, with the continuous advancement of automation and intelligent technologies, product quality control has become a key factor in maintaining competitiveness. Industrial anomaly detection, as a crucial component of quality control, aims to locate abnormal features in products during or after production to minimize potential defects. With the rapid development of deep learning technology, image anomaly detection has made significant progress, significantly improving its accuracy and robustness. It has gradually broken away from the limitations of manual feature engineering and instead learned more complex patterns to address diverse detection problems.

[0003] In real-world applications, high-quality annotated datasets are difficult to obtain due to the low proportion of abnormal regions, unknown anomaly patterns, and high annotation costs. Traditionally, anomaly detection relies on human inspectors or image-based methods. Furthermore, research and application in industrial anomaly detection are gradually shifting from traditional 2D images to richer 3D data. The recent introduction of 3D anomaly detection benchmark datasets, such as MVTec3D-AD, Eyecandies, and Real3D-AD, has spurred research on multimodal industrial anomaly detection, particularly by combining point clouds and RGB images, which provide more comprehensive product surface information, crucial for detecting defects. However, despite the significant potential of 3D data for anomaly detection, effectively fusing and utilizing this multimodal data remains a challenge. For example, while point clouds provide unique spatial information, they are less sensitive to color perception than RGB images. Therefore, leveraging the redundancy and complementarity between modalities is crucial for improving the performance of industrial anomaly detection (IAD).

[0004] Among existing deep learning-based detection methods, unsupervised methods are more widespread than supervised methods. Current industrial anomaly detection primarily includes feature embedding-based methods and reconstruction-based methods. Feature embedding-based methods typically use pretrained feature extractors and memory banks, which require more computational resources. Asymmetric teacher-student networks for industrial anomaly detection use pretrained feature extractors to learn deep feature representations of the data. These feature representations are used to estimate the normal distribution of the data. During the test phase, the deviation of the feature representation of new samples from the normal distribution is used as an anomaly score. A typical example of this approach is PatchCore, which stores the features of normal samples in a memory bank and compares the differences between the test sample and the memory bank at test time to locate anomalous regions. Furthermore, methods based on normalization flows, such as CFLOW-AD, learn the feature distribution of normal samples and map the features of test samples to a Gaussian distribution. Anomalous samples are detected due to their large deviation from the Gaussian distribution.

[0005] The core idea of reconstruction-based methods is to use a model to learn a representation of normal data and then detect anomalies by comparing the differences between test samples and samples reconstructed by the model. These methods typically rely on generative models such as autoencoders, generative adversarial networks, and variational autoencoders. Autoencoders learn effective representations of data by minimizing the difference between input and reconstructed output. However, anomalous samples are often not accurately reconstructed by the model, resulting in large reconstruction errors. Generative adversarial networks, on the other hand, introduce a discriminator to distinguish between model-generated samples and real samples. Anomalous samples are often identified as fake by the discriminator due to their differences from normal samples. Variational autoencoders also learn the probability distribution of the data while reconstructing it. Anomalous samples are detected due to their deviation from the normal distribution.

[0006] Reconstruction-based methods are able to understand and reproduce complex spatial relationships and subtle structural differences to accurately identify tiny defects. At the same time, as the input data increases, the reconstruction model can continue to learn, thereby improving the accuracy and efficiency of detection and achieving real-time detection. Current 3D industrial anomaly detection overuses memory banks and relies on pre-trained models, resulting in large model memory overhead and difficulty in deployment on industrial production lines. In addition, some research is not enough to capture complex features. For example, the CNN-based 3D anomaly detection method 3D-ST has limitations in processing complex 3D features. It is difficult to capture global and local features at the same time, resulting in unsatisfactory detection performance. For example, M3DM constructs three memory banks for anomaly detection, resulting in huge memory requirements and slow inference speed. Summary of the Invention

[0007] To address the above challenges, this paper proposes a multimodal reconstruction framework for self-supervised learning, which consists of two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modality-shared and modality-specific components. Instead of using pre-trained models and memory bank methods, this paper adopts a reconstruction-based approach to reconstruct point clouds and RGB images. This approach effectively reduces the consumption of model computing resources and enhances the real-time performance of the model. To better interact the semantic and geometric information of 2D-3D, we introduce a local alignment attention mechanism in the joint encoder and a cross-modal reconstruction loss to achieve self-supervision. With this design, our framework can effectively fuse information from different modalities and improve the accuracy and robustness of anomaly detection.

[0008] The technical solution adopted by the present invention is: a multimodal industrial anomaly detection method based on a mask reconstruction mechanism, comprising the following steps:

[0009] Step (1): for a given 3D point cloud and corresponding 2D image input, a geometry-aware point cloud sampling method is used to process the original 3D point cloud to obtain a sampling point set;

[0010] Step (2): Build a hierarchical 2D-3D embedding module, combine the sampling point set, process and tokenize the 3D point cloud and 2D image separately, and randomly mask some of the tokens before extracting the visible tokens;

[0011] Step (3) builds a 2D-3D joint Transformer, which includes a joint encoder and a joint decoder. The processed 2D and 3D visible markers are input into the 2D-3D joint Transformer for multimodal reconstruction, where the joint decoder includes a modality-sharing decoder and a modality-specific decoder.

[0012] Step (4), after the joint decoder outputs the result, the masked part is restored through the linear layer, and the total loss is calculated by 2D reconstruction loss, 3D reconstruction loss and cross reconstruction loss;

[0013] In step (5), the trained modality-specific decoder is used as a feature extractor to extract and fuse the features of the input original and reconstructed 3D point clouds and 2D images respectively, thereby calculating and aggregating the 2D and 3D modality anomaly scores to obtain the final anomaly score map and complete anomaly detection.

[0014] Furthermore, the specific implementation of step (1) is as follows:

[0015] (1a) Computing normal vectors for the input original 3D point cloud: , in radius inside, point The neighborhood point set of , by solving the optimization problem: Available points Normal vector , the normal vector can make the point The sum of squares of distances from each point in the neighborhood to the corresponding plane is minimized; It is from the point Arrive vector, Represented by the normal vector Defined plane to point The signed distance of ;

[0016] (1b) Calculate the curvature of the point, the curvature value The calculation formula is:

[0017]

[0018] in and is the covariance matrix The eigenvalues of is a small positive number to avoid division by zero;

[0019] To quantify two points and The rate of change of the normal vector and curvature between is defined as follows:

[0020]

[0021]

[0022] in, It is from the point Arrive vector, and They are points and The normal vector of and They are points and Curvature value; introduce a rate of change memory library The difference in change rate between the record point and the neighboring points, the overall change rate value The calculation formula is:

[0023]

[0024] (1c) In point cloud processing, a geometrically aware point cloud sampling strategy is used to select points with high change rates, sort all points according to the change rates, select points with large change rates, and use thresholds To sample, select the first points, where is the total number of points in the point cloud; in order to capture more points with significant change rates, the final sampling point set The definition is as follows:

[0025]

[0026] in, Indicates sorting, and k indicates the index of the point whose rate of change meets the requirement.

[0027] Furthermore, the hierarchical 2D-3D embedding module in step (2) includes a 3D branch and a 2D branch. The specific processing process is as follows:

[0028] (2a) For the 3D branch, we first use two-level geometric perception sampling and K-nearest neighbor algorithm to obtain multi-scale representation of point cloud; in the first level, we filter out Points, is the total number of points in the point cloud; in the second level, further filter out points to capture the fine features of the point cloud; then use the linear projection layer to transform the features of the original point cloud, and then perform hierarchical tokenization through the grouping block composed of mini-PointNet, aggregate the K nearest neighbor local features, and finally generate C-dimensional 3D labels ;

[0029] (2b) For the 2D branch, first the 2D image is The patches are divided into blocks and embedded into high dimensions through the linear projection layer to improve the feature expression ability; then two Convolution block, the first convolution block extracts preliminary local features, the second convolution block refines and integrates the features to generate The C-dimensional 2D mark of the downsampling ratio is expressed as , H is the height of the 2D image, W is the width of the 2D image;

[0030] The 2D-3D embedding module formula is as follows:

[0031]

[0032] Represents the 3D mark generated after the 3D point cloud is processed by the 3D branch. Represents the original 3D point cloud data, It is a 3D embedding operation, representing the 3D branch from the original 3D point cloud To generate 3D markers A series of conversion processes;

[0033] Represents the 2D mark generated after the 2D image is processed by the 2D branch, represents the original 2D image data, It is a 2D embedding operation, corresponding to the 2D branch from the original 2D image To generate 2D markers The conversion process;

[0034] (2c) Randomly mask the 2D markers at a ratio of K%. and 3D markers , representing the visible markers as and .

[0035] Furthermore, the specific processing process of the joint encoder is as follows:

[0036] The visible markers of the two modalities are concatenated along the marker dimension. Before that, the input is constructed; specifically, the 2D visible markers are first With the corresponding modal-specific markup Additive, 3D visible markers and Additive, modal-specific markup and It is randomly initialized before training, and its dimension is , C represents the channel dimension; then the results of the above two additions are concatenated along the tag dimension to complete the input construction and obtain the input of the joint encoder:

[0037]

[0038] The joint encoder consists of k Transformer blocks with self-attention layers for 2D-3D feature interaction, which is formulated as:

[0039]

[0040] in represents a 2D-3D joint encoder, Represents the visible 2D-3D features of the input point cloud.

[0041] Furthermore, the specific processing process of the joint decoder is as follows:

[0042] Adopt two learnable C-dimensional masking markers and , and compare it with the encoded 2D-3D marker Concatenate along the tag dimension to get the input of the modality-sharing decoder:

[0043]

[0044] The modality sharing decoder contains k1 Transformer blocks with self-attention layers, which fuse and transform the input data. The output fused features are represented as , which can be formulated as:

[0045]

[0046] in, represents the modality shared decoder, Represents the decoded 2D-3D features;

[0047] Will Divide along the label dimension into and ,Will Divided into and , and then input two parallel specific modality decoders; the specific modality decoder contains an input layer for obtaining shallow features, an intermediate layer for obtaining mid-level features, and an output layer for obtaining high-level features; the intermediate layer contains a block that uses a cross-attention layer to As a query, visible features of 2D and 3D modalities processed by the joint encoder As keys and values, focus on 2D and 3D unique features for reconstruction to obtain modality-specific decoding features and , which can be formulated as:

[0048]

[0049]

[0050] in represent the modality-specific decoding features of 2D and 3D respectively, For 2D modality specific decoder, 3D modality specific decoder.

[0051] Furthermore, for the 2D reconstruction loss, the input is the original 2D image and the reconstructed image , the loss of 2D reconstruction is measured by calculating their difference under the mean square error MSE, which is calculated as:

[0052]

[0053] in is the i-th pixel of the original 2D image, is the i-th pixel of the reconstructed 2D image, and n is the total number of pixels;

[0054] Furthermore, for 3D reconstruction loss, use Cross distance calculation, the input is the original 3D point cloud and the reconstructed point cloud , the calculation formula is:

[0055]

[0056] in is the original 3D point cloud, is the reconstructed 3D point cloud, Is the original 3D point cloud The points in , q is the reconstructed 3D point cloud The point in Representing a collection The number of elements in , Representing a collection The number of elements in .

[0057] Furthermore, the input of the cross reconstruction loss is compared with the original 2D input image Reconstructed point cloud after projection from the same perspective and the original 2D image , the calculation process is as follows:

[0058] The reconstructed point cloud is projected through the same viewing angle as the original 2D input image, and the mean square error loss MSE between the projected point cloud and the original 2D image is calculated. The formula is:

[0059]

[0060] in, represents the original 2D image, is a projection operation.

[0061] Furthermore, the specific implementation of step (5) is as follows:

[0062] 5.1) The two modality-specific decoders used in the training phase are used as feature extractors to decode the input and reconstructed point clouds and 2D images, respectively, extracting three layers of features: low-level features, mid-level features, and high-level features. These three layers of features are then fused and compared.

[0063] 5.2) According to the fused point cloud features, the formula is: Calculate the anomaly score of the 3D modality, where Is the original 3D point cloud after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the input 3D point cloud; It is the reconstructed 3D point cloud after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 3D point cloud;

[0064] According to the fused image features, the formula: Calculate the anomaly score for the 2D modality, where is the original 2D image after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the input 2D image; is the reconstructed 2D image after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 2D image;

[0065] 5.3) Using the pixel-wise product as the aggregation function, the anomaly scores of the 2D and 3D modalities are aggregated to obtain the final anomaly score map. The calculation formula is as follows:

[0066]

[0067] This completes anomaly detection.

[0068] The present invention also provides a multimodal industrial anomaly detection system based on a mask reconstruction mechanism, comprising:

[0069] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a multimodal industrial anomaly detection method based on a mask reconstruction mechanism as described in the above technical solution.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] The present invention has significant advantages in the field of industrial anomaly detection: existing unsupervised detection methods based on feature embedding rely on pre-trained models and memory libraries, and consume large computing resources. The present invention adopts a reconstruction-based method to effectively reduce resource consumption, enhance real-time performance, and is more suitable for industrial production line deployment. In point cloud sampling, traditional sampling methods are difficult to clearly present abnormal structures. The geometric perception sampling of the present invention selects key sampling points based on the normal vector, curvature and rate of change of the point, which greatly improves the accuracy and effectiveness of detection. In terms of multimodal fusion, the innovative hierarchical embedding module and mask reconstruction mechanism, combined with the local alignment attention mechanism and cross-modal reconstruction loss, can deeply fuse multimodal information and enhance feature interaction and self-supervised learning. During anomaly detection, the multimodal anomaly score is calculated and aggregated by fusing features from different layers, which significantly improves the detection accuracy of subtle anomalies, providing a more efficient, accurate and adaptable solution for industrial anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 It is the overall technical flow chart of the present invention.

[0073] Figure 2 This is a network architecture diagram for the training phase of the present invention.

[0074] Figure 3 Schematic diagram of the local alignment attention mechanism of the present invention.

[0075] Figure 4 This is a network architecture diagram for the inference phase of the present invention. DETAILED DESCRIPTION

[0076] In order to better understand the technical solution of the present invention, the specific embodiments of the present invention are further described below with reference to the accompanying drawings.

[0077] like Figure 1 As shown, the overall implementation process of the industrial product 3D anomaly detection method based on multimodal self-supervised learning provided by the embodiment of the present invention is as follows:

[0078] Step 1: For a given 3D point cloud and the corresponding 2D image input, a geometrically aware point cloud sampling method is used to process the point cloud, and points with high change rates are selected to obtain a sampling point set. ;

[0079] 1.1) For the input original 3D point cloud, calculate the normal vector of the point. , in radius inside, point The neighborhood point set of By solving the optimization problem Available points Normal vector , the normal vector can make the point The sum of the squares of the distances from each point in the neighborhood to the corresponding plane is minimized. It is from the point Arrive vector, Represented by the normal vector Defined plane to point The signed distance of ;

[0080] 1.2) Calculate the curvature of the point, the curvature value The calculation formula is:

[0081]

[0082] in and is the covariance matrix The eigenvalues of is a small positive number to avoid division by zero.

[0083] To quantify two points and The rate of change of the normal vector and curvature between is defined as follows:

[0084]

[0085]

[0086] in, It is from the point Arrive vector, and They are points and The normal vector of and They are points and The curvature value of . Introducing a rate of change memory library The difference in change rate between the record point and the neighboring points, the overall change rate value The calculation formula is:

[0087]

[0088] 1.3) In point cloud processing, a geometrically aware point cloud sampling strategy is used to select points with high rates of change. All points are sorted according to their rates of change, and points with large rates of change are selected. A threshold is used To sample, select the first points, where is the total number of points in the point cloud. To capture more points with significant change rates, the final set of sampling points The definition is as follows:

[0089]

[0090] Step 2: Build a hierarchical 2D-3D embedding module, which consists of a 3D branch and a 2D branch. Use the hierarchical 2D-3D embedding module to perform specific processing and tokenization on the 3D point cloud and 2D image, and randomly mask some of the tokens before extracting the visible tokens.

[0091] 2.1) In the 3D branch, two-level geometry-aware sampling (GPS) and the k-nearest neighbor (k-NN) algorithm construct a multi-scale point cloud description system. Point cloud subsets at different scales reflect structural changes from global to local. Linear projection layers and mini-PointNet grouping blocks further transform this geometric information into 3D labels with hierarchical semantics.

[0092] For the 3D branch, we first use the two-level geometry-aware sampling (GPS) and K-nearest neighbor (k-NN) algorithm to obtain the multi-scale representation of the point cloud. In the first level, GPS and k-NN algorithms are used to filter out points, which can initially reflect the general structural characteristics of the point cloud; N is the total number of points in the point cloud; the sampling point set S provides data support for operations in the 3D branch. Although all subsequent operations in the 3D branch are not performed directly on the sampling point set S, the sampling point set S is obtained through geometrically aware sampling. The points it contains have a high rate of change and better reflect the key geometric characteristics of the point cloud. These features help to screen more representative points when further obtaining multi-scale representations through two-level geometry-aware sampling (GPS) and the K-nearest neighbor (k-NN) algorithm, thereby generating C-dimensional 3D labels that more accurately reflect the point cloud's structural information.

[0093] At the second level, further screening points, which focus more on capturing the local fine features of the point cloud. Through such two-level screening, the geometric form of the point cloud can be described from different levels. The linear projection layer is used to transform the features of the original point cloud so that it can adapt to subsequent processing and analysis. Then, hierarchical tokenization is performed through the grouping blocks composed of mini-PointNet. Mini-PointNet can effectively aggregate the local features within each K nearest neighbor, integrate and encode the local geometric information of the point cloud, and finally generate C-dimensional 3D tags. ,These markers contain the geometric and local feature information of the point cloud at different scales, providing a basis for subsequent ,crossmodal processing.

[0094] 2.2) In the 2D branch, image segmentation, linear projection, and convolution block operations gradually extract multi-scale features from macroscopic layout to microscopic texture and encode them into 2D labels. This multi-scale processing approach can better adapt to the complexity and diversity of data;

[0095] For the 2D branch, we first split the 2D image into patches of size The image is divided into blocks, and then a linear projection layer is used for high-dimensional embedding to convert the pixel information of the image into a higher-dimensional feature representation, thereby enhancing the expressive power of the image features. The convolution block gradually extracts features and tokenizes the image. The first convolution block can initially extract local features of the image, such as edges, textures, etc. The second convolution block further refines and integrates the features to generate a The C-dimensional 2D mark of the downsampling ratio is expressed as , H is the height of the 2D image, and W is the width of the 2D image. The formula of the 2D-3D embedding module is as follows:

[0096]

[0097] Represents the 3D mark generated after the 3D point cloud is processed by the 3D branch. Represents the original 3D point cloud data. It is a 3D embedding operation, representing the 3D branch from the original 3D point cloud To generate 3D markers A series of conversion processes.

[0098] Represents the 2D mark generated after the 2D image is processed by the 2D branch, Represents raw RGB image data. It is a 2D embedding operation, corresponding to the 2D branch from the original 2D image To generate 2D markers conversion process.

[0099] 2.3) On this basis, randomly mask the 2D markers at a ratio of 60% and 3D markers This masking operation creates a challenging precondition for self-supervised learning, where masked autoencoding is learned to represent the visible marker as and .

[0100] Step 3: Build a 2D-3D joint Transformer, which consists of a joint encoder and a joint decoder; input the processed 2D and 3D visible markers into the 2D-3D joint Transformer for multimodal reconstruction.

[0101] 3.1) The 2D-3D Joint Transformer aims to achieve deep fusion and feature interaction between 2D image and 3D point cloud modal data. Through the collaborative operation of a joint encoder and a joint decoder, it aggregates 2D semantics from 3D labels and incorporates 3D geometric information into 2D features, enhancing the understanding and representation of multimodal data to handle complex multimodal tasks.

[0102] In the joint encoder, two randomly initialized learnable modality markers are introduced to provide modality-specific cues to the attention layer, distinguishing 2D and 3D features and guiding encoding. Furthermore, a local alignment attention mechanism is implemented in all 12 Transformer blocks. In the attention layer, markers within each modality are mutually visible and attention is computed. Feature interaction attention scores are only calculated when the center point of the 3D marker is projected into the 2D marker; otherwise, the computation is ineffective.

[0103] 3.2) A joint encoder is used to connect the processed 2D and 3D visible markers as input. The joint encoder consists of 12 Transformer blocks with self-attention layers for 2D-3D feature interaction. The self-attention layer of each Transformer block can intelligently capture the correlation between each element in the input data. When processing 2D and 3D markers, these self-attention layers will automatically focus on important information areas based on the characteristics of the data and modal features, thereby achieving efficient feature interaction and fusion. The joint encoder is used to achieve 2D-3D feature interaction and cross-modal understanding. In this process, a local alignment attention mechanism is introduced. Local alignment attention refers to the local alignment between the 3D marker and the 2D marker. That is, when the center point of the 3D marker is projected into the 2D marker, the two calculate the feature interaction attention score. Otherwise, the calculation is invalid. Figure 3 This is a schematic diagram of the local alignment attention mechanism. The letters af in the figure represent different tags, namely the elements in the 2D tag and the 3D tag, to illustrate the relationship between different tags during the attention calculation process.

[0104] The local alignment attention mechanism is implemented as follows: the visible markers of the two modalities are connected along the marker dimension. Before this, the input needs to be constructed first. Specifically, the 2D visible markers are first With the corresponding modal-specific markup Additive, 3D visible markers and Additive, modal-specific markup and It is randomly initialized before training, and its dimension is , and then concatenate the two summed results along the tag dimension to complete the input construction and obtain the input of the joint encoder:

[0105]

[0106] In order to distinguish modality features in the attention mechanism, two learnable markers are used And randomly initialize them before training, introducing modality-specific encoding. is added to all 2D and 3D markers separately to provide modality-specific cues to the attention layer, which is formulated as:

[0107]

[0108] in represents a 2D-3D joint encoder, Represents the visible 2D-3D features of the input point cloud. The constructed input is then passed through multiple attention layers. The 3D labeling can adaptively aggregate information-rich 2D semantics, realize 2D-3D feature interaction, perform feature encoding, and finally obtain cross-modal understanding of point cloud and image feature representation. .

[0109] 3.3) Using a joint decoder, we introduce the masking mark and the encoded 2D-3D mark splicing input into the modality-shared decoder. The masked information is then reconstructed using a cross-attention layer through the modality-specific decoder. The specific implementation is as follows:

[0110] In the decoder, two learnable C-dimensional mask markers are used for 2D and 3D respectively. These markers are shared for all mask positions and are represented as and . Combine the masked marker with the encoded 2D-3D marker Concatenate along the tag dimension to get the input of the modality-sharing decoder: .

[0111] The modality sharing decoder consists of two Transformer blocks with self-attention layers. The input data is processed by these two Transformer blocks for feature fusion and transformation, and the output fused feature representation is . This can be formulated as:

[0112]

[0113] in represents the modality shared decoder, Represents the decoded 2D-3D features.

[0114] After the modality-shared decoder, along the token dimension Divided into 2D and 3D markers, represented as and ,Will Divided into and . They are then input into two parallel modality-specific decoders for specific 2D and 3D decoding, respectively, which enables the network to focus on the unique features of 2D and 3D for reconstruction. The network structure of the modality-specific decoder includes an input layer for obtaining shallow features, an intermediate layer for obtaining intermediate features, and an output layer for obtaining high-level features. In the multimodal anomaly detection method, its input layer receives the features output by the modality-sharing decoder and the visible features of the corresponding modality processed by the joint encoder, providing a data basis for subsequent processing. The intermediate layer contains a block that uses a cross-attention layer. This layer uses 2D and 3D labels as queries and the visible features processed by the joint encoder as keys and values to achieve 2D and 3D information complementarity and focus on unique feature reconstruction. The output layer outputs modality-specific decoding features, which are used for subsequent tasks such as calculating anomaly scores to determine whether the data has anomalies.

[0115] The middle layer of the modality-specific decoder contains one block, which uses a cross-attention layer. As a query, as keys and values. They are the visible features of the 2D and 3D modalities processed by the joint encoder, which can provide more modality-specific hints for the mask mark in the cross-attention mechanism, focusing on the unique 2D and 3D features for reconstruction, and obtaining the reconstructed 2D and 3D feature representations. and ; Formulate it as:

[0116]

[0117]

[0118] in Representing the modality-specific decoding features for 2D and 3D respectively.

[0119] Step 4: After the joint decoder outputs the result, the masked part is restored through the linear layer, and the total loss is calculated using the mean square error, L2 cross distance and cross reconstruction loss;

[0120] 4.1) Use two simple linear layers to recover the masked 2D pixels, following the MAE and Point-MAE methods. For the 2D reconstruction loss, the input is the original RGB image. and the reconstructed image In the multimodal reconstruction network, after a series of encoding, feature processing and other operations, , these features are further processed by the 2D decoder to finally generate the reconstructed image The loss of 2D reconstruction is measured by calculating their difference under mean square error (MSE), which is:

[0121]

[0122] in is the i-th pixel of the original 2D image, is the i-th pixel of the reconstructed 2D image, and n is the total number of pixels;

[0123] 4.2) For 3D reconstruction loss, the input is the original 3D point cloud and the reconstructed point cloud In the multimodal reconstruction network, the input point cloud data and 2D image data are processed by the network to obtain , further integrated, transformed, and processed by the 3D decoder to finally generate the reconstructed point cloud .use The cross distance calculation formula is:

[0124]

[0125] Among them is Original 3D point cloud, is the reconstructed 3D point cloud, Is the original 3D point cloud The points in , q is the reconstructed 3D point cloud The point in Representing a collection The number of elements in , Representing a collection The number of elements in .

[0126] 4.3) In order to achieve 2D-3D geometric alignment, an additional cross reconstruction loss is proposed. For the cross reconstruction loss, the input is compared with the original 2D input image. Reconstructed point cloud after projection from the same perspective and the original 2D image The reconstructed point cloud is projected at the same viewing angle as the original 2D input image, and the mean squared error loss (MSE) between the projected point cloud and the original 2D image is calculated using the formula:

[0127]

[0128] in, represents the original 2D image, It is a projection operation. This loss helps to self-supervise the reconstruction of the point cloud spatial structure and strengthen the 3D-2D geometric alignment and spatial consistency.

[0129] 4.4) The pre-training loss of the entire multimodal reconstruction network is the sum of the above three losses. The pre-training loss formula is:

[0130]

[0131] After obtaining the pre-training loss formula, the back propagation algorithm is used to optimize the model parameters based on the loss to complete the training.

[0132] Step 5: After completing the optimization of the model in the training phase, the multimodal anomaly detection uses the trained specific modality decoder as a feature extractor in the inference phase to extract three layers of features from the input and reconstructed point clouds and images, and fuse and compare them to obtain the 2D and 3D modality anomaly scores respectively and aggregate them to complete the anomaly detection.

[0133] 5.1) The two modality-specific decoders (frozen Transformers) used in the training phase serve as feature extractors. They decode the input and reconstructed 3D point clouds and 2D images, respectively, and extract three layers of features: low-level features, mid-level features, and high-level features. The shallow layers of the modality-specific decoders output low-level features, the middle layers output mid-level features, and the deep layers output high-level features. These three layers of features are then fused and compared.

[0134] 5.2) According to the fused point cloud features, the formula is: Calculate the anomaly score of the 3D modality, where Is the original 3D point cloud after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the input 3D point cloud. It is the reconstructed 3D point cloud after fusion These features are obtained by fusing the low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 3D point cloud.

[0135] According to the fused image features, the formula: Calculate the anomaly score for the 2D modality, where is the original 2D image after fusion These features are obtained by fusing the low-level features, mid-level features, and high-level features extracted after decoding the input 2D image. is the reconstructed 2D image after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 2D image;

[0136] 5.3) Using the pixel-by-pixel product as the aggregation function, we aggregate the anomaly scores of the 2D and 3D modalities to produce the final anomaly score map. This approach can be viewed as a logical AND: the anomaly score at a pixel location will be high only if both modal scores are high. This means that anomaly detection must be supported by both modalities. The final anomaly score calculation formula is as follows:

[0137]

[0138] This completes anomaly detection, such as Figure 2 Shown is a network architecture diagram of the training phase of the present invention, Figure 4 This is a network architecture diagram for the inference phase of the present invention.

[0139] Next, the present invention describes the effects of the invention through specific simulation experiments:

[0140] The specific hardware equipment for this simulation experiment is as follows: computer running memory is 32GB, processor model is i7-12700, graphics card is RTX3080Ti, and video memory is 12GB;

[0141] The specific software environment for this simulation experiment is: Ubuntu 20.04.5 operating system, Pytorch 1.10.0, CUDA 11.3, cuDNN 8, Python 3.7;

[0142] The datasets used in this simulation experiment are: the benchmark datasets MVTec3D and Eyescandies. MVTec3D-AD is the first industrial dataset for unsupervised 3D anomaly detection and localization tasks. This dataset contains 10 industrial object categories and a total of 4,147 scans acquired by high-resolution industrial 3D sensors that also capture RGB data. The Eyescandies dataset is a synthetic dataset that contains objects of 10 different categories, acquired in industrial conveyor belt scenarios, and includes 10,000 training samples, 1,000 validation samples, and 4,500 test samples. The RGB images and 3D point cloud data of the two datasets have the same resolution and are one-to-one corresponding at the pixel level.

[0143] Before the experiment began, we performed necessary data preprocessing on the original dataset to optimize model training and inference efficiency. Because 3D data typically contains a large number of background points that can interfere with anomaly detection, we removed these points by estimating a background plane at the edge of the point cloud. This was done using the RANSAC algorithm to estimate the background plane and remove points within a distance of 0.005 from this plane. For RGB images, we set the pixel values at the corresponding locations to 0 to ensure a consistent match between the RGB image and the 3D point cloud data.

[0144] Simulation experiments use the area under the receiver operating characteristic curve (AUROC) to evaluate anomaly detection performance at the image level (I-AUROC) and pixel level (P-AUROC). To more accurately evaluate the prediction results for each pixel, we use the region overlap ratio (PRO) and calculate the area under the PRO curve (AUPRO) as an evaluation metric for anomaly localization. This metric uses the generated anomaly score and the true connected components. Previous work typically uses an integration threshold of 0.3 for calculating AUPRO. We believe that this may be too loose for practical industrial applications, so we calculate AUPRO at a stricter integration threshold of 0.01. Higher values for all of the above metrics indicate better model performance.

[0145] This paper uses the aforementioned evaluation metrics to conduct comparative experiments against the following cutting-edge models. Table 1 shows the experimental results for the AUPRO@30% scores for anomaly detection in all categories of MVTec-3DAD for 3D and RGB+3D data. For 3D data, methods such as Voxel GAN, Voxel AE, and Voxel VM performed relatively poorly. For example, Voxel AE scored low in multiple categories. This may be due to limitations in these methods' feature extraction and anomaly detection mechanisms, which prevent them from fully capturing the complex anomaly features in 3D point cloud data. However, methods such as BTF (FPFH), M3DM, and Shape-guided achieved comparable results, demonstrating advantages in point cloud feature representation and anomaly identification. The proposed method also demonstrated high performance, achieving impressive scores across multiple categories. This is due to our unique multimodal reconstruction network architecture, which effectively utilizes the geometric features of point clouds for anomaly detection.

[0146] Table 1 AUPRO@30% scores for anomaly detection in all categories of MVTec-3D AD in 3D and RGB+3D data

[0147] In terms of RGB+3D data, Voxel GAN, Voxel AE, and Voxel VM also performed less than ideally. For example, Voxel AE scored extremely low in some categories, which may be because it failed to fully utilize the synergistic effect of the two when fusing RGB information and 3D data. Methods such as BTF (RGB+FPFH), M3DM, and Shape-guided still maintain good detection results, indicating that they have strong capabilities in processing multimodal data. The method of the present invention also achieved excellent results in RGB+3D data, approaching or exceeding other comparison methods. This is because our model has been carefully designed in terms of multimodal fusion and anomaly detection mechanisms, which can better utilize the complementary information of RGB images and 3D point clouds, thereby more accurately detecting anomalies. Overall, our method is highly competitive in anomaly detection in both 3D and RGB+3D data, and can maintain a high AUPRO@30% score in different categories, providing an effective solution for anomaly detection in the MVTec-3D AD dataset.

[0148] The Voxel GAN method comes from the literature: Bergmann, P., Jin, X., Sattlegger, D. and Steger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint arXiv:2112.09045;

[0149] The Voxel AE method comes from the literature: Bergmann, P., Jin, X., Sattlegger, D. andSteger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomalydetection and localization. arXiv preprint arXiv:2112.09045;

[0150] Voxel VM method source: Bergmann, P., Jin, X., Sattlegger, D. andSteger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomalydetection and localization. arXiv preprint arXiv:2112.09045;

[0151] BTF method source: Horwitz, E. and Hoshen, Y., 2023. Back to the feature: classical 3d features are (almost) all you need for 3d anomalydetection. In Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition (pp. 2968-2977);

[0152] AST method origin literature: Rudolph, M., Wehrbein, T., Rosenhahn, B. and Wandt, B., 2023. Asymmetric student-teacher networks for industrial anomaly detection. In Proceedings of the IEEE / CVF winter conference on applications of computer vision (pp. 2592-2602);

[0153] M3DM method source: Wang, Y., Peng, J., Zhang, J., Yi, R., Wang, Y. andWang, C., 2023. Multimodal industrial anomaly detection via hybrid fusion. InProceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8032-8041);

[0154] The Shape-guided method comes from the literature: Chu, YM, Chieh, L., Hsieh, TI, Chen, HT and Liu, TL, 2023. Shape-guided dual-memory learning for 3D anomalydetection.

[0155] In summary, in the field of anomaly detection for 3D and RGB+3D data, faced with the difficulty in effectively capturing complex point cloud and image data and potential anomaly features, the present invention constructs a multimodal reconstruction network. This network first uses a geometry-aware sampling module to process 3D point clouds, obtain key geometric features, and sample them. It then uses a layered 2D-3D embedding module to tokenize the data. Multimodal reconstruction is then performed using a 2D-3D joint Transformer comprising a joint encoder and decoder, and the reconstruction effect is optimized with the help of a local alignment attention mechanism and a cross-reconstruction loss. During the inference phase, a specific multimodal anomaly detection module is used to evaluate the reconstruction results. Experimental results on datasets such as MVTec-3D AD show that the proposed method achieves outstanding AUPRO@30% scores across different categories, significantly outperforming other methods such as Voxel GAN and Voxel AE.

[0156] On the other hand, an embodiment of the present invention further provides a multimodal industrial anomaly detection system based on a mask reconstruction mechanism, comprising:

[0157] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a multimodal industrial anomaly detection method based on a mask reconstruction mechanism as described in the above technical solution.

[0158] It should be understood that the above description of the implementation of the invention is relatively detailed, but it should not be considered as limiting the scope of protection of the patent of the present invention. Reasonable substitutions, modifications, and other improvements made by ordinary technicians in the field under the guidance of the present invention without departing from the scope of protection of the claims of the present invention should fall within the scope of protection of the present invention, and the scope of protection of the present invention should be based on the appended claims.

Claims

1. A multimodal industrial anomaly detection method based on mask reconstruction mechanism, characterized in that: The steps include: Step (1): for a given 3D point cloud and corresponding 2D image input, a geometry-aware point cloud sampling method is used to process the original 3D point cloud to obtain a sampling point set; Step (2): Build a hierarchical 2D-3D embedding module, combine the sampling point set, process and tokenize the 3D point cloud and 2D image separately, and randomly mask some of the tokens before extracting the visible tokens; Step (3) builds a 2D-3D joint Transformer, which includes a joint encoder and a joint decoder. The processed 2D and 3D visible markers are input into the 2D-3D joint Transformer for multimodal reconstruction, where the joint decoder includes a modality-sharing decoder and a modality-specific decoder. Step (4), after the joint decoder outputs the result, the masked part is restored through the linear layer, and the total loss is calculated by 2D reconstruction loss, 3D reconstruction loss and cross reconstruction loss; In step (5), the trained modality-specific decoder is used as a feature extractor to extract and fuse the features of the input original and reconstructed 3D point clouds and 2D images respectively, thereby calculating and aggregating the 2D and 3D modality anomaly scores to obtain the final anomaly score map and complete anomaly detection.

2. The multimodal industrial anomaly detection method based on a mask reconstruction mechanism according to claim 1, characterized in that: The specific implementation of step (1) is as follows: (1a) Computing normal vectors for the input original 3D point cloud: , in radius inside, point The neighborhood point set of , by solving the optimization problem: Available points Normal vector , the normal vector can make the point The sum of squares of distances from each point in the neighborhood to the corresponding plane is minimized; It is from the point Arrive vector, Represented by the normal vector Defined plane to point The signed distance of ; (1b) Calculate the curvature of the point, the curvature value The calculation formula is: ； in and is the covariance matrix The eigenvalues of is a small positive number to avoid division by zero; To quantify two points and The rate of change of the normal vector and curvature between is defined as follows: ； ； in, It is from the point Arrive vector, and They are points and The normal vector of and They are points and Curvature value; introduce a rate of change memory library The difference in change rate between the record point and the neighboring points, the overall change rate value The calculation formula is: ； (1c) In point cloud processing, a geometry-aware point cloud sampling strategy is used to store the point cloud in a rate-of-change memory library. Select the points with high rate of change, sort all points according to the rate of change, select the points with large rate of change, and use the threshold To sample, select the first points, where is the total number of points in the point cloud; in order to capture more points with significant change rates, the final sampling point set The definition is as follows: ； in, Indicates sorting, and k indicates the index of the point whose rate of change meets the requirement.

3. The multimodal industrial anomaly detection method based on a mask reconstruction mechanism according to claim 1, characterized in that: The hierarchical 2D-3D embedding module in step (2) includes a 3D branch and a 2D branch. The specific processing process is as follows: (2a) For the 3D branch, we first use two-level geometric perception sampling and K-nearest neighbor algorithm to obtain multi-scale representation of point cloud; in the first level, we filter out Points, is the total number of points in the point cloud; in the second level, further filter out points to capture the fine features of the point cloud; then use the linear projection layer to transform the features of the original point cloud, and then perform hierarchical tokenization through the grouping block composed of mini-PointNet, aggregate the K nearest neighbor local features, and finally generate C-dimensional 3D labels ; (2b) For the 2D branch, first the 2D image is The patches are divided into blocks and embedded into high dimensions through the linear projection layer to improve the feature expression ability; then two Convolution block, the first convolution block extracts preliminary local features, the second convolution block refines and integrates the features to generate The C-dimensional 2D mark of the downsampling ratio is expressed as , H is the height of the 2D image, W is the width of the 2D image; The 2D-3D embedding module formula is as follows: ； Represents the 3D mark generated after the 3D point cloud is processed by the 3D branch. Represents the original 3D point cloud data, It is a 3D embedding operation, representing the 3D branch from the original 3D point cloud To generate 3D markers A series of conversion processes; Represents the 2D mark generated after the 2D image is processed by the 2D branch, represents the original 2D image data, It is a 2D embedding operation, corresponding to the 2D branch from the original 2D image To generate 2D markers The conversion process; (2c) Randomly mask the 2D markers at a ratio of K%. and 3D markers , K is a constant used to control the proportion of masking, and the visible mark is expressed as and .

4. The multimodal industrial anomaly detection method based on a mask reconstruction mechanism according to claim 1, characterized in that: The specific processing process of the joint encoder is as follows: The visible markers of the two modalities are concatenated along the marker dimension. Before that, the input is constructed; specifically, the 2D visible markers are first With the corresponding modal-specific markup Additive, 3D visible markers and Additive, modal-specific markup and It is randomly initialized before training, and its dimension is , C represents the channel dimension; then the results of the above two additions are concatenated along the tag dimension to complete the input construction and obtain the input of the joint encoder: ； The joint encoder consists of k Transformer blocks with self-attention layers for 2D-3D feature interaction, which is formulated as: ； in represents a 2D-3D joint encoder, Represents the visible 2D-3D features of the input point cloud.

5. The multimodal industrial anomaly detection method based on mask reconstruction mechanism according to claim 1, characterized in that: The specific processing process of the joint decoder is as follows: Adopt two learnable C-dimensional masking markers and , and compare it with the encoded 2D-3D marker Concatenate along the tag dimension to get the input of the modality-sharing decoder: ； The modality sharing decoder contains k1 Transformer blocks with self-attention layers, which fuse and transform the input data. The output fused features are represented as , which can be formulated as: ； in, represents the modality shared decoder, Represents the decoded 2D-3D features; Will Divide along the label dimension into and ,Will Divided into and , and then input two parallel specific modality decoders; the specific modality decoder contains an input layer for obtaining shallow features, an intermediate layer for obtaining mid-level features, and an output layer for obtaining high-level features; the intermediate layer contains a block that uses a cross-attention layer to As a query, visible features of 2D and 3D modalities processed by the joint encoder As keys and values, focus on 2D and 3D unique features for reconstruction to obtain modality-specific decoding features and , which can be formulated as: ； ； in represent the modality-specific decoding features of 2D and 3D respectively, For 2D modality specific decoder, 3D modality specific decoder.

6. The multimodal industrial anomaly detection method based on a mask reconstruction mechanism according to claim 1, characterized in that: For the 2D reconstruction loss, the input is the original 2D image and the reconstructed image , the loss of 2D reconstruction is measured by calculating their difference under the mean square error MSE, which is calculated as: ； in is the i-th pixel of the original 2D image, is the i-th pixel of the reconstructed 2D image, and n is the total number of pixels.

7. The multimodal industrial anomaly detection method based on mask reconstruction mechanism according to claim 1, characterized in that: For 3D reconstruction loss, use Cross distance calculation, the input is the original 3D point cloud and the reconstructed point cloud , the calculation formula is: ； in is the original 3D point cloud, is the reconstructed 3D point cloud, Is the original 3D point cloud The points in , q is the reconstructed 3D point cloud The point in Representing a collection The number of elements in , Representing a collection The number of elements in .

8. The multimodal industrial anomaly detection method based on a mask reconstruction mechanism according to claim 1, characterized in that: The input of the cross reconstruction loss is the original 2D input image Reconstructed point cloud after projection from the same perspective and the original 2D image , the calculation process is as follows: The reconstructed point cloud is projected through the same viewing angle as the original 2D input image, and the mean square error loss MSE between the projected point cloud and the original 2D image is calculated. The formula is: ； in, represents the original 2D image, is a projection operation.

9. The multimodal industrial anomaly detection method based on mask reconstruction mechanism according to claim 1, characterized in that: The specific implementation of step (5) is as follows: 1) The two modality-specific decoders used in the training phase are used as feature extractors to decode the input and reconstructed point clouds and 2D images, respectively, extracting three layers of features: low-level features, mid-level features, and high-level features. These three layers of features are then fused and compared. 2) According to the fused point cloud features, the formula: Calculate the anomaly score of the 3D modality, where Is the original 3D point cloud after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the input 3D point cloud; It is the reconstructed 3D point cloud after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 3D point cloud; According to the fused image features, the formula: Calculate the anomaly score for the 2D modality, where is the original 2D image after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the input 2D image; is the reconstructed 2D image after fusion These features are obtained by fusion of low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 2D image; 3) Use the pixel-by-pixel product as the aggregation function to aggregate the anomaly scores of the 2D and 3D modalities to obtain the final anomaly score map. The calculation formula is as follows: ； This completes anomaly detection.

10. A multimodal industrial anomaly detection system based on mask reconstruction mechanism, characterized in that: include: A processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute a multimodal industrial anomaly detection method based on a mask reconstruction mechanism as described in any one of claims 1 to 9.

Images