A multi-modal industrial anomaly detection method and system based on a mask reconstruction mechanism

Abstract

The application discloses a kind of multi-modal industrial anomaly detection method and system based on mask reconstruction mechanism, the method is difficult to obtain high-quality labeled dataset in industrial production, and 3D multi-modal data fusion is difficult to use, etc. Discard pre-training model and memory bank method, adopt the way of reconstruction based on point cloud and RGB image processing, to reduce the consumption of computing resources, enhance real-time performance. The method of the application includes using mask reconstruction mechanism in RGB image and 3D point cloud data respectively, accurately extracting 2D and 3D data features, helping to improve anomaly detection performance; In the extraction of 3D point cloud information, introduce GPS geometric perception sampling, and add graph attention mechanism in the joint encoder, strengthen 2D-3D feature interaction and cross-modal understanding ability. The application carries out a large number of experiments on MVTec3D-AD and Eyescandies dataset, fully verifies the effectiveness of the method, provides strong support for industrial anomaly detection to achieve a new performance height, and gives a new idea for guaranteeing industrial product quality.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of industrial manufacturing and computer vision, and specifically relates to a multi-modal industrial anomaly detection method based on a mask reconstruction mechanism, which can be used for high-precision anomaly detection of various industrial products in industrial production scenarios to ensure product quality. BACKGROUND

[0002] In today's rapidly developing industrial manufacturing field, with the continuous progress of automation and intelligent technology, product quality control has become one of the key factors for enterprises to maintain competitiveness. As an important part of quality control, the core goal of industrial anomaly detection is to locate abnormal features in products during production or after production to minimize potential defects. With the rapid development of deep learning technology, image anomaly detection technology has made significant progress, and its accuracy and robustness have been significantly improved, gradually breaking away from the limitations of manual feature engineering and turning to learn more complex patterns to cope with diversified detection problems.

[0003] In actual application scenarios, due to the low proportion of abnormal areas, unknown abnormal patterns and high labeling costs, it is difficult to obtain high-quality labeled data sets. Traditionally, anomaly detection relies on human inspectors or image-based methods. In addition, the research and application of industrial anomaly detection is gradually shifting from traditional 2D images to more diverse 3D data. Recently, with the introduction of 3D anomaly detection benchmark datasets such as MVTec3D-AD, Eyecandies and Real3D-AD, especially the combination of point clouds and RGB images, more comprehensive product surface information can be provided, which is crucial for capturing product defects and promoting multi-modal industrial anomaly detection research. However, although 3D data has shown great potential in anomaly detection, how to effectively fuse and utilize these multi-modal data remains a challenge. For example, although point cloud data can provide unique spatial information, it is not as sensitive to color perception as RGB images. Therefore, how to utilize the redundancy and complementarity between modalities has become a key to improving the performance of industrial anomaly detection IAD.

[0004] Among existing deep learning-based detection methods, unsupervised methods are more widely used than supervised methods. Current industrial anomaly detection mainly includes feature embedding-based methods and reconstruction-based methods. Feature embedding-based methods usually use pre-trained feature extractors and memory banks, which require more computing resources. The asymmetric teacher-student network for industrial anomaly detection learns deep feature representations of data through pre-trained feature extractors, which are used to estimate the normal distribution of the data. In the testing phase, the deviation of the feature representation of the new sample from the normal distribution is used as an anomaly score. A typical example of this method is PatchCore, which locates abnormal areas by storing the features of normal samples in a memory bank and comparing the differences between test samples and features in the memory bank during testing. In addition, normalization flow-based methods, such as CFLOW-AD, map the features of test samples to a Gaussian distribution by learning the feature distribution of normal samples, and abnormal samples are detected due to their large deviation from the Gaussian distribution.

[0005] Reconstruction-based methods learn the representation of normal data using a model, and then detect anomalies by comparing the differences between test samples and model-reconstructed samples. These methods usually rely on generative models such as autoencoders, generative adversarial networks, and variational autoencoders. Autoencoders learn effective representations of data by minimizing the difference between input and reconstructed output, while abnormal samples often cannot be accurately reconstructed by the model, resulting in larger reconstruction errors. Generative adversarial networks distinguish between model-generated samples and real samples by introducing a discriminator, and abnormal samples are often identified as fake samples by the discriminator due to their differences from normal samples. Variational autoencoders learn the probability distribution of data while reconstructing, and abnormal samples are detected due to their deviation from the normal distribution.

[0006] Reconstruction-based methods can understand and reproduce complex spatial relationships and subtle structural differences to accurately identify minor defects. At the same time, as the amount of input data increases, the reconstruction model can continue to learn, thereby improving the accuracy and efficiency of detection and achieving real-time detection. Current 3D industrial anomaly detection overuses memory banks and relies on pre-trained models, resulting in large memory overhead and difficulty in deploying on industrial production lines. In addition, some research is not sufficient to capture complex features. For example, the CNN-based 3D anomaly detection method 3D-ST has limitations when dealing with complex 3D features. It is difficult to capture both global and local features simultaneously, resulting in unsatisfactory detection performance. For example, M3DM constructs three memory banks for anomaly detection, resulting in huge memory requirements and slow inference speed. SUMMARY

[0007] To address the above challenges, this paper proposes a multi-modal reconstruction framework for self-supervised learning, which contains two hierarchical 2D-3D embedding modules, a joint encoder, and a joint decoder with modal-shared and modal-specific components. Instead of using pre-trained models and memory bank methods, this paper adopts a reconstruction-based approach to reconstruct point clouds and RGB images. This approach effectively reduces the consumption of model computing resources and enhances the real-time performance of the model. To better interact with the semantic and geometric information of 2D-3D, we introduce a local alignment attention mechanism in the joint encoder and introduce a cross-modal reconstruction loss to achieve self-supervision. Through this design, our framework can effectively fuse information from different modalities, improving the accuracy and robustness of anomaly detection.

[0008] The technical scheme adopted by the present application is: a multi-modal industrial anomaly detection method based on a mask reconstruction mechanism, comprising the following steps:

[0009] Step (1), for a given 3D point cloud and corresponding 2D picture input, a geometry-aware point cloud sampling method is used to process the original 3D point cloud to obtain a set of sampled points;

[0010] Step (2), a hierarchical 2D-3D embedding module is built, and the sampled point set is combined to process and tokenize the 3D point cloud and 2D image, and the visible tokens are extracted after randomly masking part of the tokens;

[0011] Step (3), a 2D-3D joint Transformer is built, including a joint encoder and a joint decoder, and the processed 2D and 3D visible tokens are input into the 2D-3D joint Transformer for multi-modal reconstruction, wherein the joint decoder includes a modal-shared decoder and a specific modal decoder;

[0012] Step (4), after the joint decoder outputs the results, the masked part is restored through a linear layer, and the total loss is calculated by 2D reconstruction loss, 3D reconstruction loss, and cross-reconstruction loss;

[0013] Step (5), using the trained specific modal decoder as a feature extractor, the input original and reconstructed 3D point cloud and 2D image are respectively feature-extracted and fused to calculate the 2D and 3D modal anomaly scores and aggregate them to obtain the final anomaly score map, completing the anomaly detection.

[0014] Further, the specific implementation of step (1) is as follows:

[0015] (1a) Calculate the normal vector for the input 3D point cloud: for the input point cloud Within a radius , point the neighborhood point set of , the normal vector of point can be obtained by solving the optimization problem: , which can make the sum of squared distances from each point in the neighborhood of point to the corresponding plane minimum; where is the vector from point to point , and represents the signed distance from the plane defined by the normal vector to point ;

[0016] (1b) Calculate the curvature of point , the curvature value is calculated as follows:

[0017]

[0018] where and are the eigenvalues of the covariance matrix , and is a small positive number to avoid division by zero;

[0019] In order to quantify the rate of change of the normal vector and the curvature between two points and , the following formula is defined:

[0020]

[0021]

[0022] where is the vector from point to point , and and are the normal vectors of points and , respectively, and are the curvature values of points and , respectively; a change rate memory is introduced to record the difference in change rate between points and neighborhood points, and the overall change rate value is calculated as follows:

[0023]

[0024] (1c) In point cloud processing, a geometric-aware point cloud sampling strategy is used to select points with high change rates, and all points are sorted according to the change rate. Points with high change rates are selected, and a threshold sampling is performed to select the top points from the sorted list, where is the total number of points in the point cloud; to capture more points with significant variation rate, the final sampled point set is defined as follows:

[0025]

[0026] where, denotes the sorting and k denotes the index of the point with variation rate satisfying the requirement.

[0027] Further, the 2D-3D embedding module in step (2) includes a 3D branch and a 2D branch, and the specific processing process is as follows:

[0028] (2a) For the 3D branch, first, a two-stage geometric perception sampling and K-neighbor algorithm are used to obtain a multi-scale representation of the point cloud; in the first stage, the top points are selected, is the total number of points in the point cloud; in the second stage, the top points are further selected to capture fine features of the point cloud; then, a linear projection layer is used to convert the features of the original point cloud, and then a grouping block composed of a mini-PointNet is used for hierarchical tokenization to aggregate K-neighbor local features, and finally a C-dimensional 3D token is generated.

[0029] (2b) For the 2D branch, first, the 2D image is divided into patches, which are embedded into high dimensions through a linear projection layer to improve the expression ability of the features; then, two convolution blocks are used, the first convolution block extracts preliminary local features, and the second convolution block refines and integrates the features to generate a C-dimensional 2D token with a downsampling ratio, denoted as , where H is the height of the 2D image and W is the width of the 2D image.

[0030] The formula of the 2D-3D embedding module is as follows:

[0031]

[0032] denotes the 3D token generated by the 3D branch after processing the 3D point cloud, denotes the original 3D point cloud data, is a 3D embedding operation, representing a series of conversion processes from the original 3D point cloud to the generated 3D token ;

[0033] represents 2D labels generated by 2D branch processing on 2D images, represents original 2D image data, is a 2D embedding operation, corresponding to the conversion process from original 2D image to generate 2D labels ;

[0034] (2c) randomly mask 2D labels and 3D labels respectively at a proportion of K%, and represent the visible labels as and .

[0035] Further, the specific processing process of the joint encoder is as follows:

[0036] Connect the visible labels of the two modalities along the label dimension, before which input construction is performed; specifically, first add 2D visible labels and corresponding modality-specific labels , add 3D visible labels and , and add modality-specific labels and are randomly initialized before training, with a dimension of , and C represents the channel dimension; then connect the above two added results along the label dimension to complete input construction, and obtain the input of the joint encoder:

[0037]

[0038] The joint encoder is composed of k Transformer blocks with self-attention layers for 2D-3D feature interaction, which is formulated as:

[0039]

[0040] wherein represents a 2D-3D joint encoder, represents visible 2D-3D features of the input point cloud.

[0041] Further, the specific processing process of the joint decoder is as follows:

[0042] Two learnable C-dimensional mask labels and are used, which are spliced with the encoded 2D-3D labels along the label dimension to obtain the input of the modality-shared decoder:

[0043]

[0044] The modality-shared decoder contains k1 Transformer blocks with self-attention layers, which perform feature fusion and transformation on the input data, and output the fused feature representation as follows: Formulate it as follows:

[0045]

[0046] in, Indicates a modal shared decoder. This represents the decoded 2D-3D features;

[0047] Will Divided along the label dimension and ,Will Divided into and Then, two parallel modality-specific decoders are input; each modality-specific decoder contains an input layer for acquiring shallow features, an intermediate layer for acquiring mid-level features, and an output layer for acquiring high-level features; the intermediate layer contains a block employing a cross-attention layer to... As a query, the visible features of the 2D and 3D modalities after processing by the joint encoder. Using these as keys and values, unique 2D and 3D features are reconstructed to obtain modality-specific decoding features. and Formulate it as follows:

[0048]

[0049]

[0050] in These represent modality-specific decoding features for 2D and 3D, respectively. For 2D modality-specific decoders, A 3D modal-specific decoder.

[0051] Furthermore, for the 2D reconstruction loss, the input is the original 2D image. and the reconstructed image The loss of 2D reconstruction is measured by calculating the difference between them under the mean squared error (MSE). The calculation formula is as follows:

[0052]

[0053] in It is the i-th pixel of the original 2D image. It is the i-th pixel of the reconstructed 2D image, and n is the total number of pixels;

[0054] Further, for 3D reconstruction loss, use cross distance calculation, input is original 3D point cloud and reconstructed point cloud , the calculation formula is:

[0055]

[0056] wherein is the original 3D point cloud, is the reconstructed 3D point cloud, is the point in the original 3D point cloud , q is the point in the reconstructed 3D point cloud , represents the number of elements in the set , represents the number of elements in the set . Further, the input of the cross reconstruction loss is the reconstructed point cloud projected from the same perspective as the original 2D input image and the original 2D image

[0057] , the calculation process is as follows:

[0058] Project the reconstructed point cloud from the same perspective as the original 2D input image, calculate the mean square error loss MSE between the projected point cloud and the original 2D image, the formula is:

[0059]

[0060] wherein, represents the original 2D image, is a projection operation.

[0061] Further, the specific implementation method of step (5) is as follows:

[0062] 5.1) Take the two specific modal decoders in the training stage as feature extractors, respectively decode the input and reconstructed point cloud and 2D image, extract their three-layer features, and fuse and compare the three-layer features;

[0063] 5.2) According to the fused point cloud feature, calculate the 3D modal anomaly score through the formula: , wherein is the feature of the fused original 3D point cloud , which is the low-level feature, middle-level feature and high-level feature extracted after decoding the input 3D point cloud and fused; is the fused reconstructed 3D point cloud​​​ characteristics are low-level features, middle-level features and high-level features extracted after decoding the reconstructed 3D point cloud and are fused;

[0064] According to the fused image features, the abnormal score of the 2D modality is calculated by the formula: , wherein is the fused original 2D image characteristics are low-level features, middle-level features and high-level features extracted after decoding the input 2D image and are fused; is the fused reconstructed 2D image characteristics are low-level features, middle-level features and high-level features extracted after decoding the reconstructed 2D image and are fused;

[0065] 5.3) The abnormal scores of the 2D and 3D modalities are aggregated using a pixel-by-pixel product as an aggregation function to obtain a final abnormal score map, and the calculation formula is as follows:

[0066]

[0067] to complete the anomaly detection.

[0068] The application also provides a multi-modal industrial anomaly detection system based on a mask reconstruction mechanism, comprising:

[0069] a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the multi-modal industrial anomaly detection method based on the mask reconstruction mechanism as described in the above technical solution.

[0070] Compared with the prior art, the application has the following beneficial effects:

[0071] The application has obvious advantages in the field of industrial anomaly detection: existing unsupervised detection methods based on feature embedding rely on pre-trained models and memory banks, which consume a lot of computing resources, while the application uses a reconstruction-based method to effectively reduce resource consumption, enhance real-time performance, and better adapt to industrial production line deployment. In point cloud sampling, traditional sampling methods cannot clearly present abnormal structures, and the geometric perception sampling of the application selects key sampling points according to the normal vector, curvature and change rate of the points, greatly improving the accuracy and effectiveness of detection. In multi-modal fusion, the innovative hierarchical embedding module and mask reconstruction mechanism, combined with the local alignment attention mechanism and cross-modal reconstruction loss, can deeply fuse multi-modal information, enhance feature interaction and self-supervised learning. In anomaly detection, the multi-modal abnormal scores are calculated by fusing different layer features and are aggregated, which significantly improves the detection accuracy of subtle abnormalities and provides a more efficient, accurate and adaptable solution for industrial anomaly detection. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating the overall technical process of the present invention.

[0073] Figure 2 This is a network architecture diagram for the training phase of this invention.

[0074] Figure 3 This is a schematic diagram of the local alignment attention mechanism of the present invention.

[0075] Figure 4 This is a network architecture diagram for the inference stage of this invention. Detailed Implementation

[0076] To better understand the technical solution of the present invention, the specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0077] like Figure 1 As shown in the figure, the overall implementation process of the 3D anomaly detection method for industrial products based on multimodal self-supervised learning provided by this invention is as follows:

[0078] Step 1: For a given 3D point cloud and its corresponding 2D image input, process the point cloud using a geometry-aware point cloud sampling method, select points with high rates of change, and obtain a set of sampled points. ;

[0079] 1.1) The normal vector of the input point cloud is calculated from the original 3D point cloud. In radius inside, point The neighborhood point set is By solving the optimization problem Points can be obtained normal vector This normal vector can make the point The sum of the squared distances from each point in the neighborhood to its corresponding plane is minimized. From point Time The vector, Represented by the normal vector Defined plane to point The signed distance;

[0080] 1.2) Calculate the curvature of the point, curvature value The calculation formula is:

[0081]

[0082] in and It is the covariance matrix Eigenvalues of the characteristic, is a small positive number to avoid division by zero.

[0083] To quantify the rate of change of normal vector and curvature between two points and , the following formula is defined:

[0084]

[0085]

[0086] where, is the vector from point to point , and and are the normal vectors of points and , respectively, and are the curvature values of points and , respectively. A change rate memory bank is introduced to record the difference in change rate between points and neighboring points, and the overall change rate value is calculated as:

[0087]

[0088] 1.3) In point cloud processing, a geometry-aware point cloud sampling strategy is used to select points with high change rates. According to the change rate, all points are sorted, and points with high change rates are selected. A threshold is used to sample points from the sorted list, and the first points are selected from the sorted list, where is the total number of points in the point cloud. More points with significant change rates are captured, and the final sampling point set is defined as follows:

[0089]

[0090] Step 2, build a hierarchical 2D-3D embedding module composed of 3D branches and 2D branches, use the hierarchical 2D-3D embedding module to process and tokenization 3D point cloud and 2D image respectively, and extract visible tokens after randomly masking part of the tokens;

[0091] 2.1) In the 3D branch, two-level geometry-aware sampling (GPS) and K-Nearest Neighbor (k-NN) algorithm construct a multi-scale point cloud description system. Different scale point cloud subsets reflect the structure changes from the whole to the local, and the linear projection layer and mini-PointNet grouping block further convert these geometric information into 3D tokens with hierarchical semantics.

[0092] For the 3D branch, a two-level geometric perception sampling (GPS) and k-nearest neighbor (k-NN) algorithm is first used to obtain a multi-scale representation of the point cloud. In the first level, GPS and k-NN algorithms are used to filter... The sampled point set S represents the initial structural features of the point cloud; N is the total number of points in the point cloud; and the sampled point set S provides data support for the 3D branch operations. In the 3D branch, although not all subsequent operations are performed directly on the sampled point set S, S is obtained through geometrically perceptual sampling, and the points it contains have a high rate of change, better reflecting the key geometric features of the point cloud. These features, when further processed through two-level geometrically perceptual sampling (GPS) and the K-nearest neighbor (k-NN) algorithm to obtain multi-scale representations, help to filter out more representative points, thereby generating C-dimensional 3D labels that more accurately reflect the structural information of the point cloud.

[0093] In the second level, further screening is performed. These points are selected based on their local fine features, focusing on capturing the point cloud's finer details. This two-stage selection allows for the description of the point cloud's geometry at different levels. A linear projection layer transforms the original point cloud's features to adapt them for subsequent processing and analysis. Next, hierarchical labeling is performed using blocks composed of mini-PointNets. Mini-PointNets effectively aggregate local features within each K-nearest neighbor, integrating and encoding the point cloud's local geometric information to ultimately generate C-dimensional 3D labels. These labels contain geometric and local feature information of point clouds at different scales, providing a foundation for subsequent cross-modal processing.

[0094] 2.2) In the 2D branch, image segmentation, linear projection, and convolutional block operations progressively extract multi-scale features from macroscopic layout to microscopic texture and encode them into 2D labels. This multi-scale processing method can better adapt to the complexity and diversity of data.

[0095] For the 2D branch, first, the 2D image is set to a patch size of... The image is segmented into blocks, and then high-dimensional embedding is performed using a linear projection layer to transform the pixel information of the image into a higher-dimensional feature representation, thereby enhancing the expressive power of image features. Two layers are used... Convolutional blocks progressively extract and tokenize features from the image. The first convolutional block can initially extract local features of the image, such as edges and textures; the second convolutional block further refines and integrates these features to generate a more comprehensive representation. The C-dimensional 2D label of the downsampling ratio is represented as H is the height of the 2D image, and W is the width of the 2D image. The formula for the 2D-3D embedding module is as follows:

[0096]

[0097] represents 3D labels generated after 3D branch processing of 3D point cloud, represents original 3D point cloud data. is a 3D embedding operation, representing a series of conversion processes from original 3D point cloud to generate 3D labels .

[0098] represents 2D labels generated after 2D branch processing of 2D image, represents original RGB image data. is a 2D embedding operation, corresponding to a conversion process from original 2D image to generate 2D labels .

[0099] 2.3) On this basis, 2D labels and 3D labels are randomly masked at a proportion of 60% respectively. This masking operation creates a challenging pre-task for self-supervised learning, which learns through masked auto-encoding to represent visible labels as and .

[0100] Step 3, build a 2D-3D joint Transformer, which consists of a joint encoder and a joint decoder; input the processed 2D and 3D visible labels into the 2D-3D joint Transformer for multi-modal reconstruction.

[0101] 3.1) The 2D-3D joint Transformer aims to realize the deep fusion and feature interaction of 2D image and 3D point cloud modal data. Through the cooperative operation of joint encoder and joint decoder, it promotes the aggregation of 3D labels and the integration of 2D features into 3D geometric information, enhances the understanding and expression of multi-modal data, and processes complex multi-modal tasks.

[0102] In the joint encoder part, two randomly initialized learnable modal labels are introduced to provide modal-specific hints for attention layers, so as to distinguish 2D and 3D features and guide encoding. In addition, local alignment attention mechanism is introduced in all 12 Transformer blocks. In the attention layer, the labels within each modality are visible to each other and calculate attention, while only when the center point of the 3D label is projected into the 2D label, the two calculate feature interaction attention score, otherwise it is invalid calculation.

[0103] 3.2) Use the joint encoder to connect the processed 2D and 3D visible markers as input, which is composed of 12 Transformer blocks with self-attention layers for 2D-3D feature interaction. Each Transformer block can intelligently capture the correlation between elements in the input data. When processing 2D and 3D markers, these self-attention layers will automatically focus on important information areas according to the characteristics of the data and modal features, thereby achieving efficient feature interaction and fusion. The joint encoder is used to realize 2D-3D feature interaction and cross-modal understanding, and a local alignment attention mechanism is introduced in this process. Local alignment attention refers to the local alignment between 3D markers and 2D markers, i.e. when the center point of the 3D marker is projected into the 2D marker, the two calculate feature interaction attention scores, otherwise it is invalid calculation. Figure 3 The figure shows the local alignment attention mechanism, where letters a-f represent different markers, i.e. elements in 2D and 3D markers, to illustrate the relationship between different markers in the attention calculation process.

[0104] The local alignment attention mechanism is implemented as follows: connect the visible markers of the two modalities along the marker dimension. Before that, input construction is needed. Specifically, first add the 2D visible marker to the corresponding modal-specific marker , add the 3D visible marker to , and add the modal-specific marker to , which are randomly initialized before training, with dimensions . Then connect the above two added results along the marker dimension to complete the input construction and get the input of the joint encoder:

[0105]

[0106] To distinguish modal features in the attention mechanism, two learnable markers are used and randomly initialized before training, introducing modal-specific encoding. Two modal markers are added to all 2D and 3D markers to provide modal-specific hints for the attention layer, which is formulated as:

[0107]

[0108] where represents the 2D-3D joint encoder, Visible 2D-3D features representing the input point cloud. The constructed input is then passed through multiple attention layers, and the 3D labels can adaptively aggregate information-rich 2D semantics, realize 2D-3D feature interaction, and perform feature encoding to ultimately obtain cross-modal understanding of point cloud and image feature representations .

[0109] 3.3) Use a joint decoder to introduce a mask label and splice the encoded 2D-3D label into the input modal shared decoder, and then use a cross-attention layer in the specific modal decoder to reconstruct the masked information. The specific implementation is as follows:

[0110] Two learnable C-dimensional mask labels are used for 2D and 3D in the decoder, and these labels are shared for all mask positions, represented as and respectively. The mask label and the encoded 2D-3D label are spliced along the label dimension to obtain the input of the modal shared decoder: .

[0111] The modal shared decoder contains two Transformer blocks with self-attention layers. The input data is processed through these two Transformer blocks to perform feature fusion and conversion, and the output fused feature is represented as . It can be formulated as:

[0112]

[0113] where represents the modal shared decoder, and represents the decoded 2D-3D feature.

[0114] After the modal shared decoder, divide into 2D and 3D labels along the label dimension, represented as and , and divide into andThen, they are input into two parallel modal-specific decoders for specific 2D and 3D decoding, which enables the network to focus on unique features of 2D and 3D for reconstruction, respectively. The network structure of the modal-specific decoder includes an input layer for obtaining shallow features, an intermediate layer for obtaining middle features, and an output layer for obtaining high-level features. In the multi-modal anomaly detection method, the input layer receives the features output by the modal-shared decoder and the corresponding modal visible features processed by the joint encoder, providing a data basis for subsequent processing. The intermediate layer contains a block with a cross-attention layer, which takes 2D and 3D labels as queries, and the visible features processed by the joint encoder as keys and values, to realize the complementarity of 2D and 3D information and focus on unique features for reconstruction. The output layer outputs modal-specific decoding features, which are used for subsequent tasks such as calculating anomaly scores to determine whether the data is abnormal.

[0115] The intermediate layer of the modal-specific decoder contains a block, which adopts a cross-attention layer, where the 2D and 3D modal visible features processed by the joint encoder are taken as queries, and the 2D and 3D modal visible features processed by the joint encoder are taken as keys and values. They can provide more modal-specific hints for the masked labels in the cross-attention mechanism, focusing on unique 2D and 3D features for reconstruction, to obtain reconstructed 2D and 3D feature representations and , which can be formulated as:

[0116]

[0117]

[0118] where represent the 2D and 3D modal-specific decoding features, respectively.

[0119] Step 4: After the joint decoder outputs the results, the masked parts are recovered through a linear layer, and the total loss is calculated using mean squared error, L2 cross distance, and cross-reconstruction loss.

[0120] 4.1) Two simple linear layers are used to recover the masked 2D pixels, following the MAE and Point-MAE methods. For the 2D reconstruction loss, the input is the original RGB image and the reconstructed image . In the multi-modal reconstruction network, after a series of encoding, feature processing, and other operations, we get , which are further processed by the 2D decoder to generate the reconstructed image . The loss of 2D reconstruction is measured by calculating the difference between them in mean squared error (MSE), and the formula is:

[0121]

[0122] where is the i-th pixel of the original 2D image, is the i-th pixel of the reconstructed 2D image, n is the total number of pixels;

[0123] 4.2) For the 3D reconstruction loss, the input is the original 3D point cloud and the reconstructed point cloud . In the multi-modal reconstruction network, the input point cloud data and 2D image data are processed by the network to obtain , which is further integrated, transformed, and processed by the 3D decoder to generate the reconstructed point cloud . The cross distance is calculated using , and the formula is:

[0124]

[0125] where is the original 3D point cloud, is the reconstructed 3D point cloud, is the point in the original 3D point cloud , q is the point in the reconstructed 3D point cloud , represents the number of elements in the set , and represents the number of elements in the set .

[0126] 4.3) To achieve 2D-3D geometric alignment, an additional cross-reconstruction loss is proposed. For the cross-reconstruction loss, the input is the reconstructed point cloud projected from the same perspective as the original 2D input image and the original 2D image . The reconstructed point cloud is projected from the same perspective as the original 2D input image, and the mean squared error loss (Mean Squared Error) between the projected point cloud and the original 2D image is calculated, and the formula is:

[0127]

[0128] where, represents the original 2D image, is a projection operation, and this loss helps to supervise the spatial structure of the reconstructed point cloud, and strengthens the 3D-2D geometric alignment and spatial consistency;

[0129] 4.4) The pre-training loss of the entire multi-modal reconstruction network is the sum of the above three losses, and the pre-training loss formula is:​​​

[0130]

[0131] After obtaining the pre-training loss formula, the model parameters are optimized according to the loss by using the back propagation algorithm to complete the training.

[0132] Step 5, after the optimization of the model in the training stage is completed, the multi-modal anomaly detection uses the specific modal decoder obtained by training as the feature extractor in the inference stage, extracts three layers of features from the input and the reconstructed point cloud and image, and compares and fuses them to obtain 2D and 3D modal anomaly scores respectively and aggregate them to complete anomaly detection.

[0133] 5.1) The two specific modal decoders (frozen Transformers) in the training stage are used as feature extractors to respectively decode the input and the reconstructed 3D point cloud and 2D image, extract their three layers of features, and fuse and compare the three layers of features, where the three layers of features are low-level features, middle-level features and high-level features, the low-level features are output by the shallow layer of the specific modal decoder, the middle-level features are output by the intermediate layer, and the high-level features are output by the deep layer.

[0134] 5.2) According to the fused point cloud features, the 3D modal anomaly score is calculated by the formula: wherein is the feature of the fused original 3D point cloud , which is obtained by fusing the low-level features, middle-level features and high-level features extracted after decoding the input 3D point cloud. is the feature of the fused reconstructed 3D point cloud , which is obtained by fusing the low-level features, middle-level features and high-level features extracted after decoding the reconstructed 3D point cloud.

[0135] According to the fused image features, the 2D modal anomaly score is calculated by the formula: wherein is the feature of the fused original 2D image , which is obtained by fusing the low-level features, middle-level features and high-level features extracted after decoding the input 2D image. is the feature of the fused reconstructed 2D image , which is obtained by fusing the low-level features, middle-level features and high-level features extracted after decoding the reconstructed 2D image.

[0136] 5.3) Use the pixel-by-pixel product as the aggregation function to aggregate the anomaly scores of 2D and 3D modalities to obtain the final anomaly score map. This approach can be seen as a logical "and", only when the scores of the two modalities are both high, the anomaly score will be high at a certain pixel position, that is, the anomaly detection must be supported by two modalities. The final anomaly score calculation formula is as follows:

[0137]

[0138] Thus, the anomaly detection is completed, as shown in Figure 2 The network architecture diagram of the training stage of the present application is shown in the figure, Figure 4 The network architecture diagram of the inference stage of the present application is shown in the figure.

[0139] Next, the present application will be described by specific simulation experiments:

[0140] The specific hardware equipment of this simulation experiment is: the computer running memory is 32GB, the processor model is i7-12700, the graphics card is RTX3080Ti, and the graphics memory is 12GB;

[0141] The specific software environment of this simulation experiment is: Ubuntu20.04.5 operating system, Pytorch1.10.0, CUDA11.3, cuDNN8, Python3.7;

[0142] The data set used in this simulation experiment is: benchmark data set MVTec3D and Eyescandies. MVTec3D-AD is the first industrial data set for unsupervised 3D anomaly detection and positioning task. The data set contains 10 industrial object categories, a total of 4147 scanning data collected by high-resolution industrial 3D sensors, and the sensor also collects RGB data. The Eyescandies data set is a synthetic data set, which contains 10 different categories of objects, collected in an industrial conveyor belt scene, containing a total of 10,000 training samples, 1,000 validation samples and 4,500 test samples. The RGB images and 3D point cloud data of the two data sets have the same resolution and are one-to-one at the pixel level.

[0143] Before the experiment begins, we have carried out necessary data preprocessing on the original data set in order to optimize the training and inference efficiency of the model. Since 3D data usually contains a large number of background points, these background points may interfere with anomaly detection, so we remove the background points by estimating the background plane of the point cloud edge. The specific method is to use the RANSAC algorithm to estimate the background plane, and remove the points within 0.005 distance from the plane. For the RGB image, we set the pixel value at the corresponding position to 0 to ensure that the RGB image matches the 3D point cloud data.

[0144] The area under the receiver operating characteristic curve (AUROC) is used to evaluate the image-level (I-AUROC) and pixel-level (P-AUROC) anomaly detection performance of the simulation experiment. In order to more accurately evaluate the prediction results of each pixel, we use the region overlap rate (PRO) and calculate the area under the PRO curve (AUPRO) as the evaluation index of anomaly localization, which uses the generated anomaly score and the real connected component. Previous work usually uses 0.3 as the integral threshold for calculating AUPRO, and we think that the integral threshold 0.3 may be too loose for actual industrial application scenarios, so we calculate AUPRO under a more stringent integral threshold of 0.01. The higher the value of all the above indicators, the better the performance of the model.

[0145] The present application will use the above evaluation indicators to conduct comparative experiments on the following several frontier models. The experimental results of AUPRO@30% scores of all categories of MVTec-3D AD under 3D and RGB+3D data are shown in Table 1. In terms of 3D data, the performance of methods such as Voxel GAN, Voxel AE, Voxel VM is relatively poor, for example, the score of Voxel AE is low in multiple categories, which may be due to the limitations of these methods in feature extraction and anomaly detection mechanism, which cannot fully capture the complex abnormal feature of 3D point cloud data. Methods such as BTF (FPFH), M3DM, Shape-guided have achieved good results, and they have certain advantages in point cloud feature representation and anomaly discrimination. Among them, the method proposed by the present application also shows high performance and obtains good scores in multiple categories, which benefits from our unique multi-modal reconstruction network architecture that can effectively utilize the geometric features of point cloud for anomaly detection.

[0146] Table 1 AUPRO@30% scores of all categories of MVTec-3D AD under 3D and RGB+3D data

[0147]

[0148] In terms of RGB+3D data, Voxel GAN, Voxel AE, and Voxel VM also do not perform well, for example, Voxel AE has very low scores in some categories, which may be because it fails to fully utilize the synergy of RGB information and 3D data when fusing them. Methods such as BTF(RGB+FPFH), M3DM, Shape-guided, etc. still maintain good detection effects, indicating that they have strong capabilities in handling multi-modal data. The proposed method also achieves excellent results under RGB+3D data, approaching or exceeding other comparative methods, because our model is carefully designed in multi-modal fusion and anomaly detection mechanism, which can better utilize the complementary information of RGB images and 3D point clouds, thus more accurately detecting anomalies. Overall, our method has strong competitiveness in 3D and RGB+3D data anomaly detection, maintaining high AUPRO@30% scores in different categories, providing an effective solution for anomaly detection in the MVTec-3D AD dataset.

[0149] The Voxel GAN method is from the literature: Bergmann, P., Jin, X., Sattlegger, D. and Steger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint arXiv:2112.09045;

[0150] The Voxel AE method is from the literature: Bergmann, P., Jin, X., Sattlegger, D. and Steger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint arXiv:2112.09045;

[0151] The Voxel VM method comes from the literature: Bergmann, P., Jin, X., Sattlegger, D. and Steger, C., 2021. The mvtec 3d-ad dataset for unsupervised 3d anomaly detection and localization. arXiv preprint arXiv:2112.09045;

[0152] The BTF method comes from the literature: Horwitz, E. and Hoshen, Y., 2023. Back to the feature: classical 3d features are (almost) all you need for 3d anomaly detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2968-2977);

[0153] The AST method comes from the literature: Rudolph, M., Wehrbein, T., Rosenhahn, B. and Wandt, B., 2023. Asymmetric student-teacher networks for industrial anomaly detection. In Proceedings of the IEEE / CVF winter conference on applications of computer vision (pp. 2592-2602);

[0154] The M3DM method comes from the literature: Wang, Y., Peng, J., Zhang, J., Yi, R., Wang, Y. and Wang, C., 2023. Multimodal industrial anomaly detection via hybrid fusion. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8032-8041);

[0155] The shape-guided method comes from the literature: Chu, Y.M., Chieh, L., Hsieh, T.I., Chen, H.T. and Liu, T.L., 2023. Shape-guided dual-memory learning for 3D anomaly detection.

[0156] In summary, in the field of 3D and RGB+3D data anomaly detection, in the face of complex point cloud and image data and the problem that potential abnormal features are difficult to effectively capture, the present application constructs a multi-modal reconstruction network. The network first uses a geometric perception sampling module to process 3D point cloud, obtains key geometric features and samples; then uses a layered 2D-3D embedding module to mark the data; then performs multi-modal reconstruction through a 2D-3D joint Transformer containing a joint encoder and a decoder, and optimizes the reconstruction effect with the help of local alignment attention mechanism and cross-reconstruction loss. In the inference stage, a specific multi-modal anomaly detection module is used to evaluate the reconstruction result. From the experimental results of the MVTec-3D AD and other data sets, the AUPRO@30% score of the method proposed in the present application is outstanding in different categories, and has a significant advantage over other methods such as Voxel GAN, Voxel AE, etc.

[0157] On the other hand, the embodiment of the present application also provides a multi-modal industrial anomaly detection system based on a mask reconstruction mechanism, comprising:

[0158] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the storage instructions in the memory to execute a multi-modal industrial anomaly detection method based on a mask reconstruction mechanism as described in the above technical solution.

[0159] It should be understood that the above description of the application is more detailed, but it cannot be considered as a limitation of the scope of protection of the present application. Those skilled in the art can make reasonable substitutions or modifications without departing from the scope of the present application, and such improvements shall fall within the scope of protection of the present application. The scope of protection of the present application shall be subject to the appended claims.

Claims

1. A multimodal industrial anomaly detection method based on a mask reconstruction mechanism, characterized in that, Includes the following steps: Step (1): For a given 3D point cloud and the corresponding 2D image input, the original 3D point cloud is processed using a geometry-aware point cloud sampling method to obtain a set of sampling points; Step (2): Build a layered 2D-3D embedding module, combine the sampling point set, process and mark up the 3D point cloud and 2D image respectively, and extract the visible marks after randomly occluding some marks; Step (3): Build a 2D-3D joint Transformer, including a joint encoder and a joint decoder. Input the processed 2D and 3D visible markers into the 2D-3D joint Transformer for multimodal reconstruction. The joint decoder includes a modality-sharing decoder and a specific modality decoder. The specific processing procedure of the combined encoder is as follows: The visible markers of the two modalities are connected along the marker dimension. Before this, the input is constructed; first, the 2D visible markers are... With the corresponding modality-specific marker Addition, 3D visible marker and Addition, modality-specific markers and It is randomly initialized before training, with dimensions of... C represents the channel dimension; then the results of the above two additions are concatenated along the label dimension to complete the input construction, obtaining the input of the joint encoder: ； The joint encoder consists of k Transformer blocks with self-attention layers for 2D-3D feature interaction, and is formulated as follows: ； in Indicates a 2D-3D combined encoder. Represents the visible 2D-3D features of the input point cloud; The specific processing steps of the joint decoder are as follows: Employ two learnable C-dimensional masking markers and Combine it with the encoded 2D-3D tags By concatenating along the marked dimensions, we obtain the input to the modality-shared decoder: ； The modality-shared decoder contains k1 Transformer blocks with self-attention layers, which perform feature fusion and transformation on the input data, and output the fused feature representation as follows: Formulate it as follows: ； in, Indicates a modal shared decoder. This represents the decoded 2D-3D features; Will Divided along the label dimension and ,Will Divided into and Then, two parallel modality-specific decoders are input; each modality-specific decoder contains an input layer for acquiring shallow features, an intermediate layer for acquiring mid-level features, and an output layer for acquiring high-level features; the intermediate layer contains a block employing a cross-attention layer to... As a query, the visible features of the 2D and 3D modalities after processing by the joint encoder. Using these as keys and values, unique 2D and 3D features are reconstructed to obtain modality-specific decoding features. and Formulate it as follows: ； ； in These represent modality-specific decoding features for 2D and 3D, respectively. For 2D modality-specific decoders, A 3D modality-specific decoder; Step (4): After the joint decoder outputs the results, the occluded part is restored through a linear layer, and the total loss is calculated using 2D reconstruction loss, 3D reconstruction loss and cross reconstruction loss. Step (5): Using the trained specific modality decoder as a feature extractor, feature extraction and fusion are performed on the original and reconstructed 3D point cloud and 2D image respectively. The 2D and 3D modal anomaly scores are calculated and aggregated to obtain the final anomaly score map, thus completing the anomaly detection.

2. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: The specific implementation method of step (1) is as follows: (1a) For the input original 3D point cloud computing normal vector: For the input point cloud In radius inside, point The neighborhood point set is By solving the optimization problem: Points can be obtained normal vector This normal vector can make the point The sum of the squared distances from each point in the neighborhood to its corresponding plane is minimized; among them, From point Time The vector, Represented by the normal vector Defined plane to point The signed distance; (1b) Calculate the curvature of the point, curvature value The calculation formula is: ； in and It is the covariance matrix eigenvalues, This is to avoid dividing a small positive number by zero; To quantify two points and The rate of change of the normal vector and curvature between them is defined by the following formula: ； ； in, From point Time The vector, and They are points and The normal vector, and They are points and The curvature value; introduce a rate of change memory library. The difference in the rate of change between the record point and its neighboring points, and the overall rate of change value. The calculation formula is: ； (1c) In point cloud processing, a geometry-aware point cloud sampling strategy is adopted in the rate of change memory. Select points with high rates of change, sort all points based on their rate of change, and use a threshold. Perform sampling and select the top from the sorted list. The point, among which This represents the total number of points in the point cloud; the final set of sampled points is used to capture more points with significant rates of change. The definition is as follows: ； in, This indicates sorting, and k represents the index of the point where the rate of change meets the requirement.

3. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: The layered 2D-3D embedding module in step (2) includes 3D branches and 2D branches. The specific processing procedure is as follows: (2a) For the 3D branch, a two-level geometric perception sampling and K-nearest neighbor algorithm are first used to obtain the multi-scale representation of the point cloud; in the first level, the points are filtered out. One point, This is the total number of points in the point cloud; at the second level, further filtering is performed. The system first captures the fine features of the point cloud using a point network; then, it transforms the features of the original point cloud using a linear projection layer; finally, it performs hierarchical labeling using blocks composed of mini-PointNets, aggregating K-nearest neighbor local features to generate C-dimensional 3D labels. ; (2b) For the 2D branch, first convert the 2D image to... Patch blocks are segmented and embedded in a high-dimensional linear projection layer to enhance feature expressiveness; subsequently, two... Convolutional blocks: the first convolutional block extracts preliminary local features, and the second convolutional block refines and integrates these features to generate a convolutional block with... The C-dimensional 2D label of the downsampling ratio is represented as H is the height of the 2D image, and W is the width of the 2D image; The formula for the 2D-3D embedding module is as follows: ； This represents the 3D markers generated after 3D branching processing of a 3D point cloud. This represents the original 3D point cloud data. It is a 3D embedding operation, representing the process of embedding from the original 3D point cloud in the 3D branch. To generate 3D markers A series of transformation processes; This represents the 2D markers generated after a 2D image undergoes 2D branching processing. Represents the original 2D image data. This is a 2D embedding operation, corresponding to the 2D branch from the original 2D image. To generate 2D markers The conversion process; (2c) Randomly mask 2D markers at a rate of K%. and 3D marking K is a constant used to control the occlusion ratio, representing the visible marker as... and .

4. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: For 2D reconstruction loss, the input is the original 2D image. and the reconstructed image The loss of 2D reconstruction is measured by calculating the difference between them under the mean squared error (MSE). The calculation formula is as follows: ； in It is the i-th pixel of the original 2D image. It is the i-th pixel of the reconstructed 2D image, and n is the total number of pixels.

5. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: For 3D reconstruction loss, use The cross distance calculation takes the original 3D point cloud as input. and the reconstructed point cloud The calculation formula is: ； in It is the original 3D point cloud. It is a reconstructed 3D point cloud. It is the original 3D point cloud In the diagram, q represents the reconstructed 3D point cloud. The point in the middle, Represents a set The number of elements in Represents a set The number of elements in the array.

6. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: The input to the cross-reconstruction loss is the image processed by crossing the original 2D input image. Reconstructed point cloud after projection from the same viewpoint and the original 2D image The calculation process is as follows: The reconstructed point cloud is projected through the same viewpoint as the original 2D input image, and the mean square error loss (MSE) between the projected point cloud and the original 2D image is calculated using the following formula: ； in, Represents the original 2D image. It is a projection operation.

7. The multimodal industrial anomaly detection method based on mask reconstruction mechanism as described in claim 1, characterized in that: The specific implementation method of step (5) is as follows: 5.1) The decoders of two specific modalities during the training phase are used as feature extractors to decode the input and reconstructed point clouds and 2D images respectively, and extract their three-layer features, namely low-level features, medium-level features and high-level features. These three-layer features are then fused and compared. 5.2) Based on the features of the fused point cloud, using the formula: Calculate the anomaly score for the 3D modality, where It is the original 3D point cloud after fusion. The features are low-level features, mid-level features, and high-level features extracted after decoding the input 3D point cloud and then fused together. It is a reconstructed 3D point cloud after fusion. The features are low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 3D point cloud and then fused together. Based on the fused image features, using the formula: Calculate the anomaly score for the 2D modality, where It is the original 2D image after fusion. The features are low-level features, mid-level features, and high-level features extracted after decoding the input 2D image and then fused together. It is the fused reconstructed 2D image The features are low-level features, mid-level features, and high-level features extracted after decoding the reconstructed 2D image and then fused together. 5.3) Using pixel-wise multiplication as the aggregation function, the anomaly scores of the 2D and 3D modalities are aggregated to obtain the final anomaly score map. The calculation formula is as follows: ； This completes the anomaly detection.

8. A multimodal industrial anomaly detection system based on a mask reconstruction mechanism, characterized in that, include: The processor and memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the multimodal industrial anomaly detection method based on a mask reconstruction mechanism as described in any one of claims 1-7.

Images