Industrial anomaly detection method and device based on multi-mode joint memory pool

Abstract

The invention provides an industrial anomaly detection method and device based on a multi-modal joint memory pool, and belongs to the technical field of industrial anomaly detection, and the method comprises the steps: carrying out the multi-modal feature extraction of input RGB image data and point cloud data through a pre-training network; feature mapping and alignment operations are introduced to align point cloud features with RGB image features in space; two groups of intra-modal codec and two groups of cross-modal codec are introduced to perform intra-modal and cross-modal compression and reconstruction on the obtained RGB image features and point cloud features; training four groups of codecs by adopting a cosine similarity function; constructing a multi-modal feature memory pool; and adopting a nearest neighbor algorithm to calculate an abnormal score. According to the method, the RGB image and point cloud input are comprehensively utilized, abnormal areas of different scales and different forms can be accurately detected, and the universality and robustness are high.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial anomaly detection, and in particular relates to an industrial anomaly detection method and device based on a multimodal joint memory pool. Background Art

[0002] Industrial anomaly detection usually refers to the detection and location of defects such as scratches, potholes, and breakage on the surface of products. It plays a vital role in ensuring product quality, improving production efficiency, and reducing manufacturing costs. Traditional single-modal defect detection methods based on RGB images often face problems such as reduced accuracy under complex lighting conditions, insufficient sensitivity to surface texture changes, and lack of geometric feature description, resulting in unstable and unreliable detection results. In response to these limitations, we propose a multimodal defect detection algorithm based on point clouds and images. Through the multimodal data input of point clouds and RGB images, combined with the geometric shape and texture information of objects, it not only effectively improves the robustness of anomaly detection, but also significantly enhances the ability to recognize complex defects, achieving higher detection accuracy and reliability. Summary of the invention

[0003] The purpose of the present invention is to propose an industrial anomaly detection method and device based on a multimodal joint memory pool, which can realize accurate defect detection by utilizing multimodal information.

[0004] The present invention is achieved through the following technical solutions:

[0005] An industrial anomaly detection method based on a multimodal joint memory pool comprises the following steps:

[0006] Step S1, obtaining pre-trained features of the RGB image and pre-trained features of the point cloud of the input sample, wherein the RGB image features have rich texture and color information, and the point cloud features have rich geometric structure information;

[0007] Step S2, post-processing the point cloud features, mapping the point cloud features to a two-dimensional plane, so that they are aligned with the RGB features on the two-dimensional plane;

[0008] Step S3: construct four sets of feature codecs, including two sets of intra-modality codecs and two sets of cross-modality codecs, using the intra-modality codecs to encode and decode features of the same modality, and using the cross-modality codecs to encode and decode features of different modalities;

[0009] Step S4, constructing a loss function to train the feature codec, and obtaining a final loss function according to the losses of the feature codecs of different groups;

[0010] Step S5: Use multiple sets of feature codecs to obtain features of known normal samples, and store low-dimensional compressed features of different codecs to obtain a multimodal feature library. Perform feature clustering on each feature library using the KMeans algorithm, and construct a multimodal feature memory pool through the feature clustering center.

[0011] Step S6: In the inference stage, first obtain the RGB image features and point cloud features of the test sample, use four encoders to obtain multimodal compressed features, use the nearest neighbor search algorithm to find the nearest neighbor in the multimodal memory pool and calculate the L2 distance with the nearest neighbor, and obtain the final anomaly score by fusing the nearest neighbor distances of multiple memory pools.

[0012] Furthermore, in step S1, according to the formula Get the first feature of the input RGB image According to the formula Get the features of the point cloud input, where and Represent the RGB image input and point cloud input of the i-th sample, respectively, i∈{1,2,...,N}, where N represents the number of known normal samples, f rgb (·) represents the RGB image feature extractor, using the WideResNet50 network pre-trained on the ImageNet dataset, f pc (·) represents the point cloud feature extractor, using the PointMAE network pre-trained on the ShapeNet dataset.

[0013] Furthermore, in step S2, in the point cloud feature processing, the point cloud feature is grouped according to the center coordinates of the point cloud group obtained in the farthest point sampling. Mapped to a two-dimensional plane, the point cloud features are adjusted to match The same spatial resolution, and the smoothed point cloud features are obtained by a two-dimensional mean filter operation with a kernel size of 3

[0014] Furthermore, the step S3 specifically includes the following steps:

[0015] Step S31: Construct an intramodal encoder E whose input and output are both RGB features r→r With decoder D r→r , the output is expressed as The encoder E r→r The structure is a three-layer cascade CBR ↓ Module, including CBR ↓The module consists of a 3×3 convolutional layer-batch normalization layer-nonlinear activation layer. The number of output channels of the convolutional layer is half of the number of input channels. The momentum parameter in the batch normalization layer is set to 0.9. The nonlinear activation layer uses the ReLU activation function. The decoder D r→r The structure is a three-layer cascade CBR ↑ The module adds a linear mapping layer, where CBR ↑ The module consists of a 3×3 convolutional layer-batch normalization layer-nonlinear activation layer, where the number of output channels of the convolutional layer is twice the number of input channels, the momentum parameter in the batch normalization layer is set to 0.9, and the nonlinear activation layer uses the ReLU activation function;

[0016] Step S32: construct an intramodal encoder E whose input and output are both point cloud features p→p With decoder D p→p , the output is expressed as Use the same structure as the RGB feature codec;

[0017] Step S33: construct a cross-modal encoder / decoder E with RGB image features as input and point cloud features as output r→p With D r→p , the output is expressed as Use the same structure as the RGB feature codec;

[0018] Step S34: construct a cross-modal encoder / decoder E with point cloud features as input and RGB image features as output p→r With D p→r , the output is expressed as The same structure as the RGB feature codec is used.

[0019] Furthermore, in step S4: the loss function is expressed as L=L r→r +L p→p +L r→p +L p→r ,in is the feature reconstruction loss function within the RGB image modality, Cosine(·) represents the cosine similarity loss function, Reconstruct the loss function for the feature in the point cloud modality, and are the cross-modal feature reconstruction loss functions that represent RGB image features to point cloud features and point cloud features to RGB image features respectively.

[0020] Furthermore, in step S5, the multimodal feature pool is divided into four parts, namely: RGB feature memory pool, point cloud feature memory pool, cross-modal feature memory pool 1, and cross-modal feature memory pool 2. The above feature memory pools store cluster center features obtained by the KMeans algorithm.

[0021] Furthermore, in step S6, the parameters of the four encoders are frozen, and the anomaly scores obtained according to the four feature memory pools are fused by addition as the final detection result.

[0022] The present invention is also achieved through the following technical solutions:

[0023] 1. An industrial anomaly detection device based on a multimodal joint memory pool, characterized in that it includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:

[0024] Multimodal feature extraction module: obtain RGB image features and point cloud features respectively;

[0025] Point cloud feature alignment module: spatially aligns point cloud features with RGB image features;

[0026] Codec module: build multiple sets of feature compression and reconstruction networks formed by encoders and decoders, and train them according to the cosine similarity loss function;

[0027] Multimodal memory pool module: Four feature memory pools are constructed to store multimodal features, and anomaly detection is achieved based on the distance to the features in the memory pool as the anomaly score.

[0028] The present invention has the following beneficial effects:

[0029] 1. The present invention combines the RGB image features and point cloud features of the input samples to obtain rich detailed texture information and geometric information, and then constructs two intra-modal codecs and two cross-modal codecs, and then performs intra-modal feature reconstruction of RGB image, intra-modal feature reconstruction of point cloud, cross-modal feature reconstruction from RGB image to point cloud, and cross-modal feature reconstruction from point cloud to RGB image on normal samples, and then constructs four feature memory banks to store the low-dimensional compressed features output by the four encoders, and performs feature clustering on the four feature memory banks respectively through the KMeans algorithm, and finally constructs the cluster center feature. The feature memory pool is used, and the distance between the test sample feature and the known features in the four memory pools is used as the anomaly score to detect and locate defects. The intra-modal and cross-modal compressed features are obtained through multiple codecs, and the rich feature representation of the anomaly is obtained. By storing the compressed features output by the encoder and the KMeans algorithm, it is beneficial to reduce the memory consumption of the feature memory pool and improve the reasoning speed. Compared with the traditional single-modal anomaly detection method, the robustness and accuracy of anomaly detection are improved by fusing the feature information of RGB images and point clouds, which provides a new method for solving the problem of multi-modal surface anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described in detail below with reference to the accompanying drawings.

[0031] Figure 1 Flow chart of the method of the present invention.

[0032] Figure 2 It is a principle block diagram of the device of the present invention. DETAILED DESCRIPTION

[0033] like Figure 1 and Figure 2 As shown, an industrial anomaly detection method based on a multimodal joint memory pool includes the following steps:

[0034] Step S1, obtaining pre-trained features of the RGB image and point cloud pre-trained features of the input sample;

[0035] Specifically, the WideResNet50 network pre-trained on the ImageNet dataset and the PointMAE network pre-trained on the ShapeNet dataset are used as feature extractors. According to the formula Get the first feature of the input RGB image According to the formula Get the features of the point cloud input, where and Represent the RGB image input and point cloud input of the i-th sample, respectively, i∈{1,2,...,N}, where N represents the number of known normal samples, f rgb (·) represents the RGB image feature extractor, f pc (·) represents the point cloud feature extractor.

[0036] Step S2, post-processing the point cloud features, mapping the point cloud features to a two-dimensional plane, so that they are aligned with the RGB features on the two-dimensional plane;

[0037] Specifically, according to the horizontal and vertical coordinates of the center point of the point cloud group obtained in the farthest point sampling, the point cloud features are Mapped to a two-dimensional plane, the point cloud features are adjusted to match The same spatial resolution, and the smoothed point cloud features are obtained by a two-dimensional mean filter operation with a kernel size of 3

[0038] Step S3: construct four sets of feature codecs, including two sets of intra-modality codecs and two sets of cross-modality codecs, using the intra-modality codecs to encode and decode features of the same modality, and using the cross-modality codecs to encode and decode features of different modalities;

[0039] The specific steps include:

[0040] Step S31: Construct an intramodal encoder E whose input and output are both RGB features r→r With decoder D r→r , the output is expressed as The encoder E r→r The structure is a three-layer cascade CBR ↓ Module, including CBR ↓ The module consists of a 3×3 convolutional layer-batch normalization layer-nonlinear activation layer. The number of output channels of the convolutional layer is half of the number of input channels. The momentum parameter in the batch normalization layer is set to 0.9. The nonlinear activation layer uses the ReLU activation function. The decoder D r→r The structure is a three-layer cascade CBR ↑ The module adds a linear mapping layer, where CBR ↑ The module consists of a 3×3 convolutional layer-batch normalization layer-nonlinear activation layer, where the number of output channels of the convolutional layer is twice the number of input channels, the momentum parameter in the batch normalization layer is set to 0.9, and the nonlinear activation layer uses the ReLU activation function;

[0041] Step S32: construct an intramodal encoder E whose input and output are both point cloud features p→p With decoder D p→p , the output is expressed as Use the same structure as the RGB feature codec;

[0042] Step S33: construct a cross-modal encoder / decoder E with RGB image features as input and point cloud features as output r→p With D r→p , the output is expressed as Use the same structure as the RGB feature codec;

[0043] Step S34: construct a cross-modal encoder / decoder E with point cloud features as input and RGB image features as output p→r With D p→r , the output is expressed as The same structure as the RGB feature codec is used.

[0044] Step S4, constructing a loss function to train the feature codec, and obtaining a final loss function according to the losses of the feature codecs of different groups;

[0045] Specifically, the loss function is expressed as L = L r→r +L p→p +L r→p +L p→r ,in is the feature reconstruction loss function within the RGB image modality, Cosine(·) represents the cosine similarity loss function, Reconstruct the loss function for the feature in the point cloud modality, and are the cross-modal feature reconstruction loss functions that represent RGB image features to point cloud features and point cloud features to RGB image features respectively.

[0046] Step S5: Use multiple sets of feature codecs to obtain features of known normal samples, and store low-dimensional compressed features of different codecs to obtain a multimodal feature library. Perform feature clustering on each feature library using the KMeans algorithm, and construct a multimodal feature memory pool through the feature clustering center.

[0047] Specifically, the constructed multimodal feature pool is divided into four parts, namely: RGB feature memory pool, point cloud feature memory pool, cross-modal feature memory pool 1, and cross-modal feature memory pool 2. The above feature memory pools store the cluster center features obtained by the KMeans algorithm.

[0048] Step S6: In the inference stage, first obtain the RGB image features and point cloud features of the test sample, use four encoders to obtain multimodal compressed features, use the nearest neighbor search algorithm to find the nearest neighbor in the multimodal memory pool and calculate the L2 distance with the nearest neighbor, and obtain the final anomaly score by fusing the nearest neighbor distances of multiple memory pools;

[0049] Specifically, the parameters of the four encoders are frozen, and the anomaly scores obtained from the four feature memory pools are fused by addition as the final detection result. The normal multimodal features after compression by the encoder are closer to the features in the memory pool, while the abnormal features are farther away. Therefore, the distance to the features in the memory pool can be used as the basis for detecting anomalies.

[0050] Correspondingly, the industrial anomaly detection device based on the multimodal joint memory pool includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules:

[0051] Multimodal feature extraction module: obtain RGB image features and point cloud features respectively;

[0052] Point cloud feature alignment module: spatially aligns point cloud features with RGB image features;

[0053] Codec module: build multiple sets of feature compression and reconstruction networks formed by encoders and decoders, and train them according to the cosine similarity loss function;

[0054] Multimodal memory pool module: Four feature memory pools are constructed to store multimodal features, and anomaly detection is achieved based on the distance to the features in the memory pool as the anomaly score.

[0055] The above description is only a preferred embodiment of the present invention, and therefore cannot be used to limit the scope of implementation of the present invention. That is, equivalent changes and modifications made according to the scope of the patent application of the present invention and the contents of the specification should still fall within the scope covered by the patent of the present invention.

Claims

1. An industrial anomaly detection method based on multimodal joint memory pool, Features: The steps include: Step S1, using a pre-trained network to extract input RGB image features and point cloud features; Step S2, mapping the grouped point cloud features onto a two-dimensional plane and aligning them spatially with the RGB features; Step S3: construct four sets of feature codecs, including two sets of intra-modal codecs and two sets of cross-modal codecs; Step S4: construct a loss function to train the feature codec, and obtain the final loss function according to the losses of the feature codecs of different groups; Step S5: Use the feature encoder to obtain low-dimensional compressed features, and construct a multimodal feature memory pool through the KMeans algorithm; Step S6: Use the distance to the feature in the memory pool as the anomaly score, and fuse the anomaly scores from multiple memory pools as the final detection result.

2. According to claim 1, an industrial anomaly detection method based on a multimodal joint memory pool, Features: In step S1, the WideResNet50 network pre-trained on the ImageNet dataset is used as the RGB feature extractor, and the PointMAE network pre-trained on the ShapeNet dataset is used as the point cloud feature extractor. According to the industrial anomaly detection method based on a multimodal joint memory pool according to claim 1, it is characterized in that: in the step S2, the point cloud features are mapped to a two-dimensional plane according to the center coordinates of the point cloud grouping in the farthest point sampling operation, the features adjust the spatial resolution and are smoothed by a mean filtering operation.

3. According to claim 1, an industrial anomaly detection method based on a multimodal joint memory pool, Features: In step S3, the input and output of the intra-modal codec are both the same modal features, and the input and output of the cross-modal codec are different modal features.

4. According to claim 1, an industrial anomaly detection method based on a multimodal joint memory pool, Features: In step S4, the loss function uses the features of the input codec as a supervisory signal, and performs self-supervisory training based on the cosine similarity between the output features of the codec and the input features.

5. The industrial anomaly detection method based on multimodal joint memory pool according to claim 5, Features: In step S5, a feature memory pool is constructed using KMeans cluster center features, and the multimodal feature memory pool is divided into four parts, namely: RGB feature memory pool, point cloud feature memory pool, cross-modal feature memory pool 1, and cross-modal feature memory pool 2.

6. The industrial anomaly detection method based on multimodal joint memory pool according to claim 1, Features: In step S6, the nearest neighbor search algorithm is used to find the nearest neighbor in the multimodal memory pool and the L2 distance to the nearest neighbor is used as the anomaly score, and the nearest neighbor distances of multiple memory pools are combined to obtain the anomaly score of the sample.

7. An industrial anomaly detection device based on multimodal joint memory pool, Features: It includes a processor and a memory, the memory stores program instructions, and the processor calls the program instructions in the memory to implement the following modules: Multimodal feature extraction module: obtain RGB image features and point cloud features respectively; Point cloud feature alignment module: spatially aligns point cloud features with RGB image features; Codec module: build multiple sets of feature compression and reconstruction networks formed by encoders and decoders, and train them according to the cosine similarity loss function; Multimodal memory pool module: Four feature memory pools are constructed to store multimodal features, and anomaly detection is achieved based on the distance to the features in the memory pool as the anomaly score.

Images