A trimodal unsupervised industrial anomaly detection method based on reconstruction network

Abstract

The present invention provides a trimodal unsupervised industrial anomaly detection method based on a reconstruction network, which relates to the technical field of industrial anomaly detection. The present invention first restores the synthesized abnormal images of three different modalities into normal images, enabling the reconstruction network to capture and restore abnormal images. As a result, the method proposed by the present invention does not need to rely on an external memory module to store feature information during operation, significantly improving the deployment efficiency and response speed of the model in resource-constrained environments. In addition, the Transformer-based reconstruction network can effectively extract fine-grained features of the image. At the same time, the present invention fully utilizes a multi-layer self-attention mechanism to fuse texture details in RGB images, thermal distribution features in infrared images, and geometric shape information in 3D point cloud images. This not only improves the fine-grained recognition capability of anomaly detection, but also demonstrates strong robustness and generalization capabilities in complex industrial environments.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial anomaly detection, and in particular to a trimodal unsupervised industrial anomaly detection method based on a reconstruction network. Background Art

[0002] Unsupervised industrial anomaly detection is a key task in intelligent industrial production. It aims to automatically identify abnormal conditions such as product defects, production failures, and operational deviations in industrial scenarios using only normal data or a small amount of unlabeled mixed data when manually labeled abnormal samples are scarce, thereby achieving efficient and accurate automated production.

[0003] Traditional industrial anomaly detection relies primarily on manually defined standards or relies on visual inspection by industrial assembly line inspectors to check for product anomalies. However, with increasing production complexity and precision requirements, these limitations are becoming increasingly prominent. While these manually defined standards offer a high degree of uniformity for product inspection on industrial assembly lines, due to the limited experience and lags of engineers, these methods lack adaptability to complex working conditions and generalizability to emerging industries. Furthermore, these anomaly detection methods struggle to effectively extract high-dimensional data features from subtle local anomalies in product surface textures, limiting detection accuracy and reliability. While visual inspection utilizes both three-dimensional shape and color features to determine if a product is defective, the physical condition of human inspectors is easily affected by the length and intensity of their work, making it difficult to achieve the real-time and high-precision requirements of industrial assembly line anomaly detection.

[0004] In recent years, the rapid development of deep learning, multimodal learning, and computer vision technologies has brought new possibilities to unsupervised industrial anomaly detection, effectively promoting the automation and intelligent development of anomaly detection. Deep learning technologies, such as convolutional neural networks and Transformers, can effectively identify and locate abnormal characteristics of industrial products based on the normal characteristics of products on industrial assembly lines, leveraging their powerful learning capabilities. This reduces redundant manual operating costs and significantly improves detection accuracy and speed. In particular, the combination of multimodal data such as RGB images, infrared images, and 3D point cloud images further enhances the efficiency and accuracy of unsupervised industrial anomaly detection.

[0005] While unsupervised industrial anomaly detection plays a crucial role in intelligent manufacturing, its technological development still suffers from numerous shortcomings in data quality and algorithm performance. These shortcomings not only make it difficult to deploy industrial anomaly detection models on industrial equipment but also severely restrict their effectiveness in actual industrial applications.

[0006] First, at the data level, a single modality can only capture some of the abnormal features of assembly line products and cannot fully characterize the diverse faults such as surface defects and structural anomalies that may exist simultaneously in industrial scenarios. The limitations of this single-source information, coupled with the severe data imbalance caused by the scarcity of abnormal samples, make it impossible for the model to establish a complete abnormal feature system and to cover the various abnormal patterns in industrial scenarios. Secondly, at the algorithm level, because multimodal information is only processed through simple feature splicing, it performs weakly in multimodal collaborative analysis and fails to fully explore the correlation between data. In application scenarios, faced with complex and changing industrial environments, existing unsupervised industrial anomaly detection methods often show a lack of adaptability.

[0007] Therefore, researchers have proposed an anomaly detection method based on a memory bank. This method uses a convolutional neural network to extract and fuse the features of normal samples in the training set and store these features in a memory bank. During the testing phase, the extracted test image features are compared with the features in the memory bank. If the difference is within a set threshold, the sample is judged to be a normal sample, otherwise it is an anomaly sample. Although this type of method improves detection robustness to a certain extent, it comes with significant memory overhead and inference latency. Deploying such a method not only faces the consumption of a large amount of hardware resources and deployment costs, but also leads to a significant decrease in model inference speed, making it difficult to meet the strict real-time and high efficiency requirements of industrial production lines, thus limiting its implementation in practical scenarios. Summary of the Invention

[0008] In view of the shortcomings of the existing technology, the purpose of the present invention is to propose a trimodal unsupervised industrial anomaly detection method based on a reconstruction network, comprising:

[0009] Step 1: Obtain the target product that meets the preset production standards in the industrial production process, and then obtain the RGB image of the target product, and use the RGB image of the target product as the normal RGB image , obtain the infrared image of the target product, and use the infrared image of the target product as the normal infrared image , obtain the 3D point cloud image of the target product, and use the 3D point cloud image of the target product as the normal 3D point cloud image ;

[0010] Step 2: Process the normal RGB image, the normal infrared image and the normal 3D point cloud image to obtain an abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images ;

[0011] Step 3: Convert the abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images As input sample, the normal RGB image As output samples, the input samples and the output samples constitute training samples, and multiple training samples constitute a training sample set;

[0012] Step 4: Abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images Reconstruct and obtain the reconstructed RGB image features , reconstruct infrared image features and reconstruct 3D point cloud image features ;

[0013] Step 4.1: For the input samples in the training sample set, the abnormal RGB images in the input samples are Perform preprocessing to obtain a preprocessed RGB image, input the preprocessed RGB image into the first reconstruction network, and obtain the reconstructed RGB image features in the first reconstruction network ;

[0014] Step 4.2: Abnormal infrared images in the input sample Perform preprocessing to obtain a preprocessed infrared image, input the preprocessed infrared image into the second reconstruction network to obtain the reconstructed infrared image features ;

[0015] Step 4.3: Abnormal 3D point cloud images in the input sample Perform preprocessing to obtain a preprocessed 3D point cloud image, input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image features ;

[0016] Step 5: Reconstruct RGB image features through multi-layer self-attention fusion module , reconstruct infrared image features and reconstruct 3D point cloud image features Fusion is performed to obtain the final fusion feature ;

[0017] Step 6: Calculate the final fusion features and normal RGB images Outliers s , determine outliers s Is it greater than the preset threshold? sIf the value is greater than the preset threshold, the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are modified to obtain the next training sample in the training sample set, and return to step 4. s When the value is not greater than a preset threshold, the parameters of the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are fixed;

[0018] Step 7: Obtain the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected of the industrial product to be detected, reconstruct the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected to obtain the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image, and then fuse the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image based on the multi-layer self-attention fusion module to obtain the fused feature to be detected, and then calculate the outlier value between the fused feature to be detected and the RGB image to be detected. When the outlier value is greater than the preset threshold, it indicates that the industrial product to be detected has an anomaly. When the outlier value is not greater than the preset threshold, it indicates that the industrial product to be detected does not have an anomaly.

[0019] Optionally, step 2 is specifically implemented by the following formula:

[0020] ;

[0021] ;

[0022] ;

[0023] in, represents the anomaly segmentation mask generated by the Perlin noise generator and binarized, for The inverse matrix of Represents element-wise multiplication operation; is the opacity parameter in the blending process; Represents an abnormal texture source image in an existing abnormal source image dataset.

[0024] Optionally, the preprocessing in step 4.1, step 4.2, and step 4.3 specifically includes:

[0025] ;

[0026] in, t = r , i , p , represents the first abnormal image after division and flattening, Indicates the N Anomaly images after segmentation and flattening; Indicates the number of divisions; is a learnable projection matrix; Encode for position; is the preprocessed image.

[0027] Optionally, the first reconstruction network, the second reconstruction network and the third reconstruction network are all encoder-decoder structures, the encoder includes L layers of multi-head self-attention and feedforward network, and the decoder includes L layers of multi-head self-attention, cross attention and feedforward network.

[0028] Optionally, in each layer of the encoder, the input image is processed by a multi-head self-attention and feedforward network, which is specifically implemented by the following formula:

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] in, Indicates the encoder The output of the multi-head self-attention layer; Indicates the The output of the multi-head self-attention mechanism of the layer; Indicates the i Head self-attention mechanism, represents the total number of attention heads, Represents a splicing operation, is the output projection matrix, Indicates the i Query of attention heads, Indicates the i The key of an attention head, Indicates the i The value of an attention head; represents the activation function; Indicates the i The query matrix of the attention heads, Indicates the i The key matrix of the attention heads, Indicates thei The value matrix of the attention head; Represents the dimension of the key in the multi-head self-attention mechanism; Represents the result of multi-head self-attention output after layer normalization; represents the output of the feedforward neural network, represents the first layer weight matrix; represents the second layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; Representation layer normalization function; Indicates the encoder l The output of the layer;

[0036] Thus, after L layers of processing of the encoder, the output of the Lth layer of the encoder is obtained .

[0037] Optionally, in each layer of the decoder, the input image is processed through multi-head self-attention, cross-attention and feed-forward networks, which is implemented by the following formula:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] in, Indicates the l The output of the multi-head self-attention layer is the result of layer normalization. Indicates the decoder The output of the layer, Indicates the decoder The output of the multi-head self-attention mechanism, It represents the result of the cross attention layer output after layer normalization, represents the output of the cross-self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross attention mechanism, represents the query matrix, represents the matrix of bonds, A matrix representing the values, Represents the result of the feedforward network output after layer normalization, represents the dimension of the key in the cross-attention mechanism, Represents the input of the decoder, that is, the output of the encoder ;

[0044] Thus, after the decoder L The decoder gets the L Output of the layer , the decoder L Output of the layer as reconstructed image features.

[0045] Optionally, step 5 specifically includes:

[0046] Step 5.1: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct infrared image features Perform fusion to obtain the first initial fusion feature , which is specifically achieved through the following formula:

[0047] ;

[0048] ;

[0049] ;

[0050] in, Represents the reconstructed RGB image features and reconstruct infrared image features The spliced ​​image features; represents the query of the first layer of self-attention mechanism, represents the key of the first layer of self-attention mechanism, represents the value of the first layer of self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0051] Step 5.2: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct 3D point cloud image features Fusion is performed to obtain the second initial fusion feature , which is specifically achieved through the following formula:

[0052] ;

[0053] ;

[0054] ;

[0055] in, Represents the reconstructed RGB image features and reconstruct 3D point cloud image features The spliced ​​image features, represents the query of the second layer of self-attention mechanism, represents the key of the second layer of self-attention mechanism, represents the value of the second layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0056] Step 5.3: Through the multi-layer self-attention fusion mechanism, the first initial fusion feature and the second initial fusion feature Perform hybrid fusion to obtain hybrid fusion features , which is specifically achieved through the following formula:

[0057] ;

[0058] ;

[0059] ;

[0060] in, Represents the first initial fusion feature and the second initial fusion feature The spliced ​​image features, represents the query of the third layer of self-attention mechanism, represents the key of the third layer of self-attention mechanism, Represents the value of the third layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0061] Step 5.4: Mixed fusion features and reconstruct RGB image features Splice to get the final fusion feature , which is specifically achieved through the following formula:

[0062] .

[0063] Optionally, calculate the final fusion feature in step 6 and normal RGB images Outliers s , which is specifically achieved through the following formula:

[0064] ;

[0065] in, represents the L2 norm.

[0066] The beneficial effects of adopting the above technical solution are:

[0067] The present invention first uses the reconstruction network as the core framework of trimodal unsupervised industrial anomaly detection. By restoring the synthesized abnormal images of three different modalities to normal images, the reconstruction network is able to capture and restore abnormal images. Therefore, the method proposed in the present invention does not need to rely on an external memory module to store feature information during operation, effectively avoiding the large amount of memory consumption and performance bottlenecks in the inference process brought by the memory-based memory library method, and significantly improving the deployment efficiency and response speed of the model in resource-constrained environments. In addition, the Transformer-based reconstruction network can effectively extract fine-grained features of the image. At the same time, the present invention makes full use of the multi-layer self-attention mechanism to fuse the texture details in the RGB image, the thermal distribution features in the infrared image, and the geometric shape information in the 3D point cloud image, which not only improves the fine-grained recognition ability of anomaly detection, but also shows strong robustness and generalization ability in complex industrial environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Schematic diagram of a process of a trimodal unsupervised industrial anomaly detection method based on a reconstruction network in an embodiment of the present invention;

[0069] Figure 2 This is an overall framework diagram of a trimodal unsupervised industrial anomaly detection method based on a reconstruction network in an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0071] To address the challenges of existing technologies, this paper aims to develop a trimodal unsupervised industrial anomaly detection method based on a reconstruction network. First, the present invention leverages the complementary information between RGB images, infrared images, and 3D point cloud images to significantly improve the accuracy and robustness of industrial anomaly detection, overcoming the limitations of single-source data. Furthermore, the present invention utilizes a reconstruction network-based approach and multi-layer attention fusion to significantly improve the performance of the anomaly detection model while significantly reducing the model's memory storage space.

[0072] Specifically, the present invention uses an anomaly generator to synthesize normal RGB images, normal infrared images, and normal 3D point cloud images from an unsupervised industrial anomaly detection dataset into abnormal RGB images, abnormal infrared images, and abnormal 3D point cloud images. The present invention then trains a reconstruction network composed of Transformers to reconstruct the abnormal RGB images, abnormal infrared images, and abnormal 3D point cloud images into reconstructed RGB images, reconstructed infrared images, and reconstructed 3D point cloud images. The goal is to enable the Transformer-based reconstruction network to extract trimodal abnormal image features and restore them to normal image features. Next, the present invention uses a multi-layer self-attention fusion mechanism to pairwise fuse the three modal reconstruction features extracted by the Transformer-based reconstruction network. Specifically, the multi-layer self-attention fusion mechanism is used to fuse the reconstructed RGB image features with the reconstructed infrared image features and the reconstructed 3D point cloud image features to obtain two initial fused features. The two initial fused features are then fused using the multi-layer self-attention fusion mechanism to obtain a mixed fused feature, which is then concatenated with the reconstructed RGB image features to obtain the final fused feature. Finally, the final fused feature and the normal RGB image are fed into an anomaly discriminator for image anomaly identification and localization.

[0073] Specifically, the present invention provides a trimodal unsupervised industrial anomaly detection method based on a reconstruction network, combining Figure 1 and Figure 2 , which may include the following steps:

[0074] Step 1: Obtain the target product that meets the preset production standards in the industrial production process, and then obtain the RGB image of the target product, and use the RGB image of the target product as the normal RGB image , obtain the infrared image of the target product, and use the infrared image of the target product as the normal infrared image , obtain the 3D point cloud image of the target product, and use the 3D point cloud image of the target product as the normal 3D point cloud image ;

[0075] The preset production standard can be a product standard on an industrial assembly line. If a product has defects such as damage, stains, scratches, deformation, roller marks, pits, roughness, ripples, color difference, etc., then the product does not meet the preset production standard; the RGB image, infrared image and 3D point cloud image corresponding to the target product that meets the preset production standard are normal RGB images. , normal infrared image and normal 3D point cloud image .

[0076] Among them, RGB images can be obtained by taking pictures with a camera, infrared images can be obtained by taking pictures with an infrared sensor, and 3D point cloud images can be obtained by laser scanning, structured light or other three-dimensional imaging technologies.

[0077] Step 2: Process the normal RGB image, the normal infrared image and the normal 3D point cloud image to obtain an abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images , which is specifically achieved through the following formula:

[0078] ;

[0079] ;

[0080] ;

[0081] in, represents the anomaly segmentation mask generated by the Perlin noise generator and binarized, for The inverse matrix of Represents element-wise multiplication operation; is the opacity parameter in the blending process; Represents an abnormal texture source image in an existing abnormal source image dataset.

[0082] Among them, the existing abnormal source image dataset is an image library specifically used to generate and train anomaly detection models. The large number of abnormal samples contained in it may come from historical defect records, experimental data, artificially synthesized defect images, or are obtained by simulating different fault conditions in industrial production.

[0083] Among them, step 2 can be understood as processing the normal image through the abnormal generator to obtain the abnormal image.

[0084] Step 3: Convert the abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images As input sample, the normal RGB image As output samples, the input samples and the output samples constitute training samples, and multiple training samples constitute a training sample set;

[0085] Step 4: Abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images Reconstruct and obtain the reconstructed RGB image features , reconstruct infrared image features and reconstruct 3D point cloud image features ;

[0086] Step 4.1: For the input samples in the training sample set, the abnormal RGB images in the input samples are Perform preprocessing to obtain a preprocessed RGB image, input the preprocessed RGB image into the first reconstruction network, and obtain the reconstructed RGB image features in the first reconstruction network ;

[0087] Step 4.2: Abnormal infrared images in the input sample Perform preprocessing to obtain a preprocessed infrared image, input the preprocessed infrared image into the second reconstruction network to obtain the reconstructed infrared image features ;

[0088] Step 4.3: Abnormal 3D point cloud images in the input sample Perform preprocessing to obtain a preprocessed 3D point cloud image, input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image features ;

[0089] The pre-processing in step 4.1, step 4.2 and step 4.3 specifically includes:

[0090] ;

[0091] in, t = r , i , p , represents the first abnormal image after division and flattening, Indicates the N Anomaly images after segmentation and flattening; Indicates the number of divisions; is a learnable projection matrix; Encode for position; is the preprocessed image.

[0092] That is, in step 4.1 t = r ,at this time, and Expressed as and , that is, the first abnormal RGB image after division and flattening and the Nth abnormal RGB image after division and flattening, and then, Represented as the preprocessed RGB image, similarly in step 4.2 t = i ,at this time, and Expressed as and , that is, the first abnormal infrared image after being divided and flattened and the Nth abnormal infrared image after being divided and flattened, and then, Represented as the preprocessed infrared image; in step 4.3 t = p ,at this time, and Expressed as and , that is, the first abnormal 3D point cloud image after division and flattening and the Nth abnormal 3D point cloud image after division and flattening, and then, Represented as a preprocessed 3D point cloud image.

[0093] Among them, the first reconstruction network, the second reconstruction network and the third reconstruction network are all encoder-decoder structures, the encoder includes L layers of multi-head self-attention and feedforward network, and the decoder includes L layers of multi-head self-attention, cross attention and feedforward network.

[0094] It should be noted that the first reconstruction network, the second reconstruction network and the third reconstruction network are all reconstruction networks based on Transformer. The architectures of the first reconstruction network, the second reconstruction network and the third reconstruction network are the same, except that the parameters of the encoder and decoder are different.

[0095] In each layer of the encoder, the input image is processed by multi-head self-attention and feedforward network, which is specifically achieved by the following formula:

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] in, Indicates the encoder The output of the multi-head self-attention layer; Indicates the The output of the multi-head self-attention mechanism of the layer; Indicates the i Head self-attention mechanism, h represents the total number of attention heads, Represents a splicing operation, is the output projection matrix, Indicates the i Query of attention heads, Indicates the i The key of an attention head, Indicates the i The value of an attention head; represents the activation function; Indicates the i The query matrix of the attention heads, Indicates the i The key matrix of the attention heads, Indicates the i The value matrix of the attention head; Represents the dimension of the key in the multi-head self-attention mechanism; Represents the result of multi-head self-attention output after layer normalization; represents the output of the feedforward neural network, represents the first layer weight matrix; represents the second layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; Representation layer normalization function; Indicates the encoder l The output of the layer;

[0103] Thus, after L layers of processing of the encoder, the output of the Lth layer of the encoder is obtained .

[0104] in, The input of the first layer of the encoder is processed by L layers to obtain the output of the Lth layer of the encoder. , the output of the Lth layer of the encoder As the input of the first layer of the decoder, .

[0105] In each layer of the decoder, the input image is processed through multi-head self-attention, cross-attention and feedforward networks, which is specifically implemented by the following formula:

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] in, Indicates the l The output of the multi-head self-attention layer is the result of layer normalization. Indicates the decoder The output of the layer, Indicates the decoder The output of the multi-head self-attention mechanism, It represents the result of the cross attention layer output after layer normalization, represents the output of the cross-self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross attention mechanism, represents the query matrix, represents the matrix of bonds, A matrix representing the values, Represents the result of the feedforward network output after layer normalization, represents the dimension of the key in the cross-attention mechanism, Represents the input of the decoder, that is, the output of the encoder ;

[0112] Thus, after the decoder L The decoder gets the L Output of the layer , the decoder L Output of the layer as reconstructed image features.

[0113] It can be seen that in the present invention, the preprocessed RGB image is input into the encoder-decoder of the first reconstruction network, and the decoder can be obtained. L Output of the layer , that is, reconstructing RGB image features At the same time, the preprocessed infrared image is input into the encoder-decoder of the second reconstruction network, and the decoder can be obtained. L Output of the layer , that is, reconstructing infrared image features At the same time, the pre-processed 3D point cloud image is input into the encoder-decoder of the third reconstruction network, and the decoder can be obtained. L Output of the layer , that is, reconstructing 3D point cloud image features .

[0114] Step 5: Reconstruct RGB image features through multi-layer self-attention fusion module , reconstruct infrared image features and reconstruct 3D point cloud image features Fusion is performed to obtain the final fusion feature ;

[0115] Step 5.1: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct infrared image features Perform fusion to obtain the first initial fusion feature , which is specifically achieved through the following formula:

[0116] ;

[0117] ;

[0118] ;

[0119] in, Represents the reconstructed RGB image features and reconstruct infrared image features The spliced ​​image features; represents the query of the first layer of self-attention mechanism, represents the key of the first layer of self-attention mechanism, represents the value of the first layer of self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0120] Step 5.2: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct 3D point cloud image features Fusion is performed to obtain the second initial fusion feature , which is specifically achieved through the following formula:

[0121] ;

[0122] ;

[0123] ;

[0124] in, Represents the reconstructed RGB image features and reconstruct 3D point cloud image features The spliced ​​image features, represents the query of the second layer of self-attention mechanism, represents the key of the second layer of self-attention mechanism, represents the value of the second layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0125] Step 5.3: Through the multi-layer self-attention fusion mechanism, the first initial fusion feature and the second initial fusion feature Perform hybrid fusion to obtain hybrid fusion features , which is specifically achieved through the following formula:

[0126] ;

[0127] ;

[0128] ;

[0129] in, Represents the first initial fusion feature and the second initial fusion feature The spliced ​​image features, represents the query of the third layer of self-attention mechanism, represents the key of the third layer of self-attention mechanism, Represents the value of the third layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions;

[0130] Step 5.4: Mixed fusion features and reconstruct RGB image features Splice to get the final fusion feature , which is specifically achieved through the following formula:

[0131] .

[0132] Step 6: Calculate the final fusion features and normal RGB images Outliers s , which is specifically achieved through the following formula:

[0133] ;

[0134] in, represents the L2 norm.

[0135] Among them, the process of calculating the outlier value can be understood as the final fusion feature in the anomaly discriminator. and normal RGB images Perform discrimination and obtain outliers s .

[0136] Identifying outliers s Is it greater than the preset threshold? s If the value is greater than the preset threshold, the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are modified to obtain the next training sample in the training sample set, and return to step 4. s When the value is not greater than a preset threshold, the parameters of the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are fixed;

[0137] Among them, the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are modified. Specifically, the query, key, value mapping weights in the multi-head self-attention mechanism in the encoder, and the linear transformation weights in the feedforward network can be modified. 、 and bias parameters 、 Modifications are made. At the same time, in the decoder, in addition to the above structure, the weight matrix of the cross attention mechanism for inter-modal information integration is also modified. 、 、 , and finally modify the parameters of the self-attention fusion module used to fuse the three-modal features.

[0138] Step 7: Obtain the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected of the industrial product to be detected, reconstruct the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected to obtain the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image, and then fuse the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image based on the multi-layer self-attention fusion module to obtain the fused feature to be detected, and then calculate the outlier value between the fused feature to be detected and the RGB image to be detected. When the outlier value is greater than the preset threshold, it indicates that the industrial product to be detected has an anomaly. When the outlier value is not greater than the preset threshold, it indicates that the industrial product to be detected does not have an anomaly.

[0139] In summary, in order to reduce the resource overhead and computing cost pressure during model deployment, while improving the speed and real-time performance of anomaly detection, the present invention adopts an efficient and lightweight reconstruction network structure. This structure is centered on the reconstruction of abnormal image features. By learning the feature distribution of normal images, it realizes the reconstruction of the input image, thereby judging the abnormal area through the reconstruction error in the inference stage, eliminating the heavy memory feature matching operation and significantly improving the inference efficiency of the model. The reconstruction network combines the advantages of global feature modeling in the Transformer architecture to enhance the model's ability to perceive fine-grained features of multimodal information.

[0140] In addition, in response to the bottleneck problems of existing single-modal unsupervised industrial anomaly detection methods, such as insufficient data representation capabilities, poor robustness, and difficulty in adapting to complex and changeable industrial scenarios, the present invention proposes a tri-modal unsupervised industrial anomaly detection solution. This solution achieves deep multi-modal feature extraction and interaction through a multi-layer self-attention fusion mechanism, fully integrating the detailed texture information of RGB images, the thermal distribution characteristics of infrared images, and the spatial structure information of 3D point cloud images, effectively improving the model's ability to understand and generalize multi-source data. Therefore, by combining the Transformer-based reconstruction network and the multi-layer self-attention fusion mechanism, while ensuring the accuracy of anomaly detection, the present invention has stronger environmental adaptability, providing an efficient, accurate and scalable solution for intelligent detection under complex working conditions in industrial production processes.

[0141] This paper proposes a trimodal unsupervised industrial anomaly detection method based on a reconstruction network, providing an efficient, stable, and practical solution for resource-constrained industrial inspection platforms. Using a Transformer-based reconstruction network, this method constructs a lightweight trimodal anomaly detection model with low memory usage and fast inference capabilities, effectively meeting the dual real-time and reliability requirements of industrial production lines. A multi-head self-attention mechanism models the deep dependencies between modalities and guides the reconstruction network to learn the normal feature space of the product. During the testing phase, the model automatically identifies anomaly regions based on the reconstruction error of the input image, without relying on manual annotation or additional memory modules, significantly improving the flexibility and efficiency of model deployment. Furthermore, this paper leverages a multi-layer self-attention mechanism to integrate texture details in RGB images, thermal distribution features in infrared images, and geometric shape information in 3D point cloud images. This not only improves the fine-grained recognition capabilities of anomaly detection, but also demonstrates strong robustness and generalization capabilities in complex industrial environments. Ultimately, it achieves excellent performance with high-precision recognition, high-speed inference, and low resource consumption, providing solid technical support for intelligent quality inspection of industrial products.

[0142] The above description is merely an illustration of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art should understand that the scope of the invention encompassed by the embodiments of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A trimodal unsupervised industrial anomaly detection method based on reconstruction network, characterized in that: include: Step 1: Obtain the target product that meets the preset production standards in the industrial production process, and then obtain the RGB image of the target product, and use the RGB image of the target product as the normal RGB image , obtain the infrared image of the target product, and use the infrared image of the target product as the normal infrared image , obtain the 3D point cloud image of the target product, and use the 3D point cloud image of the target product as the normal 3D point cloud image ; Step 2: Process the normal RGB image, the normal infrared image and the normal 3D point cloud image to obtain an abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images ; Step 3: Convert the abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images As input sample, the normal RGB image As output samples, the input samples and the output samples constitute training samples, and multiple training samples constitute a training sample set; Step 4: Abnormal RGB image , abnormal infrared images and abnormal 3D point cloud images Reconstruct and obtain the reconstructed RGB image features , reconstruct infrared image features and reconstruct 3D point cloud image features ; Step 4.1: For the input samples in the training sample set, the abnormal RGB images in the input samples are Perform preprocessing to obtain a preprocessed RGB image, input the preprocessed RGB image into the first reconstruction network, and obtain the reconstructed RGB image features in the first reconstruction network ; Step 4.2: Abnormal infrared images in the input sample Perform preprocessing to obtain a preprocessed infrared image, input the preprocessed infrared image into the second reconstruction network to obtain the reconstructed infrared image features ; Step 4.3: Abnormal 3D point cloud images in the input sample Perform preprocessing to obtain a preprocessed 3D point cloud image, input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image features ; Step 5: Reconstruct RGB image features through multi-layer self-attention fusion module , reconstruct infrared image features and reconstruct 3D point cloud image features Fusion is performed to obtain the final fusion feature ; Step 5.1: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct infrared image features Perform fusion to obtain the first initial fusion feature , which is specifically achieved through the following formula: ； ； ； in, Represents the reconstructed RGB image features and reconstruct infrared image features The spliced ​​image features; represents the query of the first layer of self-attention mechanism, represents the key of the first layer of self-attention mechanism, represents the value of the first layer of self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions; Step 5.2: Reconstruct RGB image features through multi-layer self-attention fusion mechanism and reconstruct 3D point cloud image features Fusion is performed to obtain the second initial fusion feature , which is specifically achieved through the following formula: ； ； ； in, Represents the reconstructed RGB image features and reconstruct 3D point cloud image features The spliced ​​image features, represents the query of the second layer of self-attention mechanism, represents the key of the second layer of self-attention mechanism, represents the value of the second layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions; Step 5.3: Through the multi-layer self-attention fusion mechanism, the first initial fusion feature and the second initial fusion feature Perform hybrid fusion to obtain hybrid fusion features , which is specifically achieved through the following formula: ； ； ； in, Represents the first initial fusion feature and the second initial fusion feature The spliced ​​image features, represents the query of the third layer of self-attention mechanism, represents the key of the third layer of self-attention mechanism, Represents the value of the third layer self-attention mechanism, express The parameter matrix of express The parameter matrix of express The parameter matrix of express Dimensions; Step 5.4: Mixed fusion features and reconstruct RGB image features Splice to get the final fusion feature , which is specifically achieved through the following formula: ； Step 6: Calculate the final fusion features and normal RGB images Outliers s , determine outliers s Is it greater than the preset threshold? s If the value is greater than the preset threshold, the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are modified to obtain the next training sample in the training sample set, and return to step 4. s When the value is not greater than a preset threshold, the parameters of the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module are fixed; Step 7: Obtain the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected of the industrial product to be detected, reconstruct the RGB image to be detected, the infrared image to be detected and the 3D point cloud image to be detected to obtain the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image, and then fuse the reconstructed RGB image, the reconstructed infrared image and the reconstructed 3D point cloud image based on the multi-layer self-attention fusion module to obtain the fused feature to be detected, and then calculate the outlier value between the fused feature to be detected and the RGB image to be detected. When the outlier value is greater than the preset threshold, it indicates that the industrial product to be detected has an anomaly. When the outlier value is not greater than the preset threshold, it indicates that the industrial product to be detected does not have an anomaly.

2. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 1 is characterized in that: Step 2 is specifically implemented by the following formula: ； ； ； in, represents the anomaly segmentation mask generated by the Perlin noise generator and binarized, for The inverse matrix of Represents element-wise multiplication operation; is the opacity parameter in the blending process; Represents an abnormal texture source image in an existing abnormal source image dataset.

3. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 1 is characterized in that: The preprocessing in steps 4.1, 4.2 and 4.3 specifically includes: ； in, t = r , i , p , represents the first abnormal image after division and flattening, Indicates the N Anomaly images after segmentation and flattening; Indicates the number of divisions; is a learnable projection matrix; Encode for position; is the preprocessed image.

4. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 1 is characterized in that: The first reconstruction network, the second reconstruction network and the third reconstruction network are all encoder-decoder structures, the encoder includes L layers of multi-head self-attention and feedforward network, and the decoder includes L layers of multi-head self-attention, cross attention and feedforward network.

5. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 4 is characterized in that: In each layer of the encoder, the input image is processed by multi-head self-attention and feedforward network, which is specifically achieved by the following formula: ； ； ； ； ； ； in, Indicates the encoder The output of the multi-head self-attention layer; Indicates the The output of the multi-head self-attention mechanism of the layer; Indicates the i Head self-attention mechanism, represents the total number of attention heads, Represents a splicing operation, is the output projection matrix, Indicates the i Query of attention heads, Indicates the i The key of an attention head, Indicates the i The value of an attention head; represents the activation function; Indicates the i The query matrix of the attention heads, Indicates the i The key matrix of the attention heads, Indicates the i The value matrix of the attention head; Represents the dimension of the key in the multi-head self-attention mechanism; Represents the result of multi-head self-attention output after layer normalization; represents the output of the feedforward neural network, represents the first layer weight matrix; represents the second layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; Representation layer normalization function; Indicates the encoder l The output of the layer; Thus, after L layers of processing of the encoder, the output of the Lth layer of the encoder is obtained .

6. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 4 is characterized in that: In each layer of the decoder, the input image is processed by multi-head self-attention, cross-attention and feedforward network, which is implemented by the following formula: ； ； ； ； ； in, Indicates the l The output of the multi-head self-attention layer is the result of layer normalization. Indicates the decoder The output of the layer, Indicates the decoder The output of the multi-head self-attention mechanism, It represents the result of the cross attention layer output after layer normalization, represents the output of the cross-self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross attention mechanism, represents the query matrix, represents the matrix of bonds, A matrix representing the values, Represents the result of the feedforward network output after layer normalization, represents the dimension of the key in the cross-attention mechanism, Represents the input of the decoder, that is, the output of the encoder ; Thus, after the decoder L The decoder gets the L Output of the layer , the decoder L Output of the layer as reconstructed image features.

7. The trimodal unsupervised industrial anomaly detection method based on reconstruction network according to claim 1 is characterized in that: Calculate the final fusion features in step 6 and normal RGB images Outliers s , which is specifically achieved through the following formula: ； in, represents the L2 norm.

Images