Three-mode unsupervised industrial anomaly detection method based on reconstruction network

Abstract

The invention provides a three-mode unsupervised industrial anomaly detection method based on a reconstruction network, and relates to the technical field of industrial anomaly detection.The method comprises the steps that firstly, synthesized abnormal images of three different modes are recovered into normal images, so that the reconstruction network has the capacity of capturing and recovering the abnormal images; therefore, the method provided by the invention does not need to depend on an external memory module to store feature information in an operation process, and the deployment efficiency and response speed of the model in a resource-constrained environment are remarkably improved. Besides, the reconstruction network based on Transform can effectively extract the fine-grained features of the image, and meanwhile, texture details in the RGB image, heat distribution features in the infrared image and geometric shape information in the 3D point cloud image are fused by fully utilizing a multi-layer self-attention mechanism, so that the fine-grained recognition capability of anomaly detection is improved, and the accuracy of anomaly detection is improved. And the method also shows relatively strong robustness and generalization ability in a complex industrial environment.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial anomaly detection, and particularly to a three-modal unsupervised industrial anomaly detection method based on a reconstruction network. Background Art

[0002] Unsupervised industrial anomaly detection is a key task in intelligent industrial production. It aims to automatically identify anomalies such as product defects, production failures, and operation deviations in industrial scenarios by only using normal data or a small amount of unlabeled mixed data in the case of scarce artificially labeled anomaly samples, so as to achieve efficient and accurate automated production.

[0003] In traditional industrial anomaly detection, anomaly detection mainly relies on artificially defined standards or on industrial assembly line quality inspectors to visually check whether there are anomalies in products. However, with the increasing complexity and precision requirements of production, its limitations are becoming increasingly prominent. For the artificially formulated standards, although they have high uniformity in product detection on the industrial assembly line, due to the limited and lagging experience of engineers, such methods have obvious deficiencies in the adaptability to complex working conditions and the generalization ability to new industries. In addition, this anomaly detection method is difficult to effectively extract high-dimensional data features of local subtle anomalies on the surface texture of products, which limits the accuracy and reliability of detection. For manual visual inspection, although three-dimensional solid shapes and color features are used simultaneously to determine whether a product is defective, the physical condition of manual inspectors is easily affected by working hours and intensity, and it is difficult to meet the real-time and high-precision requirements of industrial assembly line anomaly detection.

[0004] In recent years, the rapid development of deep learning, multi-modal learning, and computer vision technologies has brought new possibilities to unsupervised industrial anomaly detection, which can effectively promote the automation and intelligent development of anomaly detection. Deep learning technology can, through powerful neural networks such as convolutional neural networks and Transformers, based on the normal features of products on the industrial assembly line, effectively identify and locate the anomaly features of industrial products with its powerful learning ability, thereby reducing redundant manual operation costs and significantly improving the accuracy and speed of detection. Especially in combination with multi-modal data such as RGB images, infrared images, and 3D point cloud images, the efficiency and accuracy of industrial unsupervised anomaly detection are further improved.

[0005] Although unsupervised industrial anomaly detection plays an important role in intelligent manufacturing, there are still many defects in its technical development in terms of data quality, algorithm performance, etc. These defects not only make it difficult to deploy industrial anomaly detection models on industrial equipment, but also seriously restrict the actual industrial application effect.

[0006] First, at the data level, a single modality can only capture partial abnormal features of pipeline products and cannot comprehensively represent diverse faults such as surface defects and structural abnormalities that may coexist in industrial scenarios. The limitations of this single-source information, combined with the severe data imbalance problem caused by the scarcity of abnormal samples, make it impossible for the model to establish a complete abnormal feature system and difficult to cover multiple abnormal patterns in industrial scenarios. Second, at the algorithm level, due to the simple feature concatenation for processing multi-modal information, it is weak in multi-modal collaborative analysis and fails to fully explore the correlations between data. In application scenarios, facing the complex and changing industrial environment, existing unsupervised industrial anomaly detection methods often show problems of insufficient adaptability.

[0007] Therefore, researchers have proposed an anomaly detection method based on a memory bank. This method uses a convolutional neural network to extract and fuse the features of normal samples in the training set and stores these features in a memory bank. In the testing phase, the features of the test images extracted are compared with the features in the memory bank. If the difference is within the set threshold, the sample is judged as a normal sample; otherwise, it is an abnormal sample. Although such methods improve the detection robustness to a certain extent, they bring significant memory overhead and inference latency. Deploying such methods not only faces the consumption of a large amount of hardware resources and deployment costs but also causes a significant drop in the model inference speed, making it difficult to meet the strict requirements of industrial production lines for real-time and high efficiency, thus limiting its implementation in actual scenarios. Summary of the Invention

[0008] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to propose a three-modal unsupervised industrial anomaly detection method based on a reconstruction network, including: Step 1: Obtain target products that meet the preset production standards in the industrial production process, and then obtain the RGB images of the target products, and use the RGB images of the target products as normal RGB images , obtain the infrared images of the target products, and use the infrared images of the target products as normal infrared images , obtain the 3D point cloud images of the target products, and use the 3D point cloud images of the target products as normal 3D point cloud images ; Step 2: Process the normal RGB images, the normal infrared images, and the normal 3D point cloud images to obtain abnormal RGB images , abnormal infrared images and abnormal 3D point cloud images ; Step 3: Use the abnormal RGB images , abnormal infrared images and abnormal 3D point cloud images as input samples, and use the normal RGB images Taking the output sample, the input sample and the output sample form a training sample, and multiple training samples form a training sample set; Step 4: Reconstruct the abnormal RGB image , the abnormal infrared image and the abnormal 3D point cloud image to obtain the reconstructed RGB image feature , the reconstructed infrared image feature and the reconstructed 3D point cloud image feature ; Step 4.1: For the input sample in the training sample in the training sample set, preprocess the abnormal RGB image in the input sample to obtain the preprocessed RGB image, and input the preprocessed RGB image into the first reconstruction network to obtain the reconstructed RGB image feature ; Step 4.2: Preprocess the abnormal infrared image in the input sample to obtain the preprocessed infrared image, and input the preprocessed infrared image into the second reconstruction network to obtain the reconstructed infrared image feature ; Step 4.3: Preprocess the abnormal 3D point cloud image in the input sample to obtain the preprocessed 3D point cloud image, and input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image feature ; Step 5: Through the multi-layer self-attention fusion module, fuse the reconstructed RGB image feature , the reconstructed infrared image feature and the reconstructed 3D point cloud image feature to obtain the final fusion feature ; Step 6: Calculate the outlier value of the final fusion feature and the normal RGB image s , and judge whether the outlier value s is greater than the preset threshold. If the outlier value s is greater than the preset threshold, modify the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module, obtain the next training sample in the training sample set, and return to execute Step 4. If the outlier value s is not greater than the preset threshold, fix the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module; Step 7: Obtain the RGB image to be detected, the infrared image to be detected, and the 3D point cloud image to be detected of the industrial product to be detected. Reconstruct the RGB image to be detected, the infrared image to be detected, and the 3D point cloud image to be detected to obtain the reconstructed RGB image, the reconstructed infrared image, and the reconstructed 3D point cloud image. Furthermore, based on the multi-layer self-attention fusion module, fuse the reconstructed RGB image, the reconstructed infrared image, and the reconstructed 3D point cloud image to obtain the fusion feature to be detected. Then, calculate the outlier between the fusion feature to be detected and the RGB image to be detected. When the outlier is greater than the preset threshold, it indicates that there is an abnormality in the industrial product to be detected. When the outlier is not greater than the preset threshold, it indicates that there is no abnormality in the industrial product to be detected.

[0009] Optionally, Step 2 is specifically implemented by the following formula: ; ; ; where represents the anomaly segmentation mask generated by the Perlin noise generator and binarized, is the inverse matrix of; represents the element-wise multiplication operation; is the opacity parameter in the mixing process; represents the anomaly texture source image in the existing anomaly source image dataset.

[0010] Optionally, the preprocessing in Steps 4.1, 4.2, and 4.3 specifically includes: ; where t = r , i , p , represents the first anomaly image after division and flattening, represents the N th anomaly image after division and flattening; represents the number of divisions; is the learnable projection matrix; is the position encoding; is the preprocessed image.

[0011] Optionally, the first reconstruction network, the second reconstruction network, and the third reconstruction network are all encoder-decoder structures. The encoder contains L layers of multi-head self-attention and feed-forward networks, and the decoder contains L layers of multi-head self-attention, cross-attention, and feed-forward networks.

[0012] Optionally, in each layer of the encoder, the input image is processed through multi-head self-attention and a feed-forward network, which is specifically implemented by the following formula: ; ; ; ; ; ; where, represents the output of the multi-head self-attention in the -th layer of the encoder; represents the output of the multi-head self-attention mechanism in the -th layer; represents the i -th head self-attention mechanism, represents the total number of attention heads, represents the concatenation operation, is the output projection matrix, represents the query of the i -th attention head, represents the key of the i -th attention head, represents the value of the i -th attention head; represents the activation function; represents the query matrix of the i -th attention head, represents the key matrix of the i -th attention head, represents the value matrix of the i -th attention head; represents the dimension of the key in the multi-head self-attention mechanism; represents the result after layer normalization of the output of the multi-head self-attention; represents the output of the feed-forward neural network, represents the first-layer weight matrix; represents the second-layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; represents the layer normalization function; represents the output of the l -th layer of the encoder; Thus, after processing through the L layers of the encoder, the output of the L-th layer of the encoder is obtained.

[0013] Optionally, in each layer of the decoder, the input image is processed by multi-head self-attention, cross-attention, and a feed-forward network, which is specifically implemented through the following formulas: ; ; ; ; ; where, represents the result after layer normalization of the output of the multi-head self-attention in the l th layer, represents the output of the th layer of the decoder, represents the output of the multi-head self-attention mechanism in the th layer in the decoder, represents the result after layer normalization of the output of the cross-attention layer, represents the output of the cross self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross-attention mechanism, represents the matrix of the query, represents the matrix of the key, represents the matrix of the value, represents the result after layer normalization of the output of the feed-forward network, represents the dimension of the key in the cross-attention mechanism, represents the input of the decoder, i.e., the output of the encoder ; Thus, after processing by the L th layer of the decoder, the output L of the th layer of the decoder is obtained. The output L of the th layer of the decoder is used as the reconstructed image feature.

[0014] Optionally, step 5 specifically includes: Step 5.1: Through a multi-layer self-attention fusion mechanism, fuse the reconstructed RGB image feature and the reconstructed infrared image feature to obtain the first initial fusion feature , which is specifically implemented through the following formulas: ; ; ; Among them, represents the reconstructed RGB image feature and the reconstructed infrared image feature of the spliced image feature; represents the query of the first-layer self-attention mechanism, represents the key of the first-layer self-attention mechanism, represents the value of the first-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.2: Through the multi-layer self-attention fusion mechanism, for the reconstructed RGB image feature and the reconstructed 3D point cloud image feature are fused to obtain the second initial fusion feature , which is specifically implemented through the following formula: ; ; ; Among them, represents the reconstructed RGB image feature and the reconstructed 3D point cloud image feature of the spliced image feature, represents the query of the second-layer self-attention mechanism, represents the key of the second-layer self-attention mechanism, represents the value of the second-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.3: Through the multi-layer self-attention fusion mechanism, the first initial fusion feature and the second initial fusion feature are mixed and fused to obtain the mixed fusion feature , which is specifically implemented through the following formula: ; ; ; Among them, represents the concatenated image feature of the first initial fusion feature and the second initial fusion feature ; represents the query of the third-layer self-attention mechanism, represents the key of the third-layer self-attention mechanism, represents the value of the third-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.4: Concatenate the hybrid fusion feature and the reconstructed RGB image feature to obtain the final fusion feature , which is specifically implemented by the following formula: .

[0015] Optionally, in step 6, calculate the outlier of the final fusion feature and the normal RGB image s , which is specifically implemented by the following formula: ; Among them, represents the L2 norm.

[0016] The beneficial effects of adopting the above technical solutions are as follows: The present invention first takes the reconstruction network as the core framework of three-modal unsupervised industrial anomaly detection. By restoring the synthesized abnormal images of three different modalities into normal images respectively, the reconstruction network is enabled to capture and restore abnormal images. Therefore, the method proposed by the present invention does not need to rely on an external memory module to store feature information during operation, effectively avoiding the large memory consumption and performance bottleneck in the inference process brought by the method based on the memory bank, and significantly improving the deployment efficiency and response speed of the model in resource-constrained environments. In addition, the reconstruction network based on Transformer can effectively extract the fine-grained features of images. At the same time, the present invention makes full use of the multi-layer self-attention mechanism to fuse the texture details in the RGB image, the thermal distribution features in the infrared image, and the geometric shape information in the 3D point cloud image, not only improving the fine-grained recognition ability of anomaly detection, but also showing strong robustness and generalization ability in complex industrial environments. Description of the Drawings

[0017] Figure 1 It is a schematic flowchart of a three-modal unsupervised industrial anomaly detection method based on a reconstruction network in an embodiment of the present invention; Figure 2 It is an overall framework diagram of a three-modal unsupervised industrial anomaly detection method based on a reconstruction network in an embodiment of the present invention. Specific embodiments

[0018] The following combines the accompanying drawings and embodiments to further describe in detail the specific embodiments of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0019] Aiming at the problems existing in the prior art, the present invention aims to develop a three-modal unsupervised industrial anomaly detection method based on a reconstruction network. First, the present invention makes full use of the complementarity of information among RGB images, infrared images, and 3D point cloud images to significantly improve the accuracy and robustness of industrial anomaly detection and make up for the limitations of single data. In addition, the present invention adopts a method based on a reconstruction network and a multi-layer attention fusion method, which significantly reduces the memory storage space of the model and greatly improves the performance of the anomaly detection model.

[0020] Specifically, the present invention uses an anomaly generator to synthesize normal RGB images, normal infrared light images, and normal 3D point cloud images in an unsupervised industrial anomaly detection dataset into abnormal RGB images, abnormal infrared light images, and abnormal 3D point cloud images. Then, the present invention trains a reconstruction network composed of Transformers to reconstruct abnormal RGB images, abnormal infrared light images, and abnormal 3D point cloud images into reconstructed RGB images, reconstructed infrared light images, and reconstructed 3D point cloud images, aiming to enable the Transformer-based reconstruction network to have the ability to extract three-modal abnormal image features and restore them to normal image features. Next, the present invention uses a multi-layer self-attention fusion mechanism to fuse the three-modal reconstructed features extracted by the Transformer-based reconstruction network pairwise, that is, uses a multi-layer self-attention fusion mechanism to fuse the reconstructed RGB image features with the reconstructed infrared light image features and the reconstructed 3D point cloud image features respectively to obtain two initial fusion features. Then, the two initial fusion features are fused using a multi-layer self-attention fusion mechanism to obtain a hybrid fusion feature, which is concatenated with the reconstructed RGB image features to obtain a final fusion feature. Finally, the final fusion feature and the normal RGB image are sent to an anomaly discriminator for anomaly discrimination and localization of the image.

[0021] Specifically, the present invention provides a three-modal unsupervised industrial anomaly detection method based on a reconstruction network, combined with Figure 1 and Figure 2 , which may include the following steps: Step 1: Obtain the target product that meets the preset production standards during the industrial production process, and then obtain the RGB image of the target product. Use the RGB image of the target product as the normal RGB image , obtain the infrared image of the target product, and use the infrared image of the target product as the normal infrared image , obtain the 3D point cloud image of the target product, and use the 3D point cloud image of the target product as the normal 3D point cloud image ; Among them, the preset production standards can be product standards on the industrial assembly line. If the product has defects such as damage, stains, scratches, deformation, roll marks, pits, roughness, ripples, color difference, etc., then the product does not meet the preset production standards; the RGB image, infrared image, and 3D point cloud image corresponding to the target product that meets the preset production standards are the normal RGB image , normal infrared image, and normal 3D point cloud image .

[0022] Among them, the RGB image can be obtained by shooting with a camera, the infrared image can be obtained by shooting with an infrared sensor, and the 3D point cloud image can be obtained by laser scanning, structured light, or other three-dimensional imaging technologies.

[0023] Step 2: Process the normal RGB image, the normal infrared image, and the normal 3D point cloud image to obtain an abnormal RGB image , abnormal infrared image and abnormal 3D point cloud image , which is specifically implemented through the following formula: ; ; ; Among them, represents the abnormal segmentation mask generated and binarized by the Perlin noise generator, is the inverse matrix of ; represents the element-wise multiplication operation; is the opacity parameter in the mixing process; represents the abnormal texture source image in the existing abnormal source image dataset.

[0024] Among them, the existing abnormal source image dataset is an image library specifically used to generate and train the abnormal detection model. A large number of abnormal samples included in it may come from historical defect records, experimental data, artificially synthesized defect images, or are obtained by simulating different fault situations in industrial production.

[0025] Among them, step 2 can be understood as processing a normal image through an anomaly generator to obtain an abnormal image.

[0026] Step 3: Use the abnormal RGB image , the abnormal infrared image and the abnormal 3D point cloud image as input samples, and use the normal RGB image as the output sample. The input samples and the output sample constitute a training sample, and multiple training samples constitute a training sample set; Step 4: Reconstruct the abnormal RGB image , the abnormal infrared image and the abnormal 3D point cloud image to obtain the reconstructed RGB image feature , the reconstructed infrared image feature and the reconstructed 3D point cloud image feature ; Step 4.1: For the input samples in the training samples in the training sample set, preprocess the abnormal RGB image in the input samples to obtain a preprocessed RGB image, and input the preprocessed RGB image into the first reconstruction network to obtain the reconstructed RGB image feature ; Step 4.2: Preprocess the abnormal infrared image in the input samples to obtain a preprocessed infrared image, and input the preprocessed infrared image into the second reconstruction network to obtain the reconstructed infrared image feature ; Step 4.3: Preprocess the abnormal 3D point cloud image in the input samples to obtain a preprocessed 3D point cloud image, and input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image feature ; Among them, the preprocessing in steps 4.1, 4.2, and 4.3 specifically includes: ; Among them, t = r , i , p , represents the 1st abnormal image after division and flattening, represents the N th abnormal image after division and flattening; represents the number of divisions; is a learnable projection matrix; is the positional encoding; is the preprocessed image.

[0027] That is to say, in step 4.1 t = r , at this time, and are represented as and , that is, the first abnormal RGB image after partitioning and flattening and the Nth abnormal RGB image after partitioning and flattening. Furthermore, is represented as the preprocessed RGB image. Similarly, in step 4.2 t = i , at this time, and are represented as and , that is, the first abnormal infrared image after partitioning and flattening and the Nth abnormal infrared image after partitioning and flattening. Furthermore, is represented as the preprocessed infrared image; in step 4.3 t = p , at this time, and are represented as and , that is, the first abnormal 3D point cloud image after partitioning and flattening and the Nth abnormal 3D point cloud image after partitioning and flattening. Furthermore, is represented as the preprocessed 3D point cloud image.

[0028] Among them, the first reconstruction network, the second reconstruction network, and the third reconstruction network are all encoder-decoder structures. The encoder contains L layers of multi-head self-attention and feed-forward networks, and the decoder contains L layers of multi-head self-attention, cross-attention, and feed-forward networks.

[0029] It should be noted that the first reconstruction network, the second reconstruction network, and the third reconstruction network are all Transformer-based reconstruction networks. The architectures of the first reconstruction network, the second reconstruction network, and the third reconstruction network are the same, but the parameters of the encoder and decoder are different.

[0030] Among them, in each layer of the encoder, the input image is processed through multi-head self-attention and feed-forward networks, which is specifically implemented by the following formula: ; ; ; ; ; ; Among them, represents the output of the multi-head self-attention in the th layer of the encoder; represents the output of the multi-head self-attention mechanism in the th layer; represents the i th head self-attention mechanism, h represents the total number of attention heads, represents the concatenation operation, is the output projection matrix, represents the query of the i th attention head, represents the key of the i th attention head, represents the value of the i th attention head; represents the activation function; represents the query matrix of the i th attention head, represents the key matrix of the i th attention head, represents the value matrix of the i th attention head; represents the dimension of the key in the multi-head self-attention mechanism; represents the result after layer normalization of the output of the multi-head self-attention; represents the output of the feed-forward neural network, represents the first-layer weight matrix; represents the second-layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; represents the layer normalization function; represents the output of the l th layer of the encoder; Thus, after processing through the L layers of the encoder, the output of the Lth layer of the encoder is obtained .

[0031] Among them, is the input of the first layer of the encoder. After processing through L layers, the output of the Lth layer of the encoder is obtained , and the output of the Lth layer of the encoder is used as the input of the first layer of the decoder, that is, .

[0032] Among them, in each layer of the decoder, the input image is processed through multi-head self-attention, cross-attention, and feed-forward network, and is specifically implemented through the following formula: ; ; ; ; ; wherein, represents the result after layer normalization of the output of the l -th layer of multi-head self-attention, represents the output of the -th layer of the decoder, represents the output of the multi-head self-attention mechanism of the -th layer in the decoder, represents the result after layer normalization of the output of the cross-attention layer, represents the output of the cross self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross-attention mechanism, represents the matrix of the query, represents the matrix of the key, represents the matrix of the value, represents the result after layer normalization of the output of the feed-forward network, represents the dimension of the key in the cross-attention mechanism, represents the input of the decoder, i.e., the output of the encoder ; Thus, after being processed by the L -th layer of the decoder, the output of the L -th layer of the decoder is obtained. The output L of the -th layer of the decoder is used as the reconstructed image feature.

[0033] It can be obtained therefrom that in the present invention, the preprocessed RGB image is input into the encoder-decoder of the first reconstruction network, and the output L of the -th layer of the decoder can be obtained, that is, the reconstructed RGB image feature . At the same time, the preprocessed infrared image is input into the encoder-decoder of the second reconstruction network, and the output L of the -th layer of the decoder can be obtained, that is, the reconstructed infrared image feature . At the same time, the preprocessed 3D point cloud image is input into the encoder-decoder of the third reconstruction network, and the output L of the -th layer of the decoder can be obtained, that is, the reconstructed 3D point cloud image feature 。

[0034] Step 5: Through the multi-layer self-attention fusion module, fuse the reconstructed RGB image features , the reconstructed infrared image features and the reconstructed 3D point cloud image features to obtain the final fusion features ; Step 5.1: Through the multi-layer self-attention fusion mechanism, fuse the reconstructed RGB image features and the reconstructed infrared image features to obtain the first initial fusion features , which is specifically implemented through the following formula: ; ; ; where, represents the concatenated image features of the reconstructed RGB image features and the reconstructed infrared image features ; represents the query of the first-layer self-attention mechanism, represents the key of the first-layer self-attention mechanism, represents the value of the first-layer self-attention mechanism, represents 's parameter matrix, represents 's parameter matrix, represents 's parameter matrix, represents 's dimension; Step 5.2: Through the multi-layer self-attention fusion mechanism, fuse the reconstructed RGB image features and the reconstructed 3D point cloud image features to obtain the second initial fusion features , which is specifically implemented through the following formula: ; ; ; where, represents the concatenated image features of the reconstructed RGB image features and the reconstructed 3D point cloud image features , represents the query of the second-layer self-attention mechanism, represents the key of the second-layer self-attention mechanism, The value representing the second - layer self - attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.3: Through the multi - layer self - attention fusion mechanism, mix - fuse the first initial fusion feature and the second initial fusion feature to obtain the mixed fusion feature , which is specifically implemented through the following formula: ; ; ; where, represents the concatenated image feature of the first initial fusion feature and the second initial fusion feature , represents the query of the third - layer self - attention mechanism, represents the key of the third - layer self - attention mechanism, represents the value of the third - layer self - attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.4: Concatenate the mixed fusion feature and the reconstructed RGB image feature to obtain the final fusion feature , which is specifically implemented through the following formula: .

[0035] Step 6: Calculate the outlier of the final fusion feature and the normal RGB image s , which is specifically implemented through the following formula: ; where, represents the L2 norm.

[0036] Among them, the process of calculating the outlier can be understood as in the outlier discriminator, for the final fused feature and the normal RGB image to conduct discrimination to obtain the outlier s .

[0037] Judge whether the outlier s is greater than the preset threshold. When the outlier s is greater than the preset threshold, modify the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module, obtain the next training sample in the training sample set, and return to execute step 4. When the outlier s is not greater than the preset threshold, fix the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module; Among them, when modifying the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module, specifically, the query, key, value mapping weights in the multi-head self-attention mechanism in the encoder, and the linear transformation weights , and the bias parameters , can be modified. At the same time, in the decoder, in addition to the above structure, the weight matrix , , of the cross-attention mechanism for inter-modal information integration, as well as the parameters of the self-attention fusion module finally used for fusing the three-modal features are modified.

[0038] Step 7: Obtain the to-be-detected RGB image, to-be-detected infrared image and to-be-detected 3D point cloud image of the industrial product to be detected, reconstruct the to-be-detected RGB image, to-be-detected infrared image and to-be-detected 3D point cloud image to obtain the reconstructed RGB image, reconstructed infrared image and reconstructed 3D point cloud image, and then based on the multi-layer self-attention fusion module, fuse the reconstructed RGB image, reconstructed infrared image and reconstructed 3D point cloud image to obtain the to-be-detected fused feature, and then calculate the outlier between the to-be-detected fused feature and the to-be-detected RGB image. When the outlier is greater than the preset threshold, it indicates that the industrial product to be detected has an abnormality. When the outlier is not greater than the preset threshold, it indicates that the industrial product to be detected has no abnormality.

[0039] In summary, to reduce the resource overhead and computational cost pressure during model deployment and improve the speed and real-time performance of anomaly detection, the present invention adopts an efficient and lightweight reconstruction network structure. This structure focuses on reconstructing the features of abnormal images. By learning the feature distribution of normal images, it realizes the reconstruction of the input images. Thus, during the inference stage, the abnormal regions are judged through the reconstruction error, eliminating the heavy memory feature matching operation and significantly improving the inference efficiency of the model. The reconstruction network combines the advantage of global feature modeling in the Transformer architecture, enhancing the model's fine-grained feature perception ability for multi-modal information.

[0040] In addition, aiming at the bottleneck problems of existing single-modal unsupervised industrial anomaly detection methods, such as insufficient data representation ability, poor robustness, and difficulty in adapting to complex and changeable industrial scenarios, the present invention proposes a three-modal unsupervised industrial anomaly detection scheme. This scheme realizes deep multi-modal feature extraction and interaction through a multi-layer self-attention fusion mechanism, fully integrating the detailed texture information of RGB images, the thermal distribution characteristics of infrared images, and the spatial structure information of 3D point cloud images, effectively improving the model's understanding ability and generalization ability for multi-source data. Therefore, by combining the reconstruction network based on Transformer and the multi-layer self-attention fusion mechanism, while ensuring the accuracy of anomaly detection, the present invention has stronger environmental adaptability, providing an efficient, accurate and scalable solution for intelligent detection under complex working conditions in industrial production processes.

[0041] A three-modal unsupervised industrial anomaly detection method proposed by the present invention provides an efficient, stable and practical solution for resource-constrained industrial detection platforms. This method uses a reconstruction network based on Transformer to construct a lightweight three-modal anomaly detection model with low memory occupancy and fast inference ability, which can effectively meet the dual requirements of industrial production lines for real-time performance and reliability. The deep dependence relationship between modalities is modeled through the multi-head self-attention mechanism, and the reconstruction network is guided to learn the normal feature space of the product. During the testing stage, the model automatically discriminates the abnormal regions based on the reconstruction error of the input images, without relying on manual annotation or additional memory modules, greatly improving the flexibility and efficiency of model deployment. In addition, the present invention makes full use of the multi-layer self-attention mechanism to fuse the texture details in RGB images, the thermal distribution characteristics in infrared images, and the geometric shape information in 3D point cloud images, not only improving the fine-grained recognition ability of anomaly detection, but also showing strong robustness and generalization ability in complex industrial environments, ultimately achieving excellent performance of high-precision recognition, high-speed inference and low resource overhead, providing a solid technical support for intelligent quality inspection of industrial products.

[0042] The above description is only for the preferred embodiments of the present disclosure and the explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the embodiments of the present disclosure that have similar functions.

Claims

1. A three-modal unsupervised industrial anomaly detection method based on a reconstruction network, characterized in that Including: Step 1: Obtain the target product that meets the preset production standards during the industrial production process, and then obtain the RGB image of the target product. Use the RGB image of the target product as the normal RGB image , obtain the infrared image of the target product, and use the infrared image of the target product as the normal infrared image , obtain the 3D point cloud image of the target product, and use the 3D point cloud image of the target product as the normal 3D point cloud image ; Step 2: Process the normal RGB image, the normal infrared image, and the normal 3D point cloud image to obtain an abnormal RGB image , an abnormal infrared image , and an abnormal 3D point cloud image ; Step 3: Take the abnormal RGB image , the abnormal infrared image and the abnormal 3D point cloud image as input samples, and take the normal RGB image as the output sample. The input samples and the output sample form a training sample, and multiple training samples form a training sample set; Step 4: Reconstruct the abnormal RGB image , the abnormal infrared image and the abnormal 3D point cloud image to obtain the reconstructed RGB image features , the reconstructed infrared image features and the reconstructed 3D point cloud image features ; Step 4.1: For the input samples in the training samples in the training sample set, preprocess the abnormal RGB images in the input samples to obtain the preprocessed RGB images, and input the preprocessed RGB images into the first reconstruction network to obtain the reconstructed RGB image features in the first reconstruction network ; ; Step 4.2: For the abnormal infrared images in the input samples perform preprocessing to obtain the preprocessed infrared images, and input the preprocessed infrared images into the second reconstruction network to obtain the reconstructed infrared image features ; Step 4.3: Preprocess the abnormal 3D point cloud image in the input sample to obtain the preprocessed 3D point cloud image, and input the preprocessed 3D point cloud image into the third reconstruction network to obtain the reconstructed 3D point cloud image features ; Step 5: Through the multi-layer self-attention fusion module, fuse the reconstructed RGB image features , the reconstructed infrared image features and the reconstructed 3D point cloud image features to obtain the final fused features ; Step 6: Calculate the final fused feature and the normal RGB image of the outliers s , judge the outliers s whether it is greater than the preset threshold. In the case where the outlier s is greater than the preset threshold, modify the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module, obtain the next training sample in the training sample set, return to execute Step 4. In the case where the outlier s is not greater than the preset threshold, fix the parameters in the first reconstruction network, the second reconstruction network, the third reconstruction network and the multi-layer self-attention fusion module; Step 7: Obtain the RGB image to be detected, the infrared image to be detected, and the 3D point cloud image to be detected of the industrial product to be detected. Reconstruct the RGB image to be detected, the infrared image to be detected, and the 3D point cloud image to be detected to obtain the reconstructed RGB image, the reconstructed infrared image, and the reconstructed 3D point cloud image. Then, based on the multi-layer self-attention fusion module, fuse the reconstructed RGB image, the reconstructed infrared image, and the reconstructed 3D point cloud image to obtain the fusion feature to be detected. Further, calculate the outlier between the fusion feature to be detected and the RGB image to be detected. When the outlier is greater than the preset threshold, it indicates that the industrial product to be detected is abnormal. When the outlier is not greater than the preset threshold, it indicates that the industrial product to be detected is normal.

2. The method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 1, wherein Step 2 is specifically implemented by the following formula: ； ； ； Among them, represents the abnormal segmentation mask generated by the Perlin noise generator and binarized, is the inverse matrix of; represents the element-wise multiplication operation; is the opacity parameter in the mixing process; represents the abnormal texture source image in the existing abnormal source image dataset.

3. A method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 1, characterized in that, The preprocessing in Step 4.1, Step 4.2, and Step 4.3 specifically includes: ； Among them, t = r , i , p , represents the first divided and flattened abnormal image, represents the N th divided and flattened abnormal image; represents the number of divisions; is a learnable projection matrix; is the positional encoding; is the preprocessed image.

4. A method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 1, characterized in that, The first reconstruction network, the second reconstruction network, and the third reconstruction network are all encoder-decoder structures. The encoder contains L layers of multi-head self-attention and feed-forward networks, and the decoder contains L layers of multi-head self-attention, cross-attention, and feed-forward networks.

5. The method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 4, wherein In each layer of the encoder, the input image is processed through multi-head self-attention and feed-forward networks, which is specifically implemented by the following formula: ； ； ； ； ； ； Among them, represents the output of the multi-head self-attention in the th layer of the encoder; represents the output of the multi-head self-attention mechanism in the th layer; represents the i th head self-attention mechanism, represents the total number of attention heads, represents the concatenation operation, is the output projection matrix, represents the query of the i th attention head, represents the key of the i th attention head, represents the value of the i th attention head; represents the activation function; represents the query matrix of the i th attention head, represents the key matrix of the i th attention head, represents the value matrix of the i th attention head; represents the dimension of the key in the multi-head self-attention mechanism; represents the result after layer normalization of the output of the multi-head self-attention; represents the output of the feed-forward neural network, represents the first layer weight matrix; represents the second layer weight matrix; represents the bias term of the first layer; represents the bias term of the second layer; represents the layer normalization function; represents the output of the l th layer of the encoder; Thus, after being processed by the L layers of the encoder, the output of the L-th layer of the encoder is obtained .

6. The method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 4, wherein In each layer of the decoder, the input image is processed through multi-head self-attention, cross-attention, and feed-forward networks, which is specifically implemented by the following formula: ； ； ； ； ； Among them, represents the result after layer normalization of the output of the l -layer multi-head self-attention, represents the output of the -th layer of the decoder, represents the output of the multi-head self-attention mechanism in the -th layer of the decoder, represents the result after layer normalization of the output of the cross-attention layer, represents the output of the cross self-attention mechanism, represents the query of the cross-attention mechanism, represents the key of the cross-attention mechanism, represents the value of the cross-attention mechanism, represents the matrix of the query, represents the matrix of the key, represents the matrix of the value, represents the result after layer normalization of the output of the feed-forward network, represents the dimension of the key in the cross-attention mechanism, represents the input of the decoder, that is, the output of the encoder ; Thus, after passing through the decoder L layer processing, the output of the L layer of the decoder is obtained . The output of the L layer of the decoder is used as the reconstructed image feature.

7. A method for tri-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Through a multi-layer self-attention fusion mechanism, fuse the reconstructed RGB image features and the reconstructed infrared image features to obtain the first initial fusion feature , which is specifically implemented through the following formula: ； ； ； Among them, represents the reconstructed RGB image feature and the reconstructed infrared image feature of the spliced image feature; represents the query of the first-layer self-attention mechanism, represents the key of the first-layer self-attention mechanism, represents the value of the first-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.2: Through a multi-layer self-attention fusion mechanism, fuse the reconstructed RGB image features and the reconstructed 3D point cloud image features to obtain the second initial fusion feature , which is specifically implemented through the following formula: ； ； ； Among them, represents the reconstructed RGB image feature and the reconstructed 3D point cloud image feature of the spliced image feature. represents the query of the second-layer self-attention mechanism, represents the key of the second-layer self-attention mechanism, represents the value of the second-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.3: Through a multi-layer self-attention fusion mechanism, the first initial fusion feature and the second initial fusion feature are mixed and fused to obtain a mixed fusion feature , which is specifically implemented through the following formula: ； ； ； Among them, represents the concatenated image feature of the first initial fusion feature and the second initial fusion feature ; represents the query of the third-layer self-attention mechanism, represents the key of the third-layer self-attention mechanism, represents the value of the third-layer self-attention mechanism, represents the parameter matrix of represents the parameter matrix of represents the parameter matrix of represents the dimension of Step 5.4: For the mixed fusion feature and the reconstructed RGB image feature perform splicing to obtain the final fusion feature , which is specifically implemented through the following formula: 。 8. A method for three-modal unsupervised industrial anomaly detection based on a reconstruction network according to claim 1, characterized in that, Calculating the final fused feature in step 6 and the normal RGB image of outliers s , which is specifically implemented by the following formula: ； Among them, represents the L2 norm.

Images