A noise suppression modality fusion alignment method for industrial image anomaly detection

Abstract

The present invention discloses a method for detecting anomalies in industrial images using noise-suppressed modal fusion alignment, which belongs to the field of image detection technology. The method comprises: obtaining normal industrial image samples and abnormal industrial image samples with labels; constructing an industrial image anomaly detection model, including a text encoder, a dual-channel image encoder, an MLP module, and a multimodal interaction module; training the industrial image anomaly detection model using a training set; and performing anomaly detection on image samples in a test set using the trained industrial image anomaly detection model. The present invention uses a noise-suppressed attention mechanism in the text encoder to enable the model to pay more attention to the key semantics in the prompt words and reduce the interference of irrelevant semantics; by introducing position information for the abnormal prompt, the amount of information brought by the text modality is increased, helping the model to better understand the specific area where the anomaly occurs and enhancing the spatial perception ability of the model.

Description

Technical Field

[0001] The present invention relates to an industrial image anomaly detection method, in particular to an industrial image anomaly detection method based on noise suppression modal fusion alignment, belonging to the technical field of image detection. Background Art

[0002] Image anomaly detection is a key task in computer vision, aiming to identify unusual regions or objects in images. In the industrial sector, it is primarily used for product quality control and defect detection, such as detecting damaged electronic components. Its accuracy is crucial: high-precision detection prevents defective products from entering the market, reducing losses and safeguarding corporate reputation. It also reduces manual intervention, improving production efficiency and competitiveness. With the digital transformation of industry, data security and privacy protection are becoming increasingly important. Zero-shot learning methods utilize pre-trained models and a small amount of annotated data to reduce annotation costs and the risk of data leakage, thereby promoting intelligent, efficient, and data-secure industrial production.

[0003] In recent years, multimodal pre-training technology based on contrastive learning has achieved important breakthroughs in the field of computer vision. By constructing a joint semantic space of language and vision, this method enables the model to understand image content directly through text semantics, demonstrating remarkable zero-shot reasoning capabilities. Its core innovation lies in breaking through the limitations of traditional unimodal learning and mapping image features and text descriptions to a unified representation space through a cross-modal alignment mechanism. This technical approach effectively alleviates the dependence on massive amounts of labeled data and demonstrates excellent domain adaptability in tasks such as image classification and object recognition. It is particularly suitable for industrial quality inspection scenarios where abnormal samples are scarce and labeling costs are high. At the same time, the characteristics of multimodal interaction replace raw data processing with semantic-level feature extraction, providing a new paradigm for building privacy-friendly detection systems.

[0004] Although current anomaly detection methods based on this paradigm have made progress in text-guided mechanisms and visual feature modeling, they still face multiple technical challenges: First, text semantic guidance modules generally lack spatial positioning capabilities, and existing prompt template designs cannot effectively integrate location prior knowledge. Because the original framework focuses on global semantic matching, local area features and text descriptions cannot be fine-grainedly aligned. Secondly, there is an information bottleneck in the cross-modal interaction mechanism, the text encoding process is easily interfered by redundant vocabulary, and the attention allocation mechanism fails to fully focus on key semantic units. Furthermore, complex background interference in industrial scenes (such as metal reflections, surface textures, etc.) poses a severe challenge to visual feature extraction. Current methods are still imperfect in noise suppression and feature decoupling, which can easily lead to detection results being interfered with by pseudo-anomalies.

[0005] In summary, existing anomaly detection algorithms have the following defects:

[0006] (1) General text prompts lack a specific description of the defect location and cannot fully utilize the text semantic information.

[0007] (2) There are deficiencies in the information interaction between text and image modalities, and the potential of multimodal data has not been fully utilized. The semantic alignment between text embedding and image embedding is not sufficient, which limits the model's ability to understand multimodal semantics, thereby affecting the accuracy of detection and generalization performance.

[0008] (3) Both image and text modalities are affected by noise. Background noise in industrial images, such as light changes and texture interference, can cause the model to over-focus, leading to false detections and missed detections. Text encoders tend to overweight irrelevant semantics, distracting the model and affecting anomaly detection accuracy. Summary of the Invention

[0009] Purpose of the invention: In response to the above problems, the purpose of the present invention is to provide an industrial image anomaly detection method based on noise suppression modal fusion alignment, which introduces position information through text prompt templates and makes full use of text semantic information; uses a noise suppression attention mechanism in the text encoder and image encoder, introduces a dual-channel image encoder, suppresses irrelevant noise, and pays more attention to key semantic information and target object features.

[0010] Technical solution: The present invention provides an industrial image anomaly detection method based on noise suppression modal fusion alignment, comprising the following steps:

[0011] Obtain normal industrial image samples and labeled abnormal industrial image samples, build a dataset, and divide it into training set and test set according to the ratio;

[0012] Construct an industrial image anomaly detection model; wherein the industrial image anomaly detection model includes a text encoder, a dual-channel image encoder, an MLP module, and a multimodal interaction module;

[0013] Use the training set to train the industrial image anomaly detection model;

[0014] The trained industrial image anomaly detection model is used to perform anomaly detection on image samples in the test set.

[0015] Furthermore, the steps of training the industrial image anomaly detection model using the training set include:

[0016] Constructing structured normal text prompt templates and structured abnormal text prompt templates;

[0017] The normal text prompt template and five sets of abnormal text prompt templates with position information are input into the text encoder for encoding. The normal text prompt template is encoded through the noise suppression attention layer and then the normal text embedding vector is output. , five groups of abnormal text prompt templates with position information are encoded into abnormal text embedding vector groups through the noise suppression attention layer, and the five groups of abnormal text embedding vectors are averaged and pooled to generate a fused abnormal text embedding vector ; Concatenate the normal text embedding vector and the fused abnormal text embedding vector to obtain the text prompt embedding vector , , , C is the embedding dimension, R represents the real matrix space;

[0018] The images to be tested in the training set are resized to a fixed resolution and then fed into a two-channel image encoder to obtain a global image embedding vector, a four-layer patch-level embedding vector, and a four-layer denoising patch-level embedding vector.

[0019] Project the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules;

[0020] The projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector are interacted and aligned with the text prompt embedding vector respectively through the multimodal interaction module;

[0021] The cosine similarity of the interactively aligned four-layer patch-level embedding vector, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text prompt embedding vector is calculated respectively to generate anomaly maps. The loss of the obtained anomaly maps is calculated compared with the real anomaly maps, and the detection capability of the industrial image anomaly detection model is trained through loss back propagation.

[0022] Furthermore, the text encoder consists of 12 noise suppression attention layers.

[0023] Furthermore, in the noise suppression attention layer, the input is multiplied by the weight matrix , the query matrix generated and bond matrix , the generated matrix is divided into two parts, recorded as , N represents the sequence length;

[0024] Calculate the first attention weight matrix and the second attention weight matrix , where scale is the scaling factor, is the activation function;

[0025] The denoised attention output is generated by performing a difference operation between the first attention weight matrix and the second attention weight matrix, which is expressed as:

[0026] ,

[0027] Where, is the learnable weight coefficient, is a hyperparameter, Multiply the text template by the weight matrix The resulting matrix of values.

[0028] Furthermore, the step of adjusting the image to be detected in the training set to a fixed resolution and then inputting it into the dual-channel image encoder includes:

[0029] The image to be detected is input into the first channel CLIP image encoder, and the image features are extracted by the QKV attention mechanism from the 1st to the 5th layer, and the image features are extracted by the VV attention mechanism from the 6th to the 24th layer, and the global image embedding vector is output. and the patch-level embedding vectors at layers 6, 12, 18, and 24 , where i=1,2,3,4 correspond to the [6th, 12th, 18th, 24th] layers respectively;

[0030] The image to be detected is input into the second channel CLIP image encoder, and through the noise suppression attention layer, the denoised patch-level embedding vectors of the 6th, 12th, 18th, and 24th layers are output. .

[0031] Furthermore, the step of projecting the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules includes:

[0032] Patch-level embedding vectors and the denoised patch set embedding vector They are sent to their respective multi-layer perceptrons for mapping operations, and the embedded vectors are aligned. The mapped patch-level embedded vectors are recorded as ;

[0033] The multilayer perceptron consists of an input layer, an intermediate layer and an output layer.

[0034] Furthermore, the steps of interacting and aligning the projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector with the text prompt embedding vector respectively through the multimodal interaction module include:

[0035] Embedding text hints into vectors Through a learnable weight matrix Projected to the Key space, the matrix is obtained , and then embed the text prompt into the vector Through a learnable weight matrix Projected to the Value space, we get the matrix ;

[0036] The patch-level embedding vectors of the preliminary alignment Through their respective learnable weight matrices Projected to the Query space, we get ;

[0037] Calculate the attention weights separately, the formula is:

[0038] , ,in ;

[0039] The attention weight Scaling and normalization are performed, the formula is:

[0040] ,in, , yes Dimensions;

[0041] The attention weight Multiplying with the matrix V and passing through the linear projection layer, we get the mixed four-layer patch-level embedding vector, which is:

[0042] ,

[0043] ,

[0044] ,

[0045] ,

[0046] Embedding patch-level vectors and the mixed patch-level embedding vector The corresponding elements are added and normalized to obtain the final aligned patch-level embedding vector .

[0047] Furthermore, the cosine similarity between the interactively aligned four-layer patch-level embedding vector, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text hint embedding vector is calculated respectively. The steps of generating an anomaly map include:

[0048] Embedding patch-level vectors respectively with the text hint embedding vector Calculate the cosine similarity, the calculation formula is:

[0049] ,

[0050] in, Represents the number of patches at each layer. At each layer, the image is divided into 1370 patches, and each patch is converted into an embedding vector;

[0051] The similarity matrix obtained after calculating the four-layer patch-level embedding vector and the text hint embedding vector is: ;

[0052] The first patch records the global image information. When calculating the local similarity, the first patch is ignored. The similarity of 1369 patches is used for upsampling. The obtained similarity map is:

[0053] ,

[0054] in, , represents the similarity matrix without the first patch, represents upsampling;

[0055] Denoised patch-level embedding vector The similarity is calculated with the text hint embedding vector and upsampled, and the obtained similarity graph is recorded as ;

[0056] Embed the global image into a vector Embedding vectors with text hints Calculate the cosine similarity and get the similarity matrix between the global image and the text prompt: , the calculation formula is:

[0057] ,

[0058] Similar graphs and Each pixel in the dataset is converted into a normal and abnormal probability distribution using the softmax function, and the category with the highest probability is selected as the predicted label to obtain eight abnormal mask images, which are recorded as ,The eight anomaly mask images are weightedly fused to generate the final anomaly map.

[0059] Furthermore, the loss of the industrial image anomaly detection model includes pixel-level loss and image-level loss , the formula is:

[0060] .

[0061] Furthermore, the normal text prompt template is [a][photo][of][good][v1][v2][cls], and the abnormal text prompt template is [a][photo][of][damage][v3][v4][cls][located][at][the][position], where [cls] is the name of the item, [v1], [v2], [v3], and [v4] represent learnable text prompt words, and the abnormal prompt introduces position information. [position] represents the position prompt word and is set to [top], [center], [left], [right], or [bottom].

[0062] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0063] The present invention uses a noise suppression attention mechanism in the text encoder to make the model pay more attention to the key semantics in the prompt words and reduce the interference of irrelevant semantics; by introducing position information for abnormal prompts, the amount of information brought by the text modality is increased, which helps the model better understand the specific area where the abnormality occurs and enhances the spatial perception ability of the model; the dual-channel image encoder with the addition of a noise suppression attention channel can better retain the detailed information of the original image on the basis of effectively eliminating environmental noise, effectively reducing the false detection of irrelevant areas of the image and the missed detection of relevant areas; through the multimodal interaction module, the information interaction between text and image modalities is promoted, the information interaction and fusion between text prompt embedding and image patch embedding are promoted, and the image patch embedding is mapped to the same semantic space as the text embedding, so as to better complete the modal alignment; this alignment not only enhances the model's ability to understand multimodal semantics, but also improves the interpretability of the model, enabling it to more accurately locate and identify abnormal areas; at the same time, modal alignment also strengthens the model's generalization ability for unknown abnormality types, further improving the accuracy and robustness of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flowchart of an industrial image anomaly detection method based on noise suppression modal fusion alignment;

[0065] Figure 2 Architecture diagram for training anomaly detection models for industrial images;

[0066] Figure 3 Architecture diagram for testing industrial image anomaly detection models;

[0067] Figure 4 This is the MMI module architecture diagram;

[0068] Figure 5 This is the flow chart of the training phase;

[0069] Figure 6 This is the flow chart of the testing phase. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings and embodiments.

[0071] Combine Figure 1 As shown, the industrial image anomaly detection method using noise suppression modal fusion alignment described in this embodiment includes the following steps:

[0072] Step 1: Obtain normal industrial image samples and labeled abnormal industrial image samples, build a dataset, and divide it into training set and test set according to the ratio.

[0073] This example follows the zero-shot learning paradigm, where the test set images do not belong to the categories present in the auxiliary anomaly detection dataset. The MVTec AD and VisA datasets are used as sample sets. Both datasets are color image sets containing normal and labeled anomaly samples. When the MVTec AD dataset is used as the test set, the VisA dataset is used as the auxiliary anomaly detection dataset. When the VisA dataset is used as the test set, the MVTec AD dataset is used as the auxiliary anomaly detection dataset.

[0074] Step 2: Build an industrial image anomaly detection model; wherein, the industrial image anomaly detection model includes a text encoder, a dual-channel image encoder, an MLP module, and a multimodal interaction module.

[0075] The text encoder converts the input text prompt into a semantic embedding vector, which more accurately captures the key semantic information in the text while suppressing irrelevant semantic interference. In this example, the text encoder consists of 12 noise suppression attention (NSA) layers. The NSA layer is derived from the Transformer layer by adding a noise attention mechanism. This means that each improved Transformer layer includes a noise suppression attention mechanism and a feedforward neural network.

[0076] The dual-channel image encoder is used to extract the global features and local detail features of the input image, and remove irrelevant noise through the noise suppression mechanism, retaining image information useful for anomaly detection. The dual-channel image encoder includes a first-channel CLIP image encoder and a second-channel CLIP image encoder. The first-channel CLIP image encoder consists of 24 layers of visual Transformer layers ( Figure 2 The second channel CLIP image encoder consists of 24 layers of NSA layer.

[0077] The MLP (Multi-Layer Perceptron) module is used to map the patch-level embedding output by the image encoder to the same semantic space as the text embedding, and to enable effective alignment and interaction with the text embedding by adjusting the dimension and semantic expression of image features.

[0078] The Multimodal Interaction Module (MMI) facilitates information interaction and fusion between text and image modalities, achieving modality alignment and semantic consistency. This module enables industrial image anomaly detection models to better map image patch embeddings and text embeddings into the same semantic space.

[0079] Step 3: Use the training set to train the industrial image anomaly detection model.

[0080] Combine Figures 2 to 3 As shown in Figure 2, this example consists of two main phases: training and testing. The training phase is used to train the text prompt template, the MLP module, the MMI, and the weight coefficients of the noise suppression attention layer. The testing phase is used to perform anomaly detection and segmentation on unknown images to be detected.

[0081] During the training phase, normal and abnormal prompts are first generated using text prompt templates. Positional information is introduced into abnormal prompts to enhance the text modality, and a text encoder is used to generate accurate text embedding vectors. Next, the input image is resized to a fixed resolution, and a two-channel image encoder is used to obtain a global image embedding, multi-layer patch-level embeddings, and multi-layer denoised patch-level embeddings. The patch-level embeddings and denoised patch-level embeddings are each operated on through their own MLP modules. The patch-level embeddings are then interacted and aligned with the text embeddings through a multimodal interaction module. Finally, the cosine similarity of the interacted patch-level embeddings, denoised patch-level embeddings, and text embeddings is calculated to generate anomaly maps. The resulting anomaly maps are then compared with the ground-truth anomaly maps for backpropagation, thereby training the model's detection capabilities.

[0082] Combine Figure 5 ,Further, the steps of training the industrial image anomaly detection model using the ,training set include:

[0083] Constructing structured normal text prompt templates and structured abnormal text prompt templates;

[0084] The normal text prompt template and five sets of abnormal text prompt templates with position information are input into the text encoder for encoding. The normal text prompt template is encoded through the noise suppression attention layer and then the normal text embedding vector is output. , five groups of abnormal text prompt templates with position information are encoded into abnormal text embedding vector groups through the noise suppression attention layer, and the five groups of abnormal text embedding vectors are averaged and pooled to generate a fused abnormal text embedding vector ; Concatenate the normal text embedding vector and the fused abnormal text embedding vector to obtain the text prompt embedding vector , , , C is the embedding dimension, R represents the real matrix space;

[0085] The images to be tested in the training set are resized to a preset resolution and then input into a two-channel image encoder to obtain a global image embedding vector, a four-layer patch-level embedding vector, and a four-layer denoising patch-level embedding vector.

[0086] Project the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules;

[0087] The projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector are interacted and aligned with the text prompt embedding vector respectively through the multimodal interaction module;

[0088] The cosine similarity of the interactively aligned four-layer patch-level embedding vector, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text prompt embedding vector is calculated respectively to generate anomaly maps. The loss of the obtained anomaly maps is calculated compared with the real anomaly maps, and the detection capability of the industrial image anomaly detection model is trained through loss back propagation.

[0089] Furthermore, the normal text prompt template is [a][photo][of][good][v1][v2][cls], and the abnormal text prompt template is [a][photo][of][damage][v3][v4][cls][located][at][the][position], where [cls] is the name of the item, [v1], [v2], [v3], and [v4] represent learnable text prompt words, and the abnormal prompt introduces position information. [position] represents the position prompt word and is set to [top], [center], [left], [right], and [bottom]. [good] represents normal text, and [damage] represents abnormal text.

[0090] To make the model focus more on key semantics and suppress noise information, this example improves the attention mechanism in the text encoder and uses the noise suppression attention mechanism to generate more accurate text embedding vectors.

[0091] Furthermore, the text encoder consists of 12 NSA layers, which are obtained by adding a noise attention mechanism to the Transformer layer.

[0092] Furthermore, in the noise suppression attention layer, the input is multiplied by the weight matrix , the query matrix generated and bond matrix , the generated matrix is divided into two parts, recorded as , where N is the sequence length, such as 77, and C is the feature dimension, such as 768;

[0093] Calculate the first attention weight matrix and the second attention weight matrix , where scale is the scaling factor, is the activation function;

[0094] The denoised attention output is generated by performing a difference operation between the first attention weight matrix and the second attention weight matrix, which is expressed as:

[0095] ,

[0096] Where, is the learnable weight coefficient; is a hyperparameter, initialized to 0.75, Multiply the text template by the weight matrix The resulting matrix of values, .

[0097] Furthermore, the step of adjusting the image to be detected in the training set to a preset resolution and then inputting it into the dual-channel image encoder includes:

[0098] The image to be detected is input into the first channel CLIP image encoder, and the image features are extracted by the QKV attention mechanism from the 1st to the 5th layer, and the image features are extracted by the VV attention mechanism from the 6th to the 24th layer, and the global image embedding vector is output. and the patch-level embedding vectors at layers 6, 12, 18, and 24 , where i=1,2,3,4 correspond to the [6th, 12th, 18th, 24th] layers respectively, N is the number of patches with a value of 1370, and C is the embedding dimension with a value of 1024.

[0099] The image to be detected is input into the second channel CLIP image encoder, and through the noise suppression attention layer, the denoised patch-level embedding vectors of the 6th, 12th, 18th, and 24th layers are output. .

[0100] In one example, the image to be detected can be adjusted to a fixed resolution. , the real abnormal map is a black and white map, black represents the normal area, white represents the abnormal area, and the real abnormal map is adjusted to size.

[0101] Furthermore, the step of projecting the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules includes:

[0102] Embedding patch-level vectors and the denoised patch set embedding vector They are sent to their respective multi-layer perceptrons for mapping operations, and the embedded vectors are aligned. The mapped patch-level embedded vectors are recorded as , the corresponding number of patches is 1370 and the embedding dimension is 768.

[0103] The multilayer perceptron consists of an input layer, an intermediate layer and an output layer.

[0104] Combine Figure 4 As shown, further, the steps of interacting and aligning the projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector with the text prompt embedding vector respectively through the multimodal interaction module include:

[0105] Embedding text hints into vectors Includes normal text embedding and abnormal text embedding, through a learnable weight matrix Projected to the Key space, the matrix is obtained , and then embed the text prompt into the vector Through a learnable weight matrix Projected to the Value space, we get the matrix , at this time the feature dimension is 768;

[0106] The patch-level embedding vectors of the preliminary alignment Through their respective learnable weight matrices Projected to the Query space, we get ;

[0107] Calculate the attention weights separately, the formula is:

[0108] , ,in ;

[0109] The attention weight Scaling and normalization are performed, the formula is:

[0110] ,in, , the dimension remains unchanged, yes The dimension of is 768;

[0111] The attention weight Multiplying with the matrix V and passing through the linear projection layer, we get the mixed four-layer patch-level embedding vector, which is:

[0112] ,

[0113] ,

[0114] ,

[0115] ,

[0116] Patch-level embedding vectors and the mixed patch-level embedding vector The corresponding elements are added and normalized to obtain the final aligned patch-level embedding vector .

[0117] Furthermore, the cosine similarity between the interactively aligned four-layer patch-level embedding vector, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text hint embedding vector is calculated respectively. The steps of generating an anomaly map include:

[0118] Patch-level embedding vectors respectively with the text hint embedding vector Calculate the cosine similarity, the calculation formula is:

[0119] ,

[0120] in, Represents the number of patches at each layer. At each layer, the image is divided into 1370 patches, and each patch is converted into an embedding vector;

[0121] The similarity matrix obtained after calculating the four-layer patch-level embedding vector and the text hint embedding vector is: , corresponding to the similarity matrices of the 6th, 12th, 18th, and 24th layers respectively. Each matrix is constructed by calculating the cosine similarity between all patches in that layer and the text prompt embedding vector;

[0122] The first patch records the global image information. The similarity of the first patch is ignored when calculating the local information. The similarity of 1369 patches is used for upsampling to obtain Similar figures of size are:

[0123] ,

[0124] in, , represents the similarity matrix without the first patch, represents upsampling;

[0125] Denoised patch-level embedding vector Calculate the similarity with the text prompt embedding vector and upsample it to get Similar figures of different sizes are denoted by ;

[0126] Embed the global image into a vector Embedding vectors with text hints Calculate the cosine similarity and get the similarity matrix between the global image and the text prompt: : , the calculation formula is:

[0127] ,

[0128] Similar graphs and Each The pixel points are converted into normal and abnormal probability distributions using the softmax function, and the category with the highest probability is selected as the predicted label to obtain eight abnormal mask images, which are recorded as , size is ,The eight anomaly mask maps are weighted fused to generate the final anomaly map.

[0129] Furthermore, the loss of the industrial image anomaly detection model includes pixel-level loss and image-level loss , the formula is:

[0130] .

[0131] The predicted anomaly map Real anomaly diagram with pictures For comparison, the Focal loss and Dice loss are used for calculation, and the calculation formulas are:

[0132] ,

[0133] ,

[0134] in, represents the predicted anomaly map, This pixel The probability of belonging to its true class, is the balancing factor, is the modulation factor; Representative forecast anomaly map Pixels in The probability of belonging to the abnormal category; Represents the true label corresponding to the pixel.

[0135] The pixel-level loss Calculated as:

[0136] ,

[0137] The image-level loss uses binary cross entropy loss, which is calculated as:

[0138] ,

[0139] Where, represents the true label of Z samples, Represents the probability that the Z sample belongs to the abnormal type.

[0140] The image-level loss Calculated as:

[0141] .

[0142] In one example, using the total loss The text hint template, MMI module, MLP, and learnable weights in the noise suppression attention are trained for 30 rounds. The industrial image anomaly detection model with the lowest loss is selected as the final model.

[0143] Step 4: Use the trained industrial image anomaly detection model to perform anomaly detection on the image samples in the test set.

[0144] During the testing phase, the trained text prompts are loaded and fed into the improved text encoder to obtain normal and abnormal text embeddings. The test image is resized to a fixed resolution and fed into a dual-channel image encoder to obtain global image embeddings, multi-layer patch-level embeddings, and denoised patch-level embeddings. The patch-level embeddings are aligned and interacted with the text embeddings through the same operations as the training phase, and the similarity is calculated to generate an abnormal segmentation map. The category judgment result of the image is obtained by calculating the similarity between the global image embedding and the text embedding. The flowchart is as follows: Figure 6 As shown in the figure, the difference in the testing phase is that the parameters of the industrial image anomaly detection model have been determined, and there is no need to calculate the loss function. Instead, it is only necessary to calculate the anomaly segmentation map based on the generated similarity map. The specific implementation process is illustrated by the following example.

[0145] In an example, the trained text prompts are loaded. The normal prompt template is [a] [photo] [of][good] [v1] [v2] [cls], and the five groups of abnormal prompt templates are [a] [photo] [of] [damage] [v3] [v4] [cls] [located] [at] [the] [top] , [a] [photo] [of] [damage] [v3] [v4] [cls][located] [at] [the] [bottom] , [a] [photo] [of] [damage] [v3] [v4] [cls][located] [at] [the] [center] , [a] [photo] [of] [damage] [v3] [v4] [cls][located] [at] [the] [left] , [a] [photo] [of] [damage] [v3] [v4] [cls][located] [at] [the] [right].

[0146] Send the trained and embedded text prompts to the trained text encoder to get the normal text embedding and unusual text embeddings . Adjust the test image to Size and feed into the trained image encoder to get global image embedding , and four-layer patch-level embedding , , , and four-layer denoising patch-level embedding , , , . Embed the patch level , , , , , , , Perform MLP mapping and get , , , , , , , .Will , , , , , , , Embedded with normal text Calculate the cosine similarity and obtain a similarity graph with the hint text embedding: , , , , , , , : Perform softmax operation on each similar graph to obtain the probability of describing abnormality and normality. The following calculation formula is used to obtain the final abnormal segmentation map:

[0147] ,

[0148] in, 、 Respectively represent Similarity graphs predict the probability of abnormality and normality; 、 Respectively represent Similarity graphs predict the probability of abnormality and normality.

[0149] Abnormal segmentation map Perform binarization, if If it is less than 0.5, it is set to 0 to indicate a normal pixel, and vice versa, it is set to 1 to indicate an abnormal pixel. Embedded with normal text and unusual text embeddings Calculate the cosine similarity to get the normal and abnormal scores. Softmax the two scores and take the maximum probability between the normal and abnormal probabilities to determine whether the image is abnormal.

Claims

1. A noise suppression modality fusion alignment industrial image anomaly detection method, characterized by: The steps include: Obtain normal industrial image samples and labeled abnormal industrial image samples, build a dataset, and divide it into training set and test set according to the ratio; Construct an industrial image anomaly detection model; wherein the industrial image anomaly detection model includes a text encoder, a dual-channel image encoder, an MLP module, and a multimodal interaction module; Use the training set to train the industrial image anomaly detection model; Use the trained industrial image anomaly detection model to perform anomaly detection on image samples in the test set; The steps for training the industrial image anomaly detection model using the training set include: Constructing structured normal text prompt templates and structured abnormal text prompt templates; The normal text prompt template and five sets of abnormal text prompt templates with position information are input into the text encoder for encoding. The normal text prompt template is encoded through the noise suppression attention layer and then the normal text embedding vector is output. , five groups of abnormal text prompt templates with position information are encoded into abnormal text embedding vector groups through the noise suppression attention layer, and the five groups of abnormal text embedding vectors are averaged and pooled to generate a fused abnormal text embedding vector ; Concatenate the normal text embedding vector and the fused abnormal text embedding vector to obtain the text prompt embedding vector , , , C is the embedding dimension, R represents the real matrix space; The images to be tested in the training set are resized to a fixed resolution and then fed into a two-channel image encoder to obtain a global image embedding vector, a four-layer patch-level embedding vector, and a four-layer denoising patch-level embedding vector. Project the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules; The projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector are interacted and aligned with the text prompt embedding vector respectively through the multimodal interaction module; The cosine similarity of the interactively aligned four-layer patch-level embedding vector, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text prompt embedding vector is calculated respectively to generate anomaly maps. The loss of the obtained anomaly maps is calculated compared with the real anomaly maps, and the detection capability of the industrial image anomaly detection model is trained through loss back propagation.

2. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 1, characterized in that: The text encoder consists of 12 noise suppression attention layers.

3. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 2, characterized in that: In the noise suppression attention layer, the input is multiplied by the weight matrix , the query matrix generated and bond matrix , the generated matrix is divided into two parts, recorded as , N represents the sequence length; Calculate the first attention weight matrix and the second attention weight matrix , where scale is the scaling factor, is the activation function; The denoised attention output is generated by performing a difference operation between the first attention weight matrix and the second attention weight matrix, which is expressed as: ， Where, is the learnable weight coefficient, is a hyperparameter, Multiply the text template by the weight matrix The resulting matrix of values.

4. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 3 is characterized in that: The steps of adjusting the image to be detected in the training set to a fixed resolution and then inputting it into the dual-channel image encoder include: The image to be detected is input into the first channel CLIP image encoder, and the image features are extracted by the QKV attention mechanism from the 1st to the 5th layer, and the image features are extracted by the VV attention mechanism from the 6th to the 24th layer, and the global image embedding vector is output. and the patch-level embedding vectors at layers 6, 12, 18, and 24 , where i=1,2,3,4 correspond to the [6th, 12th, 18th, 24th] layers respectively; The image to be detected is input into the second channel CLIP image encoder, and through the noise suppression attention layer, the denoised patch-level embedding vectors of the 6th, 12th, 18th, and 24th layers are output. .

5. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 4, characterized in that: The steps of projecting the four-layer patch-level embedding vectors and the four-layer denoised patch-level embedding vectors through their respective MLP modules include: Embedding patch-level vectors and the denoised patch set embedding vector They are sent to their respective multi-layer perceptrons for mapping operations, and the embedded vectors are aligned. The mapped patch-level embedded vectors are recorded as ; The multilayer perceptron consists of an input layer, an intermediate layer and an output layer.

6. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 5, characterized in that: The steps of interacting and aligning the projected four-layer patch-level embedding vector and the four-layer denoised patch-level embedding vector with the text prompt embedding vector respectively through the multimodal interaction module include: Embedding text hints into vectors Through a learnable weight matrix Projected to the Key space, the matrix is obtained , and then embed the text prompt into the vector Through a learnable weight matrix Projected to the Value space, we get the matrix ; The patch-level embedding vectors of the preliminary alignment Through their respective learnable weight matrices Projected to the Query space, we get ; Calculate the attention weights separately, the formula is: , ,in ; The attention weight Scaling and normalization are performed, the formula is: ,in, , yes Dimensions; The attention weight Multiplying with the matrix V and passing through the linear projection layer, we get the mixed four-layer patch-level embedding vector, which is: ， ， ， ， Embedding patch-level vectors and the mixed patch-level embedding vector The corresponding elements are added and normalized to obtain the final aligned patch-level embedding vector .

7. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to claim 6, characterized in that: The steps of calculating the cosine similarity between the four-layer patch-level embedding vector after interactive alignment, the four-layer denoised patch-level embedding vector, and the global image embedding vector and the text hint embedding vector are as follows: Embedding patch-level vectors respectively with the text hint embedding vector Calculate the cosine similarity, the calculation formula is: ， in, Represents the number of patches at each layer. At each layer, the image is divided into 1370 patches, and each patch is converted into an embedding vector; The similarity matrix obtained after calculating the four-layer patch-level embedding vector and the text hint embedding vector is: ; The first patch records the global image information. When calculating the local similarity, the first patch is ignored. The similarity of 1369 patches is used for upsampling. The obtained similarity map is: ， in, , represents the similarity matrix without the first patch, represents upsampling; Denoised patch-level embedding vector The similarity is calculated with the text hint embedding vector and upsampled, and the obtained similarity graph is recorded as ; Embed the global image into a vector Embedding vectors with text hints Calculate the cosine similarity and get the similarity matrix between the global image and the text prompt: , the calculation formula is: ， Similar graphs and Each pixel in the dataset is converted into a normal and abnormal probability distribution using the softmax function, and the category with the highest probability is selected as the predicted label to obtain eight abnormal mask images, which are recorded as ,The eight anomaly mask images are weightedly fused to generate the final anomaly map.

8. The method for industrial image anomaly detection using noise suppression modal fusion alignment according to any one of claims 1 to 7, characterized in that: The loss of industrial image anomaly detection models includes pixel-level loss and image-level loss , the formula is: 。 9. The method for industrial image anomaly detection based on noise suppression modal fusion alignment according to any one of claims 1 to 7, characterized in that: The normal text prompt template is [a][photo][of][good][v1][v2][cls], and the abnormal text prompt template is [a][photo][of][damage][v3][v4][cls][located][at][the][position], where [cls] is the name of the item, [v1], [v2], [v3], and [v4] represent learnable text prompt words, and the abnormal prompt introduces position information. [position] represents the position prompt word and can be set to [top], [center], [left], [right], or [bottom].

Images