Abnormality detection method based on multi-mode denoising diffusion model

Abstract

The invention discloses an anomaly detection method based on a multi-mode denoising diffusion model, and belongs to the technical field of anomaly detection. Aiming at the problem that an existing anomaly detection method is easily influenced by single-mode information limitation and insufficient feature dependency relationship modeling in industrial quality inspection to cause a poor anomaly reconstruction effect, the method comprises the following steps: firstly, constructing a multi-mode large language model and a CLIP encoder, jointly generating an anomaly semantic description, and embedding a text mode prompt into a de-noising diffusion process; secondly, designing a denoising UNet network with a four-stage hierarchical structure, integrating a mixed attention enhancement Transform module in each stage, and synchronously capturing a long-range dependency relationship in space and channel dimensions and strengthening local details by performing traversal connection on channel-level self-attention and a pixel attention mechanism; and finally, realizing accurate reconstruction of the abnormal region through cross-modal feature alignment. Experiments show that the method improves the reconstruction quality of the abnormal image and the accuracy of anomaly detection, and has high robustness in a complex industrial scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of anomaly detection, and particularly relates to an anomaly detection method based on a multi-modal denoising diffusion model. BACKGROUND

[0002] Anomaly detection is a key link in industrial manufacturing process, and is related to the quality of products leaving the factory. It aims to identify whether there is an anomaly in the product and locate the abnormal area. Anomaly detection has become a popular research topic in the field of computer vision due to its wide application in industrial quality inspection. With the development of intelligent manufacturing, the requirements for product quality in industrial production are increasing, and image anomaly detection technology, as a key means to achieve efficient and reliable quality control, has been widely used in many fields such as electronics, medical treatment, metal, agriculture, and new energy. Traditional manual visual inspection method is inefficient and depends on subjective judgment, which is difficult to meet the dual requirements of precision and efficiency in the context of modern industrial manufacturing. Therefore, automatic anomaly detection technology based on computer vision and deep learning has become a key research and application direction.

[0003] With the development of deep learning technology, anomaly detection methods based on deep learning have emerged. Based on supervised deep learning method, a large number of paired normal and abnormal sample data sets are needed to train the deep network. However, due to the limited number of abnormal samples, the large difference in abnormal occurrence probability, privacy problems, labor-intensive labeling process, and subsequent production process adjustment, these factors make it very difficult to collect a large number of paired abnormal samples. Therefore, most of the current anomaly detection methods focus on unsupervised learning using only normal data.

[0004] Unsupervised anomaly detection technology can be mainly divided into two categories, namely feature embedding-based method and reconstruction-based method. The feature embedding-based method mainly measures the difference between normal samples and abnormal samples through a pre-trained network model on a general database (such as ImageNet) to determine whether there is an anomaly. This method excessively relies on the pre-trained feature extraction deep network, and does not migrate to the current anomaly detection field. When the pre-trained feature extraction network directly uses these biased features, these methods may fail. The reconstruction-based method focuses on learning a generative model from normal sample data, using an encoder-decoder structure to represent the feature distribution of normal sample data, and detecting and positioning anomalies by comparing the abnormal input with its predicted anomaly-free reconstruction. However, the current method has the problems of poor reconstruction quality or insufficient coverage of normal distribution, which may lead to inaccurate anomaly detection and easy reconstruction failure. In addition, with the increasing number of complex multi-dimensional data, anomaly detection faces new challenges.

[0005] Diffusion models have received widespread attention due to their powerful generative capabilities. Their emergence has expanded the research on unsupervised reconstruction anomaly detection. However, existing reconstruction methods based on diffusion models add noise to the entire image indiscriminately, lack a cueing mechanism for how the model reconstructs the abnormal region from the surrounding normal region, and only use single-modal image data for reconstruction, which has limitations in terms of semantic representation. Summary of the Invention

[0006] To address the problem that existing anomaly detection methods are susceptible to limitations of single-modal information and insufficient modeling of feature dependencies in industrial quality inspection, resulting in poor anomaly reconstruction performance, this invention provides an anomaly detection method based on a multimodal denoising diffusion model.

[0007] To achieve the above objectives, the present invention employs the following technical solutions:

[0008] An anomaly detection method based on a multimodal denoising diffusion model, the method comprising the following steps:

[0009] Step 1: Establish a deep network model for anomaly detection, and process the anomaly images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions;

[0010] The specific operation of step 1 is as follows:

[0011] The text semantic generation module includes a pre-trained multimodal large language model and a CLIP encoder to generate abnormal images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions. The generation process is as follows:

[0012]

[0013] in, This indicates a multimodal text prompt. This represents a pre-trained multimodal large language model. This indicates a CLIP encoder.

[0014] Step 2: Utilize the forward pass of the diffusion model to process the input anomalous image. The image is diffused to obtain a noisy perturbation image. The text description of the image generated in step 1 is fused with the noisy perturbation image through a cross-attention mechanism.

[0015] The specific operation of step 2 is as follows:

[0016] The diffusion model consists of a diffusion stage and a denoising and reconstruction stage. In the diffusion stage, noise is added to the anomalous image to generate a noisy, perturbed image. In the denoising and reconstruction stage, a one-step denoising and reconstruction method is used instead of the iterative method, saving a significant amount of computational resources and time. The diffusion model's denoising and reconstruction process is as follows:

[0017]

[0018] in, This represents the abnormal-free image obtained during the denoising and reconstruction process. Represents a noisy image. This indicates a multimodal text prompt. Representing noise, estimated by UNet, this direct prediction is t times faster than iterative prediction, thus significantly saving computational resources and inference time. , , It is a noise scale used to control the amount of noise added at each time step.

[0019] Step 3: Anomaly reconstruction is performed using the four-stage hierarchical structure of the anomaly detection deep network model, specifically the denoising UNet.

[0020] The denoising UNet described in step 3 is a four-layer symmetrical encoder-decoder structure, which also includes an input dimension-adjusted convolutional layer and an output dimension-adjusted convolutional layer. Each encoder-decoder structure includes multiple hybrid attention-enhanced transformer modules (HATEM), and there are skip connections between the encoder and the decoder.

[0021] The encoder structure includes: a hierarchical hybrid attention module and a gated feedforward network;

[0022] The decoder structure includes: a hierarchical hybrid attention module and a gated feedforward network;

[0023] Each enhanced hybrid attention Transformer module consists of a hierarchical hybrid attention module (HHAM) and a gated feed-forward network (GFFN), following the specification of normalization layer → hierarchical hybrid attention module (HHAM) → normalization layer → gated feed-forward network (GFFN), as shown in the formula:

[0024]

[0025] in, This represents the features after processing by the hierarchical hybrid attention module; This represents the features after processing by the gated feedforward network. This represents a hierarchical hybrid attention module. This indicates a gated feedforward network. This indicates the normalization layer.

[0026] The specific operation of step 3 is as follows:

[0027] Step 3.1: Adjust the dimension of the input channel of the convolutional layer by one input dimension, project the noisy image onto the feature space, and extract low-level features from the noisy abnormal image;

[0028] Step 3.2: The extracted low-level features are embedded into the encoder in the four-stage hierarchical denoising UNet, where the spatial resolution is halved at each stage while the channel capacity is increased to generate a low-resolution latent representation.

[0029] Step 3.3: The low-resolution latent representation processed by the encoder and the multimodal text prompts are fused through a cross-attention mechanism. The fused features are fed into a symmetric decoder to gradually decode the anomaly-free reconstructed image features.

[0030] Step 3.4: Adjust the feature dimensions of the reconstructed image using a 3×3 convolutional layer to output the final reconstructed image.

[0031] The hierarchical hybrid attention module consists of a channel-level self-attention module and a pixel-level attention module;

[0032] The operation of the channel-level self-attention module is as follows:

[0033] The hierarchical hybrid attention module mainly consists of channel-level self-attention and pixel-level attention layers. By performing related interactions in the channel dimension rather than the spatial dimension, the self-attention maintains linear complexity, avoiding the significant computational resource consumption caused by the quadratic increase in time and memory complexity of key (K) lookup (Q) dot product interactions with the spatial resolution of the input image in traditional Transformer models. Global contextual information is implicitly encoded by calculating the cross-covariance between channels. Furthermore, we introduce deep convolution techniques to enhance the interaction between local contextual information, thereby improving overall performance before generating the global attention map.

[0034] The operation of the channel-level self-attention module is as follows:

[0035] First, the input feature map is flattened into a set of feature vectors using a 1×1 convolutional layer;

[0036] Next, a query is generated using a 3×3 depthwise convolution. ,key Sum A matrix used to encode the channel-level spatial context;

[0037] Subsequently, pixel-level cross-channel context is aggregated through 1×1 convolution to enhance local semantics;

[0038] Next, the query and value matrices are reshaped to achieve channel-level dot product interaction between the query and key matrices, resulting in a channel interaction graph. Before the softmax operation, a learnable scaling parameter is used to control the strength of the dot product interaction. The dot product of the channel interaction graph and the reshaped value matrix yields a channel self-attention graph.

[0039] Finally, the channel self-attention map is passed to the input features through a 1×1 convolutional layer and residual connections to obtain the output; the entire process is defined as follows:

[0040]

[0041] in, This represents a 3×3 depthwise convolution. Represents a 1×1 convolution. This represents the input features generated by a 1×1 convolution. This indicates a reshaping operation. It is a learnable scaling parameter. This represents the input feature map of the channel-level self-attention module; This represents the output feature map of the channel-level self-attention module; Represents a self-attention map; It is a 1×1 convolution; This represents the reshaped query matrix, key matrix, and value matrix;

[0042] A pixel-level attention module is introduced. This module generates pixel-level attention by applying convolution operations along the channel dimension, progressively reducing the channel dimension. The calculation method of the pixel-level attention module is as follows:

[0043]

[0044] in, This represents the output feature map of the channel-level self-attention module. This represents the convolution operation. and These correspond to the ReLU and Sigmoid functions, respectively.

[0045] By cascading a channel-level self-attention module and a pixel-level attention mechanism, comprehensive extraction and reconstruction of anomalous image features are achieved, enabling the network to focus more on the reconstruction of anomalous regions. In this way, pixels with severe anomalies receive more attention to avoid underconstruction, while pixels with slight anomalies are assigned less weight to prevent overconstruction.

[0046] The specific operation of the gated feedforward network is as follows:

[0047] First, the feature channels are expanded using 1×1 convolutional layers; then, the features are non-linearly mapped using intermediate hidden layers; finally, the channels are compressed back to the original input dimension using 1×1 convolutional layers.

[0048] We introduced a gating mechanism to control the information flow, avoiding the propagation of redundant information and improving the propagation capability and efficiency of the traditional feedforward module. Simultaneously, we introduced adaptive scale residual connections to adaptively supplement lost features.

[0049] Specifically, X is first normalized and mapped to high-dimensional features. These high-dimensional features are then fed into a local detail enhancement gating mechanism structure. The gating mechanism consists of the product of linear transformation layer elements from two parallel paths, one of which is non-linearly activated by GELU. Furthermore, deep convolutions are introduced into the gated feedforward network to capture information about spatially neighboring pixel locations, which helps learn the local image structure and thus improves model performance. The gating mechanism is represented as follows:

[0050]

[0051] in, Represents the gate vector, LN represents element-wise multiplication; LN represents the normalization layer.

[0052] An adaptive scaling residual connection with a scaling factor is added after the gating mechanism. This operation helps to supplement ignored low-level features, as shown in the formula:

[0053]

[0054] in, This represents the adaptive scaling factor.

[0055] Step 4: Input the reconstructed image into the detection sub-network and compare it with the original image to obtain the anomaly detection and localization results;

[0056] Step 5: Set the loss function for the reconstruction sub-network and the loss function for the detection sub-network. Under the constraints of the loss function, iteratively train each sub-network on an industrial scenario anomaly dataset until the network converges.

[0057] The specific operation of step 5 is as follows:

[0058] The total network loss function consists of denoising loss and detection loss. By constraining and determining the convergence point of the network model through the reconstruction sub-network loss function and the detection sub-network loss function, the formula for the total network loss function is:

[0059]

[0060] in, This represents the total loss function; and These represent the loss functions of the reconstruction subnetwork and the detection subnetwork, respectively.

[0061] The formula for calculating the loss function of the reconstructed subnetwork is:

[0062]

[0063] in, Indicates noise The predicted value;

[0064] The formula for calculating the loss function of the detection subnetwork is:

[0065]

[0066] in, Indicates smoothed L1 loss, Indicates focal loss. This represents the true label of the input abnormal image. It is the output of the detection network. and These are weights used to balance the importance of the two loss functions.

[0067] Compared with the prior art, the present invention has the following advantages:

[0068] This method establishes a deep network model for anomaly detection, which utilizes a multimodal large-scale language model and a visual language model to generate comprehensive textual descriptions containing anomaly semantics to guide the anomaly reconstruction network. A hybrid attention-enhanced Transformer structure is established. This structure combines global long-range dependency modeling and local detail feature extraction capabilities, implemented through concatenated channel interactive self-attention and pixel-level attention mechanisms. A gated feedforward network with adaptive scaling is proposed. This gated structure suppresses features with less information, allowing only useful information to pass through the network layers. Adaptively scaled residual connections enable the feedforward network to adaptively supplement low-level features, resulting in more comprehensive fused features. The proposed reconstruction sub-network function and detection sub-network loss function better constrain the training of the anomaly detection network, obtaining the final anomaly-free reconstructed image. The final reconstructed image results of this method are good, and compared with other methods, this method has better universality and robustness. Attached Figure Description

[0069] Figure 1 This is the overall anomaly detection deep network model of the present invention;

[0070] Figure 2 This is a visualization of the abnormal reconstruction results of the present invention. Detailed Implementation

[0071] To gain a deeper understanding of this invention, we will provide a comprehensive and detailed description. However, this invention has various implementations and is not limited to the specific examples listed herein. These examples are presented to enhance a full understanding of the disclosure of this invention.

[0072] An anomaly detection method based on a multimodal denoising diffusion model, the method comprising the following steps:

[0073] Step 1: Establish a deep network model for anomaly detection, and process the anomaly images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions;

[0074] The specific operation of step 1 is as follows:

[0075] The text semantic generation module includes a pre-trained multimodal large language model and a CLIP encoder to generate abnormal images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions. The generation process is as follows:

[0076]

[0077] in, This indicates a multimodal text prompt. This represents a pre-trained multimodal large language model. This indicates a CLIP encoder.

[0078] Step 2: Utilize the forward pass of the diffusion model to process the input anomalous image. The image is diffused to obtain a noisy perturbation image. The text description of the image generated in step 1 is fused with the noisy perturbation image through a cross-attention mechanism.

[0079] The specific operation of step 2 is as follows:

[0080] The diffusion model consists of a diffusion stage and a denoising and reconstruction stage. In the diffusion stage, noise is added to the anomalous image to generate a noisy, perturbed image. In the denoising and reconstruction stage, a one-step denoising and reconstruction method is used instead of the iterative method, saving a significant amount of computational resources and time. The diffusion model's denoising and reconstruction process is as follows:

[0081]

[0082] in, This represents the abnormal-free image obtained during the denoising and reconstruction process. Represents a noisy image. This indicates a multimodal text prompt. Representing noise, estimated by UNet, this direct prediction is t times faster than iterative prediction, thus significantly saving computational resources and inference time. , , It is a noise scale used to control the amount of noise added at each time step.

[0083] Step 3: Anomaly reconstruction is performed using the four-stage hierarchical structure of the anomaly detection deep network model, specifically the denoising UNet.

[0084] The denoising UNet described in step 3 is a four-layer symmetrical encoder-decoder structure, which also includes an input dimension-adjusted convolutional layer and an output dimension-adjusted convolutional layer. Each encoder-decoder structure includes multiple hybrid attention-enhanced transformer modules (HATEM), and there are skip connections between the encoder and the decoder.

[0085] The encoder structure includes: a hierarchical hybrid attention module and a gated feedforward network;

[0086] The decoder structure includes: a hierarchical hybrid attention module and a gated feedforward network;

[0087] Each enhanced hybrid attention Transformer module consists of a hierarchical hybrid attention module (HHAM) and a gated feed-forward network (GFFN), following the specification of normalization layer → hierarchical hybrid attention module (HHAM) → normalization layer → gated feed-forward network (GFFN), as shown in the formula:

[0088]

[0089] in, This represents the features after processing by the hierarchical hybrid attention module; This represents the features after processing by the gated feedforward network. This represents a hierarchical hybrid attention module. This indicates a gated feedforward network. This indicates the normalization layer.

[0090] The specific operation of step 3 is as follows:

[0091] Step 3.1: Adjust the dimension of the input channel of the convolutional layer by one input dimension, project the noisy image onto the feature space, and extract low-level features from the noisy abnormal image;

[0092] Step 3.2: The extracted low-level features are embedded into the encoder in the four-stage hierarchical denoising UNet, where the spatial resolution is halved at each stage while the channel capacity is increased to generate a low-resolution latent representation.

[0093] Step 3.3: The low-resolution latent representation processed by the encoder and the multimodal text prompts are fused through a cross-attention mechanism. The fused features are fed into a symmetric decoder to gradually decode the anomaly-free reconstructed image features.

[0094] Step 3.4: Adjust the feature dimensions of the reconstructed image using a 3×3 convolutional layer to output the final reconstructed image.

[0095] The hierarchical hybrid attention module consists of a channel-level self-attention module and a pixel-level attention module;

[0096] The operation of the channel-level self-attention module is as follows:

[0097] The hierarchical hybrid attention module mainly consists of channel-level self-attention and pixel-level attention layers. By performing related interactions in the channel dimension rather than the spatial dimension, the self-attention maintains linear complexity, avoiding the significant computational resource consumption caused by the quadratic increase in time and memory complexity of key (K) lookup (Q) dot product interactions with the spatial resolution of the input image in traditional Transformer models. Global contextual information is implicitly encoded by calculating the cross-covariance between channels. Furthermore, we introduce deep convolution techniques to enhance the interaction between local contextual information, thereby improving overall performance before generating the global attention map.

[0098] The operation of the channel-level self-attention module is as follows:

[0099] First, the input feature map is flattened into a set of feature vectors using a 1×1 convolutional layer;

[0100] Next, a query is generated using a 3×3 depthwise convolution. ,key Sum A matrix used to encode the channel-level spatial context;

[0101] Subsequently, pixel-level cross-channel context is aggregated through 1×1 convolution to enhance local semantics;

[0102] Next, the query and value matrices are reshaped to achieve channel-level dot product interaction between the query and key matrices, resulting in a channel interaction graph. Before the softmax operation, a learnable scaling parameter is used to control the strength of the dot product interaction. The dot product of the channel interaction graph and the reshaped value matrix yields a channel self-attention graph.

[0103] Finally, the channel self-attention map is passed to the input features through a 1×1 convolutional layer and residual connections to obtain the output; the entire process is defined as follows:

[0104]

[0105] in, This represents a 3×3 depthwise convolution. Represents a 1×1 convolution. This represents the input features generated by a 1×1 convolution. This indicates a reshaping operation. It is a learnable scaling parameter. This represents the input feature map of the channel-level self-attention module; This represents the output feature map of the channel-level self-attention module; Represents a self-attention map; It is a 1×1 convolution; This represents the reshaped query matrix, key matrix, and value matrix;

[0106] A pixel-level attention module is introduced. This module generates pixel-level attention by applying convolution operations along the channel dimension, progressively reducing the channel dimension. The calculation method of the pixel-level attention module is as follows:

[0107]

[0108] in, This represents the output feature map of the channel-level self-attention module. This represents the convolution operation. and These correspond to the ReLU and Sigmoid functions, respectively.

[0109] By cascading a channel-level self-attention module and a pixel-level attention mechanism, comprehensive extraction and reconstruction of anomalous image features are achieved, enabling the network to focus more on the reconstruction of anomalous regions. In this way, pixels with severe anomalies receive more attention to avoid underconstruction, while pixels with slight anomalies are assigned less weight to prevent overconstruction.

[0110] The specific operation of the gated feedforward network is as follows:

[0111] First, the feature channels are expanded using 1×1 convolutional layers; then, the features are non-linearly mapped using intermediate hidden layers; finally, the channels are compressed back to the original input dimension using 1×1 convolutional layers.

[0112] We introduced a gating mechanism to control the information flow, avoiding the propagation of redundant information and improving the propagation capability and efficiency of the traditional feedforward module. Simultaneously, we introduced adaptive scale residual connections to adaptively supplement lost features.

[0113] Specifically, X is first normalized and mapped to high-dimensional features. These high-dimensional features are then fed into a local detail enhancement gating mechanism structure. The gating mechanism consists of the product of linear transformation layer elements from two parallel paths, one of which is non-linearly activated by GELU. Furthermore, deep convolutions are introduced into the gated feedforward network to capture information about spatially neighboring pixel locations, which helps learn the local image structure and thus improves model performance. The gating mechanism is represented as follows:

[0114]

[0115] in, Represents the gate vector, LN represents element-wise multiplication; LN represents the normalization layer.

[0116] An adaptive scaling residual connection with a scaling factor is added after the gating mechanism. This operation helps to supplement ignored low-level features, as shown in the formula:

[0117]

[0118] in, This represents the adaptive scaling factor.

[0119] Step 4: Input the reconstructed image into the detection sub-network and compare it with the original image to obtain the anomaly detection and localization results;

[0120] Step 5: Set the loss function for the reconstruction sub-network and the loss function for the detection sub-network. Under the constraints of the loss function, iteratively train each sub-network on an industrial scenario anomaly dataset until the network converges.

[0121] The specific operation of step 5 is as follows:

[0122] The total network loss function consists of denoising loss and detection loss. By constraining and determining the convergence point of the network model through the reconstruction sub-network loss function and the detection sub-network loss function, the formula for the total network loss function is:

[0123]

[0124] in, This represents the total loss function; and These represent the loss functions of the reconstruction subnetwork and the detection subnetwork, respectively.

[0125] The formula for calculating the loss function of the reconstructed subnetwork is:

[0126]

[0127] in, Indicates noise The predicted value;

[0128] The formula for calculating the loss function of the detection subnetwork is:

[0129]

[0130] in, Indicates smoothed L1 loss, Indicates focal loss. This represents the true label of the input abnormal image. It is the output of the detection network. and These are weights used to balance the importance of the two loss functions.

[0131] Step 6: Train the network using the publicly available MVTec-AD and VisA training sets.

[0132] Figure 2 The figures show the anomaly reconstruction and detection results of the method of this invention on different datasets. As can be seen from the figures, this method performs high-quality anomaly-free restoration and fine-grained reconstruction of normal regions for various types of anomaly images, which is beneficial for the detection and localization of various types of anomalies. This invention can effectively detect and reconstruct both large and small anomalies. Table 1 shows the quantitative indicators of anomaly detection and localization of this invention.

[0133] Table 1. Quantitative Indicators for Anomaly Detection and Localization in this Invention

[0134] I-AUROC (%) P-AUROC (%) PRO (%) AP (%) Tile 100 99.8 98.3 97.1 Toothbrush 100 99.4 97.4 67.0 Pcb2 100 99.2 95.9 31.4 Macaroni1 100 99.5 99.5 44.3 Capsules 100 99.7 98.6 55.0

[0135] As shown in Table 1, the proposed method performs excellently in quantitative evaluation data of anomaly detection and anomaly localization, reaching an advanced level and demonstrating its effectiveness.

[0136] Contents not described in detail in this specification are prior art known to those skilled in the art. Although illustrative specific embodiments of the invention have been described above to facilitate understanding by those skilled in the art, it should be understood that the invention is not limited to the scope of the specific embodiments. Various modifications are readily apparent to those skilled in the art as long as they fall within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of this invention are protected.

Claims

1. An anomaly detection method based on a multimodal denoising diffusion model, characterized in that, The method includes the following steps: Step 1: Establish a deep network model for anomaly detection, and process the anomaly images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions; Step 2: Utilize the forward pass of the diffusion model to process the input anomalous image. The image is diffused to obtain a noisy perturbation image. The text description of the image generated in step 1 is fused with the noisy perturbation image through a cross-attention mechanism. Step 3: Anomaly reconstruction is performed using the four-stage hierarchical structure of the anomaly detection deep network model, specifically the denoising UNet. Step 4: Input the reconstructed image into the detection sub-network and compare it with the original image to obtain the anomaly detection and localization results; Step 5: Set the loss function for the reconstruction sub-network and the loss function for the detection sub-network. Under the constraints of the loss function, iteratively train each sub-network on an industrial scenario anomaly dataset until the network converges.

2. The anomaly detection method based on a multimodal denoising diffusion model according to claim 1, characterized in that, The specific operation of step 1 is as follows: The text semantic generation module includes a pre-trained multimodal large language model and a CLIP encoder to generate abnormal images. and the problem The text semantic generation module in the anomaly detection deep network model generates image text descriptions. The generation process is as follows: ,in, This indicates a multimodal text prompt. This represents a pre-trained multimodal large language model. This indicates a CLIP encoder.

3. The anomaly detection method based on a multimodal denoising diffusion model according to claim 2, characterized in that, The specific operation of step 2 is as follows: The diffusion model consists of a diffusion stage and a denoising and reconstruction stage. In the diffusion stage, noise is added to the anomalous image to generate a noisy perturbation image. In the denoising and reconstruction stage, a one-step denoising and reconstruction method is used. The denoising and reconstruction process of the diffusion model is as follows: ,in, This represents the abnormal-free image obtained during the denoising and reconstruction process. Represents a noisy image. This indicates a multimodal text prompt. This represents noise, estimated by UNet; , , It is a noise scale used to control the amount of noise added at each time step.

4. The anomaly detection method based on a multimodal denoising diffusion model according to claim 3, characterized in that, The denoising UNet described in step 3 is a four-layer symmetrical encoder-decoder structure, which also includes an input dimension-adjusted convolutional layer and an output dimension-adjusted convolutional layer. Each encoder-decoder structure includes multiple enhanced hybrid attention-enhanced Transformer modules, and there are skip connections between the encoder and the decoder. The encoder structure includes: a hierarchical hybrid attention module and a gated feedforward network; The decoder structure includes: a hierarchical hybrid attention module and a gated feedforward network; Each enhanced hybrid attention Transformer module consists of a hierarchical hybrid attention module and a gated feedforward network, following the specification of normalization layer → hierarchical hybrid attention module → normalization layer → gated feedforward network, as shown in the formula: ,in, This represents the features after processing by the hierarchical hybrid attention module; This represents the features after processing by the gated feedforward network. This represents a hierarchical hybrid attention module. This indicates a gated feedforward network. This indicates the normalization layer.

5. The anomaly detection method based on a multimodal denoising diffusion model according to claim 4, characterized in that, The specific operation of step 3 is as follows: Step 3.1: Adjust the dimension of the input channel of the convolutional layer by one input dimension, project the noisy image onto the feature space, and extract low-level features from the noisy abnormal image; Step 3.2: The extracted low-level features are embedded into the encoder in the four-stage hierarchical denoising UNet, where the spatial resolution is halved at each stage while the channel capacity is increased to generate a low-resolution latent representation. Step 3.3: The low-resolution latent representation processed by the encoder and the multimodal text prompts are fused through a cross-attention mechanism. The fused features are fed into a symmetric decoder to gradually decode the anomaly-free reconstructed image features. Step 3.4: Adjust the feature dimensions of the reconstructed image using a 3×3 convolutional layer to output the final reconstructed image.

6. The anomaly detection method based on a multimodal denoising diffusion model according to claim 5, characterized in that, The hierarchical hybrid attention module consists of a channel-level self-attention module and a pixel-level attention module; The operation of the channel-level self-attention module is as follows: First, the input feature map is flattened into a set of feature vectors using a 1×1 convolutional layer; Next, a query is generated using a 3×3 depthwise convolution. ,key Sum A matrix used to encode the channel-level spatial context; Subsequently, pixel-level cross-channel context is aggregated through 1×1 convolution to enhance local semantics; Next, the query and value matrices are reshaped to achieve channel-level dot product interaction between the query and key matrices, resulting in a channel interaction graph. Before the softmax operation, a learnable scaling parameter is used to control the strength of the dot product interaction. The dot product of the channel interaction graph and the reshaped value matrix yields a channel self-attention graph. Finally, the channel self-attention map is passed to the input features through a 1×1 convolutional layer and residual connections to obtain the output; the entire process is defined as follows: ,in, This represents a 3×3 depthwise convolution. Represents a 1×1 convolution. This represents the input features generated by a 1×1 convolution. This indicates a reshaping operation. It is a learnable scaling parameter. This represents the input feature map of the channel-level self-attention module; This represents the output feature map of the channel-level self-attention module; Represents a self-attention map; It is a 1×1 convolution; This represents the reshaped query matrix, key matrix, and value matrix; A pixel-level attention module is introduced. This module generates pixel-level attention by applying convolution operations along the channel dimension, progressively reducing the channel dimension. The calculation method of the pixel-level attention module is as follows: ,in, This represents the output feature map of the channel-level self-attention module. This represents the convolution operation. and These correspond to the ReLU and Sigmoid functions, respectively.

7. The anomaly detection method based on a multimodal denoising diffusion model according to claim 6, characterized in that, The operation of the gated feedforward network is as follows: First, the feature channels are expanded using 1×1 convolutional layers; then, the features are non-linearly mapped using intermediate hidden layers; finally, the channels are compressed back to the original input dimension using 1×1 convolutional layers. Specifically, X is first normalized and mapped to high-dimensional features. These high-dimensional features are then fed into a local detail enhancement gating mechanism structure. The gating mechanism consists of the product of linear transformation layer elements from two parallel paths, one of which is non-linearly activated by GELU. Furthermore, deep convolutions are introduced into the gated feedforward network to capture information about spatially neighboring pixel locations. The gating mechanism is represented as follows: ,in, Represents the gate vector, LN represents element-wise multiplication; LN represents the normalization layer. An adaptive scaling residual connection with a scaling factor is added after the gating mechanism, as shown in the following equation: ,in, This represents the adaptive scaling factor.

8. The anomaly detection method based on a multimodal denoising diffusion model according to claim 7, characterized in that, The specific operation of step 5 is as follows: The total network loss function consists of denoising loss and detection loss. By constraining and determining the convergence point of the network model through the reconstruction sub-network loss function and the detection sub-network loss function, the formula for the total network loss function is: ,in, This represents the total loss function; and These represent the loss functions of the reconstruction subnetwork and the detection subnetwork, respectively. The formula for calculating the loss function of the reconstructed subnetwork is: ,in, Indicates noise The predicted value; The formula for calculating the loss function of the detection subnetwork is: ,in, Indicates smoothed L1 loss, Indicates focal loss. This represents the true label of the input abnormal image. It is the output of the detection network. and These are weights used to balance the importance of the two loss functions.

Images