A CLIP-based semantic collaborative guidance method for DRAEM workpiece surface anomaly detection

Abstract

This invention discloses a DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance, belonging to the fields of industrial vision inspection and unsupervised anomaly detection technology. This invention improves the matching degree of the real distribution of synthesized anomalies by using CLIP semantic prior constraints on the anomaly generation region, concentrating synthesized anomalies in the main body and key structural parts of the workpiece, highly consistent with the spatial distribution of real industrial defects, reducing the interference of background pseudo-anomalies on the model, and significantly improving the model's generalization ability to real defects. This invention enhances the perception capability of high-level anomalies by using global-local semantic difference modeling for reconstructed perception, breaking through the limitation of traditional DRAEM relying solely on pixel visual differences, effectively capturing high-level anomalies such as structural defects and semantic inconsistencies, improving the detection and localization accuracy of complex defects. This invention balances fine-grained localization and semantic guidance, employing a semantically gating fusion method, with CLIP prior serving only as supplementary high-level guidance.

Description

Technical Field

[0001] This invention relates to the field of industrial visual inspection and unsupervised anomaly detection technology, specifically to the DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance. Background Technology

[0002] Industrial surface anomaly detection aims to learn the normal distribution of targets from a large number of normal samples and identify local areas that deviate from normal appearance during the testing phase. Unlike general target recognition tasks in natural scenes, industrial defects are typically characterized by small areas, diverse shapes, class imbalance, and expensive annotation. Especially in unsupervised or weakly supervised scenarios, models often can only be trained using normal samples, making anomaly detection and localization a key problem in the field of industrial vision.

[0003] In recent years, reconstruction-based anomaly detection methods have attracted widespread attention because they do not require real anomaly annotation. DRAEM is one of the representative frameworks, which models surface anomaly detection as a joint learning process of reconstruction bias and discriminative segmentation through a two-branch structure of reconstruction subnetwork and discriminative subnetwork: the reconstruction network is responsible for restoring the input containing anomalies to an image that is closer to the normal distribution, while the discriminative network models the joint representation of the input image and the reconstructed image, thereby achieving pixel-level anomaly localization.

[0004] However, the original DRAEM still has two limitations.

[0005] Firstly, in the anomaly simulation stage, the original method generates anomaly regions by mixing Perlin noise, random textures, and transparency. Although this global random perturbation can generate enough anomaly samples, it does not explicitly constrain the anomalies to be preferentially distributed in the main body of the workpiece and its critical structural areas. Therefore, in specific industrial scenarios, noise may fall into the background area or non-critical area, weakening the consistency between the synthetic anomalies and the actual defect distribution.

[0006] Secondly, in the anomaly detection stage, the original method mainly relies on the visual differences between the input image and the reconstructed image for localization. It is relatively sensitive to local texture deviations, but its ability to perceive high-level structural anomalies, key part defects, and semantic inconsistencies is still limited.

[0007] Existing technologies that lack regional constraints in anomaly synthesis generate a large number of pseudo-anomalies that deviate from the true distribution, making the model overly sensitive to irrelevant noise and thus weakening the discriminant network's ability to generalize to identify real anomalies during the testing phase.

[0008] At the same time, relying solely on pixel-level reconstruction makes it difficult for the model to establish a high-level abstract structural mapping, and it often only learns to mechanically smooth artificially synthesized anomalies into a shallow repair effect of surrounding normal pixels.

[0009] Therefore, a new solution is needed to address the above problems. Summary of the Invention

[0010] The purpose of this invention is to provide a DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance, so as to solve the technical problems mentioned in the background art.

[0011] To achieve the above objectives, the present invention provides the following technical solution: a DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance, comprising the following steps:

[0012] S1: CLIP semantic-guided region constraint anomaly generation. Input a normal workpiece image, extract the region prior mask of the workpiece body and key structural regions through CLIP cross-modal semantic matching, constrain the simulated anomaly to the target region corresponding to the region prior mask, and generate a training image with synthetic anomaly and the corresponding anomaly ground truth mask.

[0013] S2: Abnormal Image Reconstruction. The training image with synthetic anomalous features is input into the reconstruction subnetwork, and the output is a reconstructed image that restores the normal appearance.

[0014] S3: Global-local semantic difference prior calculation for reconstructed perception. The training image with synthesis anomaly and the reconstructed image are respectively input into the frozen CLIP visual encoder, and the global semantic difference score and local semantic difference prior map of the two are calculated in the CLIP cross-modal semantic space.

[0015] S4: Semantic-guided anomaly detection and localization, inputting the training image with synthetic anomalies and the reconstructed image into the discriminative subnetwork, performing semantic gating fusion of the local semantic difference prior map and the intermediate features of the discriminative subnetwork, and outputting a pixel-level anomaly prediction map based on the fused features;

[0016] S5: Multi-objective joint optimization training, constructing a total loss function that includes reconstruction loss, segmentation loss, region constraint loss and semantic consistency loss, and performing end-to-end joint training on the reconstruction subnetwork and the discriminative subnetwork;

[0017] S6: Workpiece surface anomaly detection. Input the image of the workpiece to be detected into the trained network, and output the image-level anomaly detection result and the pixel-level anomaly localization result.

[0018] Furthermore, the extraction process of the region prior mask in S1 is specifically as follows:

[0019] A set of pre-defined text prompts describing the main body, head region, and waist region of the workpiece is used to obtain a coarse-grained semantic response map of the main body of the workpiece through similarity matching between the CLIP image encoder and the text encoder.

[0020] By combining the image geometry, the coarse-grained semantic response map is refined to obtain the workpiece body mask;

[0021] Based on the minimum bounding box height of the workpiece body mask, the head region mask and waist region mask are divided from the workpiece body mask according to the preset normalized position parameters.

[0022] The region prior mask includes the workpiece body mask, the head region mask, and the waist region mask.

[0023] Furthermore, the process of simulating anomalies in S1 specifically includes:

[0024] Generate the original anomalous shape mask using Perlin noise;

[0025] The target region mask is obtained by sampling from the region prior mask according to the preset sampling probability. The original abnormal shape mask is multiplied pixel by pixel with the target region mask to obtain the region-restricted abnormal ground truth mask.

[0026] Abnormal texture images are selected from an external texture library. The abnormal textures are then blended with normal workpiece images within the region corresponding to the abnormal ground truth mask using a transparency coefficient to generate a training image with synthetic anomalies.

[0027] Furthermore, the region constraint loss is composed of a weighted average of a background suppression term and a key region bias term:

[0028] The background suppression term is the L1 norm of the intersection of the anomaly truth mask and the background region, used to suppress anomalies from falling into the background region outside the workpiece body.

[0029] The critical region bias term is constructed based on the intersection ratio of the anomaly truth mask and the head and waist regions, and is used to guide anomalies to be preferentially distributed in the critical structural regions of the head and waist.

[0030] Furthermore, the calculation process of the global semantic difference score in S3 is as follows:

[0031] Construct a set of normal prompt words describing the normal state of the workpiece and a set of abnormal prompt words describing the abnormal state of the workpiece, respectively.

[0032] The maximum similarity between the global features of the training image and the reconstructed image with synthetic anomalies and the features of the two types of prompt words is calculated respectively to obtain the abnormal semantic score of the abnormal input image and the abnormal semantic score of the reconstructed image.

[0033] The abnormal semantic scores of the two types of images are subtracted to obtain the global semantic difference score for reconstructed perception, which is used as the image-level abnormal semantic score.

[0034] Furthermore, the calculation process of the local semantic difference prior graph in S3 is as follows:

[0035] Extract patchtoken features of training and reconstructed images with synthetic anomalies from the Transformer layer of the CLIP visual encoder;

[0036] Calculate the maximum semantic difference between each patchtoken feature and the normal prompt word set and the abnormal prompt word set to obtain the local abnormal semantic response map of the abnormal input image and the reconstructed image respectively;

[0037] The two local anomaly semantic response maps are subtracted position by position to obtain a local semantic difference map. After upsampling to the original image size, a local semantic difference prior map is obtained.

[0038] Furthermore, the semantic gating fusion in S4 specifically includes:

[0039] Upsample the local semantic difference prior map to the same size as the intermediate features of the discriminant subnetwork;

[0040] by As a semantic attention gain coefficient, it is multiplied pixel-by-pixel with the intermediate features of the discriminant subnetwork to obtain the semantically enhanced fusion feature; where α is a weight parameter that controls the strength of semantic prior guidance.

[0041] Furthermore, the expression for the total loss function is:

[0042]

[0043] in, The reconstruction loss is used to constrain the pixel consistency between the reconstructed image and the original normal image. The segmentation loss is used to supervise the consistency between the anomaly prediction map and the anomaly truth mask; For regional constraint loss; , , The weighting coefficients; the semantic consistency loss The L1 distance between the Sigmoid-normalized anomaly prediction map and the normalized local semantic difference prior map is used to constrain the trend consistency between the prediction results and the CLIP semantic prior.

[0044] Furthermore, S6 specifically includes:

[0045] The image of the workpiece to be detected is input into the trained reconstruction sub-network to obtain the reconstruction image to be detected, and then input into the discriminant sub-network to obtain the pixel-level anomaly score map. At the same time, the global semantic difference score between the image to be detected and the reconstruction image to be detected is calculated through CLIP.

[0046] Pixel-level anomaly localization results are obtained based on pixel-level anomaly score maps, and image-level anomaly detection results are obtained by combining the statistical values ​​of pixel-level anomaly score maps with global semantic difference scores.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] 1. This invention improves the matching degree of the real distribution of synthesized anomalies. By using CLIP semantic prior constraints on the anomaly generation region, the synthesized anomalies are concentrated in the main body of the workpiece and key structural parts, which is highly consistent with the spatial distribution of real industrial defects. This reduces the interference of background pseudo-anomalies on the model and significantly improves the model's generalization ability to real defects.

[0049] 2. This invention enhances the ability to perceive high-level anomalies. By reconstructing the global-local semantic difference model of perception, it breaks through the limitation of traditional DRAEM which only relies on pixel visual differences. It can effectively capture high-level anomalies such as structural defects and semantic inconsistencies, and improve the detection and localization accuracy of complex defects.

[0050] 3. This invention takes into account both fine-grained localization and semantic guidance, and adopts a semantic gating fusion method. CLIP prior is only used as a high-level supplementary guidance and does not replace DRAEM's original fine-grained visual discrimination capability. While improving semantic perception, it retains the pixel-level localization advantage of the reconstruction method.

[0051] 4. The unified and efficient training framework of this invention achieves end-to-end training through multi-loss joint optimization, eliminating the need for complex multi-stage training processes. It combines the efficiency of DRAEM with the semantic expression capabilities of CLIP, making it easy to deploy in industrial scenarios. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the overall framework of the method of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the generation of region constraint anomalies guided by CLIP semantics in this invention.

[0055] Figure 3 This is a schematic diagram illustrating the global-local semantic difference prior modeling for reconstructing perception according to the present invention.

[0056] Figure 4 This is a schematic diagram illustrating the visualization and comparison results of anomaly detection on the Copper-AD dataset of this invention;

[0057] Figure 5 This is a schematic diagram illustrating the visualization and comparison results of anomaly detection on the MVTec-AD dataset of this invention;

[0058] Figure 6 This is a schematic diagram showing the visualization comparison results of anomaly detection on the VisA dataset of this invention. Detailed Implementation

[0059] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0060] This invention introduces CLIP (Collection of Anomalies) into both the anomaly generation front-end and the anomaly discrimination back-end. At the front-end, CLIP acquires prior knowledge of the workpiece body and key structural areas such as the head and waist, restricting synthesized anomalies to locations more closely aligned with real-world defect distributions. At the back-end, by comparing the global and local semantic responses of the anomaly input map and the reconstructed map in the CLIP cross-modal space, a semantic prior for reconstruction-aware anomalies is constructed, further guiding the discrimination branch to achieve more precise anomaly localization. In this way, the model retains the advantages of DRAEM in fine-grained anomaly localization while also addressing the shortcomings of traditional reconstruction difference modeling at the semantic level.

[0061] This invention collects and organizes a dataset of real industrial anomaly detection and compares and tests various benchmark methods.

[0062] To address the issues of overly random anomaly simulation and lack of prior knowledge of real workpiece structures in the traditional DRAEM framework, this invention proposes a workpiece region constraint anomaly generation method based on CLIP semantic guidance, which can effectively enhance the consistency between synthesized anomalies and the distribution of real defects.

[0063] By constructing a global-local differential prior in CLIP space, the shortcomings of traditional reconstruction methods in perceiving structural and semantic anomalies are overcome, thereby enhancing the localization accuracy of complex anomalies.

[0064] The method of this invention generally inherits the "anomaly generation—reconstruction and recovery—joint discrimination" main line of DRAEM. Given an image of a normal workpiece... First, training samples with simulated anomalies are obtained through the anomaly generation module. and its corresponding anomaly mask ; then, Input the reconstruction subnetwork R(·) to obtain the reconstructed image. Finally, determine the subnetwork. In the input image Reconstruction diagram The pixel-level anomaly score map is output under the joint guidance of CLIP semantic prior. Unlike the original DRAEM, this invention introduces CLIP semantic priors simultaneously in both the anomaly generation and anomaly discrimination stages, constructing a front-end and back-end collaborative framework: At the front end, CLIP extracts priors for the main body of the workpiece and key structural regions, thereby limiting the location of anomaly generation; at the back end, CLIP compares the response changes of the anomaly input map and the reconstructed map in the cross-modal semantic space, thereby generating global and local semantic difference priors and guiding the discrimination network to locate anomalies. See the overall framework diagram of this invention. Figure 1 .

[0065] Specifically as follows:

[0066] The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance includes the following steps:

[0067] S1: CLIP semantic-guided region constraint anomaly generation. Input a normal workpiece image, extract the region prior mask of the workpiece body and key structural regions through CLIP cross-modal semantic matching, constrain the simulated anomaly to the target region corresponding to the region prior mask, and generate a training image with synthetic anomaly and the corresponding anomaly ground truth mask.

[0068] S2: Abnormal Image Reconstruction. The training image with synthetic anomalous features is input into the reconstruction subnetwork, and the output is a reconstructed image that restores the normal appearance.

[0069] S3: Global-local semantic difference prior calculation for reconstructed perception. The training image with synthesis anomaly and the reconstructed image are respectively input into the frozen CLIP visual encoder, and the global semantic difference score and local semantic difference prior map of the two are calculated in the CLIP cross-modal semantic space.

[0070] S4: Semantic-guided anomaly detection and localization, inputting the training image with synthetic anomalies and the reconstructed image into the discriminative subnetwork, performing semantic gating fusion of the local semantic difference prior map and the intermediate features of the discriminative subnetwork, and outputting a pixel-level anomaly prediction map based on the fused features;

[0071] S5: Multi-objective joint optimization training, constructing a total loss function that includes reconstruction loss, segmentation loss, region constraint loss and semantic consistency loss, and performing end-to-end joint training on the reconstruction subnetwork and the discriminative subnetwork;

[0072] S6: Workpiece surface anomaly detection. Input the image of the workpiece to be detected into the trained network, and output the image-level anomaly detection result and the pixel-level anomaly localization result.

[0073] The original DRAEM generates anomalous shapes using Perlin noise and blends them with external textures into normal images to obtain a large number of synthetic anomalous samples. This strategy is simple and general, but the generation of anomalous regions is mainly globally random and does not fully utilize the structural priors of the workpiece itself. Therefore, in specific industrial scenarios, noise may fall into background or unimportant areas, reducing the effectiveness of the simulated anomalous features. For the workpiece images studied in this invention, defects are more likely to appear in the main body of the workpiece and its key structural parts, such as the head and waist regions. Therefore, this invention introduces CLIP semantic guidance, utilizing its cross-modal representation capabilities to obtain priors of the main body of the workpiece and key regions, thereby achieving region-constrained anomalous generation. The model structure for semantically guided anomalous generation is as follows: Figure 2 As shown.

[0074] Let the input image be First, a coarse-grained semantic response map of the workpiece body is obtained by similarity matching between the CLIP image encoder and a predefined set of text prompts. The text prompts can be set to terms such as "industrial part," "main body of the workpiece," "head region of the workpiece," and "waist region of the workpiece." Since CLIP has limited fine segmentation capabilities on industrial grayscale images, this invention does not directly use its output as a precise mask. Instead, it uses it as a coarse semantic localization prior and combines it with image geometry to refine the workpiece body mask. .

[0075] Furthermore, based on the main body region, the main body is divided into a head region and a waist region according to the workpiece geometry. Let the minimum bounding box of the main body mask be... Its height and width are respectively denoted as And W, then the head and waist region can be defined as:

[0076]

[0077]

[0078] in , and These are normalized position parameters. Depending on the specific workpiece structure, the parameters can be taken as follows: =0.2、 =0.35、 =0.65. This allows us to obtain the mask for the workpiece head region. and waist area mask .

[0079] Following the approach of DRAEM, the noise generator first produces the original anomalous shape mask. .

[0080] Unlike the original method, this invention does not directly use it for texture blending, but instead first samples a region prior mask. ,in:

[0081]

[0082] and set ,and and Generally, a larger value is chosen to emphasize the head and waist areas. The final anomaly mask is defined as follows:

[0083]

[0084] Obtaining a region-restricted anomaly mask Subsequently, this invention still employs a texture blending method similar to DRAEM to generate samples with anomalous features. Let the external anomalous texture source image be... The transparency coefficient is Then, the training image with anomalies can be represented as:

[0085]

[0086] By combining region constraints with texture blending, the method of this invention retains the advantages of DRAEM in simulating anomaly diversity, while making the synthesized anomalies closer to the spatial distribution characteristics of real workpiece defects.

[0087] To further suppress anomaly masks from falling into the background region and encourage anomalies to be distributed more in key areas such as the head and waist, this invention introduces a region constraint loss. First, a background suppression term is defined:

[0088]

[0089] Furthermore, define the key region bias term:

[0090]

[0091] The final region constraint loss is defined as:

[0092]

[0093] Global-local CLIP semantic difference prior modeling for reconstructing perception

[0094] The original DRAEM relies primarily on the visual differences between the input and reconstructed images to locate anomalous regions during the discrimination stage. This strategy is highly sensitive to local texture anomalies, but for high-level structural anomalies, local workpiece defects, and semantic inconsistencies, relying solely on pixel reconstruction differences is often insufficient. Therefore, this invention introduces CLIP in the discrimination stage and proposes a global-local semantic difference prior modeling method for reconstruction perception. Its core idea is: instead of directly using CLIP for static discrimination of the anomalous image, it simultaneously compares the anomalous input image. With reconstruction map The semantic response changes in CLIP across modal spaces are used to construct semantic anomaly priors that better conform to the characteristics of the DRAEM structure. The model structure diagram of the CLIP semantic difference prior modeling part is shown below. Figure 3 As shown.

[0095] First, the abnormal input graph With reconstruction map They are fed into the CLIP image encoder respectively. To obtain its global feature representation and Simultaneously construct a set of normal prompt words. With the set of abnormal prompt words Normal prompts describe a workpiece as intact and without defects, such as "a defect-free workpiece" or "an intact industrial component"; abnormal prompts describe defects or abnormal states, such as "a damaged workpiece" or "a defective industrial component". The maximum similarity between the input image and the normal / abnormal prompts is calculated for both the input and reconstructed images, and the following definitions are established:

[0096]

[0097]

[0098] The global semantic difference score for reconstructed perception is defined as:

[0099]

[0100] This quantity describes the change in the degree of semantic anomaly in an image before and after reconstruction. When the input image contains obvious anomalies and the reconstructed image restores the image to a closer approximation of normality... It will obtain a larger value, so it can be used as an image-level anomaly semantic score to help determine the overall degree of anomaly.

[0101] To further construct local-level semantic guidance, this invention extracts patch tokens from the CLIP's visual Transformer, denoted as follows: and ,in This indicates the number of patches. Further, define the exception-normal semantic difference for each patch:

[0102]

[0103]

[0104] The local semantic difference prior graph can then be represented as:

[0105]

[0106] in (i) represents the change in the semantic anomaly level of the i-th local region before and after reconstruction. After upsampling the local semantic difference map to the original image size, an anomaly semantic prior map with the same resolution as the input image can be obtained. Unlike directly performing CLIP semantic matching on the anomaly map, the method of this invention effectively strengthens the dynamic relationship of "anomaly response - normal recovery" by introducing the reconstructed map as a semantic reference, making the local semantic prior more consistent with the reconstruction-based detection mechanism of DRAEM.

[0107] CLIP-guided discriminative branch anomaly localization

[0108] After obtaining the local semantic difference prior map Subsequently, this invention introduces it into the discriminative branch of DRAEM. Let the intermediate features in the DRAEM discriminative network be represented as... ,Will After upsampling, it is recorded as Then the fusion feature is defined as:

[0109]

[0110] Here, α is a weight parameter controlling the strength of semantic prior. This fusion method is essentially a semantic gating mechanism: when certain regions exhibit stronger anomalous semantic differences in the CLIP space, the discriminative branch will receive higher attention weights at these locations; conversely, when certain regions have weaker semantic anomalous responses, the model relies more on the original visual differences for judgment. Ultimately, the discriminative network is based on fused features. Output pixel-level anomaly prediction map In this way, CLIP prior does not directly replace DRAEM's visual discrimination ability, but rather serves as a high-level semantic supplement to the fine-grained localization process.

[0111] Joint optimization objective

[0112] To achieve unified learning of anomaly generation, reconstruction and recovery, and anomaly localization, this invention employs the following total loss function:

[0113]

[0114] in, The reconstruction loss is used to constrain the reconstruction graph. As close as possible to the original normal image ; The segmentation loss is used to supervise the final anomaly prediction graph. With synthetic anomaly mask Consistency between them; This is a region constraint loss, used to ensure that the synthesized anomalies mainly fall within the main body of the workpiece, especially in key areas such as the head and waist. The semantic consistency loss is used to constrain the final prediction result to maintain consistency with the CLIP local semantic prior in the overall trend.

[0115] This invention defines the semantic consistency loss as:

[0116]

[0117] Based on the above embodiments, the following specific experiments are proposed:

[0118] Experimental setup

[0119] Dataset:

[0120] To comprehensively evaluate the effectiveness of the proposed method in industrial image anomaly detection and localization tasks, this invention conducted extensive experimental validation on three industrial anomaly detection datasets. These datasets cover different industrial manufacturing scenarios and object features, specifically including the newly collected and constructed inductor copper sheet dataset (named Copper-AD), and two widely used public benchmark datasets in the field of anomaly detection: MVTec-AD and VisA. All datasets used for evaluation were equipped with image-level normal or anomaly labels and fine-grained pixel-level masks to support rigorous evaluation of the model on detection and localization tasks. It is worth emphasizing that although MVTec-AD and VisA have greatly promoted the development of unsupervised anomaly detection, they are mainly constructed in highly controlled laboratory environments, and the defects are mostly artificially manufactured. In contrast, the Copper-AD dataset proposed in this invention is directly derived from a real-world inductor component production line. This dataset contains a total of 479 images, with the training set consisting of only 190 defect-free normal images to meet the actual needs of unsupervised anomaly detection, while the test set contains 81 normal images and 208 anomaly images with naturally occurring real manufacturing defects. To provide a reliable evaluation benchmark, the team manually performed detailed pixel-level annotations on all 208 anomaly images in the test set. This dataset, which natively contains real-world workshop environmental interference and natural industrial defects, closely matches the actual quality inspection environment and business needs of industrial production lines, and can more realistically reflect the robustness of anomaly detection algorithms in actual industrial deployments. Furthermore, this invention selected the classic benchmark MVTec-AD, containing 5354 images across 15 different categories, to verify the model's basic performance in typical industrial scenarios. It also introduced the highly challenging VisA dataset, containing 10821 images across 12 categories, leveraging its complex internal structure and multiple instance scenarios to further verify the generalization ability and versatility of the proposed method when facing complex semantic features and structural interference. The relevant information about the datasets used in the experiments is as follows:

[0121] Table 1. Statistical information of the dataset used in the experiment.

[0122]

[0123] Evaluation indicators:

[0124] To comprehensively evaluate the model's anomaly detection and localization performance in industrial scenarios, this invention employs AUC, AP, and F1-MAX as integrated evaluation metrics at both the image-level (detection) and pixel-level (localization) levels, and introduces a pixel-specific key localization metric, P-AUPRO. AUC comprehensively reflects the model's overall classification and segmentation performance, AP is more sensitive to extreme class imbalances in industrial production lines, and F1-MAX indicates the optimal trade-off at the best operating point, possessing significant engineering guidance value. Specifically, for the numerous subtle local defects contained in the Copper-AD dataset, the pixel-specific metric P-AUPRO treats all connected anomaly regions as equally important. This metric effectively avoids the drawback of conventional evaluations being dominated by large-area defects, strongly demonstrating the model's robustness in locating minute defects. This invention comprehensively verifies the superiority and industrial application potential of the proposed method through multiple metrics.

[0125] Implementation details:

[0126] All experiments in this invention were implemented on the PyTorch deep learning framework. The hardware environment consisted of a workstation equipped with a single NVIDIA RTX 3090 GPU. During data preprocessing, to balance computational overhead and feature extraction accuracy, all input images were uniformly adjusted to 256 x 256 pixels. For model training, the pre-trained CLIP model used was the high-resolution ViT-L / 14@336. The network was jointly optimized end-to-end using the Adam optimizer, with an initial learning rate of 1e-4 and a weight decay of 1e-5. The batch size for training on all datasets was uniformly set to 8, and the total number of training epochs was set to 200. Furthermore, in optimizing the joint loss function, to balance the gradients of various tasks, the hyperparameters of each part of the loss were empirically set to the segmentation loss weights. Regional constraint loss weights and CLIP semantic consistency loss weights .

[0127] Experimental comparison:

[0128] To fully validate the effectiveness of the proposed method, this invention compares it with several representative unsupervised anomaly detection methods, including PaDiM and PatchCore based on statistical embedding, DRAEM, RD, and Dinomaly based on reconstruction or knowledge distillation, and FastFlow, EfficientAD, and SuperSimpleNet, which have become increasingly competitive in recent years. The experiments report both image-level detection metrics and pixel-level localization metrics. AUC, AP, and F1-MAX reflect the overall anomaly detection capability, while P-AUC, P-AP, P-F1, and P-AUPRO further measure the model's pixel-level localization quality of anomaly regions. Through a unified comparison on three datasets—Copper-AD, MVTec-AD, and VisA—the generalization and robustness of the method can be evaluated from three levels: real-world production line scenarios, classic industrial benchmarks, and complex multi-instance scenarios.

[0129] Table 2 Quantitative comparison with existing best methods

[0130]

[0131] On the Copper-AD dataset, the proposed method achieves state-of-the-art results in key metrics such as AP, F1-MAX, P-AP, P-F1, and P-AUPRO, demonstrating that the proposed CLIP semantic collaboration mechanism not only enhances image-level anomaly detection capabilities but also significantly improves pixel-level localization quality. Specifically, the improvements in P-AP and P-F1 indicate that the model can more accurately distinguish defect regions from normal backgrounds in real production lines; the advantage of P-AUPRO demonstrates that the proposed method has more stable region recall capabilities for small-area, irregular real defects. Compared to DRAEM, the proposed method retains the advantages of fine-grained segmentation in the reconstruction framework while reducing interference from irrelevant background noise during the training phase through region-constrained anomaly generation, and enhances the response of the discriminative branch to real defect structures by leveraging reconstruction-aware semantic differential.

[0132] MVTec-AD dataset.

[0133] Table 3 Quantitative Comparison Results of MVTec-AD Datasets

[0134]

[0135] On the MVTec-AD dataset, the method of this invention achieves state-of-the-art or competitively state-of-the-art performance in AUC, F1-MAX, P-AUC, P-AP, and P-AUPRO, indicating that the method is not limited to self-built datasets but can be transferred to a wider range of industrial objects and texture categories. It is worth noting that Dinomaly also performs strongly in AP and some pixel-level metrics, which is related to its reconstruction modeling ability based on strong visual representations; however, the method of this invention maintains its advantage in localization-related P-AUC, P-AP, and P-AUPRO, demonstrating that the combination of CLIP semantic prior and DRAEM discriminative branch can form a more stable anomalous response under complex material, multi-scale defect, and texture interference conditions. This result further proves that semantic guidance does not weaken the fine-grained capability of the original reconstruction and segmentation framework, but rather provides it with a more structure-aware discriminative basis.

[0136] Table 4 provides a quantitative comparison of the VisA dataset.

[0137]

[0138] On the VisA dataset, the proposed method achieves state-of-the-art results in AUC, AP, F1-MAX, P-AUC, P-F1, and P-AUPRO, while maintaining performance close to the best method in P-AP. The VisA dataset contains multi-instance targets, densely packed structures, and fine-grained anomalies, demanding higher levels of background suppression and local anomaly perception capabilities from the model. In this scenario, the proposed method still achieves stable improvements, indicating that region-constrained anomaly generation helps the model learn anomaly patterns that better match the target structure distribution, while the global-local CLIP semantic difference prior can supplement semantic changes that are difficult to capture by traditional visual differences during the discrimination stage. The combined results from the three datasets show that the proposed method exhibits good robustness in real-world production data, classic public benchmarks, and complex multi-instance scenarios, validating the rationality of the proposed collaborative design of front-end anomaly generation constraints and back-end semantic difference discrimination.

[0139] Visualization results

[0140] To avoid insufficient interpretation of results due to reliance solely on numerical indicators, this invention further visualizes and compares different methods from two levels: anomaly heatmaps and binarized prediction masks. The visualization results uniformly display the original image, ground truth annotations, and heatmaps and segmentation masks of each method. The comparison objects include the method of this invention, representative methods such as Dinomaly, DRAEM, and PatchCore, thereby providing a more intuitive analysis of the model's localization accuracy, background suppression ability, and response stability to minor defects.

[0141] Copper-AD dataset (Ours)

[0142] Figure 4 The visualization results on the Copper-AD dataset are presented. It can be seen that the method of this invention can generate a more concentrated response in the heatmap that is consistent with the real defect region, and the background noise in the binarized mask is also significantly less. In contrast, Dinomaly is more sensitive to some background textures and edge regions, and although DRAEM can locate some anomalous regions, it exhibits an overly strong response on irrelevant backgrounds. These results demonstrate that the CLIP semantic-guided anomaly generation region constraint and the reconstruction-aware semantic difference prior can enable the model to learn a discrimination boundary that is closer to the real defect distribution, thereby improving the localization stability.

[0143] MVTec-AD dataset

[0144] Figure 5 Visual comparison results on the MVTec-AD dataset are presented. This set of samples covers different materials and defect morphologies, including capsules, fabrics, leather, and pills, and can test the model's generalization ability on conventional industrial benchmarks. It can be observed that the high-response regions of the proposed method are generally closer to the actual anomaly locations, and the binarization results show high consistency with the GT mask. While Dinomaly can capture some major defects, its response boundaries are not compact enough in complex texture scenes. DRAEM and PatchCore are more likely to generate diffuse responses on the background texture or the overall structure of the target. These results corroborate the quantitative indicators on the MVTec-AD dataset, demonstrating that the proposed method can maintain relatively stable anomaly localization capabilities across different materials and defect scales.

[0145] VisA dataset

[0146] Figure 6 The visualization results on the VisA dataset are presented. The VisA dataset contains multi-instance targets, complex backgrounds, and fine-grained small defects, placing higher demands on the model's local perception capabilities and false detection suppression abilities. As shown in the figures, the method of this invention can generate a more concentrated heatmap response in small-scale anomaly regions, and the predicted mask is more consistent with the anomaly region marked by the red box. In contrast, DRAEM and PatchCore are prone to large-area false detections in multi-instance targets or regular texture regions. Dinomaly can capture major defects on some samples, but its response to extremely small anomalies still suffers from omissions or boundary shifts. These results demonstrate that the method of this invention is not only applicable to a single workpiece dataset, but also exhibits good anomaly localization generalization in more complex multi-class industrial scenarios.

[0147] Overall, the three sets of visualization results further validated the effectiveness of the proposed method from different data distributions and defect morphologies: on the one hand, the region-constrained anomaly generation makes the model focus more on the main body of the workpiece and key structural regions during the training phase; on the other hand, the CLIP semantic difference prior of reconstructed perception can suppress background noise and enhance the response to real anomalies during the discrimination phase. Therefore, the proposed method shows a more reasonable experimental trend in terms of heatmap interpretability, mask integrity, and cross-dataset stability.

[0148] ablation experiment

[0149] To further analyze the contribution of each component module to the final performance, this invention conducts ablation experiments on the Copper-AD dataset. This dataset contains subtle defects naturally generated in a real-world factory environment, allowing for a more direct examination of the impact of different modules on the localization of real-world industrial anomalies. Using the original DRAEM framework as a baseline, this invention sequentially introduces CLIP reconstruction-aware semantic constraints and CLIP main region constraint anomaly generation strategies, and compares them with the complete model. The experimental results are shown in Table 5.

[0150] Table 5 shows the results of the ablation studies of the proposed components.

[0151]

[0152] The ablation results show that by introducing only CLIP reconstruction-aware semantic constraints, the model improves upon the baseline in AUC, AP, F1-MAX, and P-AUPRO metrics. This indicates that the difference in semantic response before and after reconstruction provides effective high-level anomaly priors for the discrimination branch, compensating for the insufficient perception of structural anomalies when relying solely on pixel differences. Further addition of CLIP main body region constraints results in even more significant improvements in P-AP and P-F1, indicating that limiting the synthesized anomalies to the main body of the workpiece and key structural regions helps reduce pseudo-supervision caused by background noise, enabling the model to learn anomaly discrimination boundaries that more closely resemble the actual defect distribution. The complete model achieves optimal results across all major metrics, demonstrating a complementary relationship between front-end region-constrained anomaly generation and back-end reconstruction-aware semantic differential modeling: the former enhances the spatial rationality of training samples, while the latter strengthens the semantic interpretability and localization stability of the discrimination stage. Therefore, the ablation experiments verify at the module level that the two designs of this invention are not simply superimposed, but rather form an effective synergy between anomaly generation, reconstruction recovery, and discrimination localization.

[0153] In summary:

[0154] This invention addresses the shortcomings of the DRAEM framework, namely the lack of target region constraints in the anomaly synthesis stage and the lack of high-level semantic awareness in the anomaly discrimination stage. The method utilizes CLIP semantic priors at the front end to constrain the anomaly generation location, making the synthesized anomalies more closely aligned with the workpiece body and key structural regions. At the back end, it provides supplementary semantic anomaly priors for the discrimination branch through global-local semantic difference modeling based on reconstruction perception. Quantitative results, visualization results, and ablation experiments on the Copper-AD, MVTec-AD, and VisA datasets collectively demonstrate that the proposed method can improve the overall discrimination capability, pixel-level localization accuracy, and cross-dataset stability of anomaly detection. Future work will further explore finer-grained region cue construction methods and investigate lightweight deployment and online updating capabilities in more complex real-world industrial production lines.

[0155] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance, characterized in that: Includes the following steps: S1: CLIP semantic-guided region constraint anomaly generation. Input a normal workpiece image, extract the region prior mask of the workpiece body and key structural regions through CLIP cross-modal semantic matching, constrain the simulated anomaly to the target region corresponding to the region prior mask, and generate a training image with synthetic anomaly and the corresponding anomaly ground truth mask. S2: Abnormal Image Reconstruction. The training image with synthetic anomalous features is input into the reconstruction subnetwork, and the output is a reconstructed image that restores the normal appearance. S3: Global-local semantic difference prior calculation for reconstructed perception. The training image with synthesis anomaly and the reconstructed image are respectively input into the frozen CLIP visual encoder, and the global semantic difference score and local semantic difference prior map of the two are calculated in the CLIP cross-modal semantic space. S4: Semantic-guided anomaly detection and localization, inputting the training image with synthetic anomalies and the reconstructed image into the discriminative subnetwork, performing semantic gating fusion of the local semantic difference prior map and the intermediate features of the discriminative subnetwork, and outputting a pixel-level anomaly prediction map based on the fused features; S5: Multi-objective joint optimization training, constructing a total loss function that includes reconstruction loss, segmentation loss, region constraint loss and semantic consistency loss, and performing end-to-end joint training on the reconstruction subnetwork and the discriminative subnetwork; S6: Workpiece surface anomaly detection. Input the image of the workpiece to be detected into the trained network, and output the image-level anomaly detection result and the pixel-level anomaly localization result.

2. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 1, characterized in that: The extraction process of the region prior mask in S1 is as follows: A set of pre-defined text prompts describing the main body, head region, and waist region of the workpiece is used to obtain a coarse-grained semantic response map of the main body of the workpiece through similarity matching between the CLIP image encoder and the text encoder. By combining the image geometry, the coarse-grained semantic response map is refined to obtain the workpiece body mask; Based on the minimum bounding box height of the workpiece body mask, the head region mask and waist region mask are divided from the workpiece body mask according to the preset normalized position parameters. The region prior mask includes the workpiece body mask, the head region mask, and the waist region mask.

3. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 2, characterized in that: The specific process of simulating anomalies in S1 is as follows: Generate the original anomalous shape mask using Perlin noise; The target region mask is obtained by sampling from the region prior mask according to the preset sampling probability. The original abnormal shape mask is multiplied pixel by pixel with the target region mask to obtain the region-restricted abnormal ground truth mask. Abnormal texture images are selected from an external texture library. The abnormal textures are then blended with normal workpiece images within the region corresponding to the abnormal ground truth mask using a transparency coefficient to generate a training image with synthetic anomalies.

4. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 2, characterized in that: The region constraint loss is composed of a weighted average of a background suppression term and a key region bias term: The background suppression term is the L1 norm of the intersection of the anomaly truth mask and the background region, used to suppress anomalies from falling into the background region outside the workpiece body. The critical region bias term is constructed based on the intersection ratio of the anomaly truth mask and the head and waist regions, and is used to guide anomalies to be preferentially distributed in the critical structural regions of the head and waist.

5. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 1, characterized in that: The calculation process of the global semantic difference score in S3 is as follows: Construct a set of normal prompt words describing the normal state of the workpiece and a set of abnormal prompt words describing the abnormal state of the workpiece, respectively. The maximum similarity between the global features of the training image and the reconstructed image with synthetic anomalies and the features of the two types of prompt words is calculated respectively to obtain the abnormal semantic score of the abnormal input image and the abnormal semantic score of the reconstructed image. The abnormal semantic scores of the two types of images are subtracted to obtain the global semantic difference score for reconstructed perception, which is used as the image-level abnormal semantic score.

6. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 5, characterized in that: The calculation process of the local semantic difference prior graph in S3 is as follows: Extract patchtoken features of training and reconstructed images with synthetic anomalies from the Transformer layer of the CLIP visual encoder; Calculate the maximum semantic difference between each patchtoken feature and the normal prompt word set and the abnormal prompt word set to obtain the local abnormal semantic response map of the abnormal input image and the reconstructed image respectively; The two local anomaly semantic response maps are subtracted position by position to obtain a local semantic difference map. After upsampling to the original image size, a local semantic difference prior map is obtained.

7. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 1, characterized in that: The semantic gating fusion in S4 specifically refers to: Upsample the local semantic difference prior map to the same size as the intermediate features of the discriminant subnetwork; by As the semantic attention gain coefficient, it is multiplied stepwise with the intermediate features of the discriminant sub-network to obtain the semantically enhanced fusion feature; among which, The weighting parameters are used to control the strength of semantic prior guidance.

8. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 1, characterized in that: The expression for the total loss function is: in, The reconstruction loss is used to constrain the pixel consistency between the reconstructed image and the original normal image. The segmentation loss is used to supervise the consistency between the anomaly prediction map and the anomaly truth mask; For regional constraint loss; , , The weighting coefficients; the semantic consistency loss The L1 distance between the Sigmoid-normalized anomaly prediction map and the normalized local semantic difference prior map is used to constrain the trend consistency between the prediction results and the CLIP semantic prior.

9. The DRAEM workpiece surface anomaly detection method based on CLIP semantic collaborative guidance according to claim 1, characterized in that: Specifically, S6 is: The image of the workpiece to be detected is input into the trained reconstruction sub-network to obtain the reconstruction image to be detected, and then input into the discriminant sub-network to obtain the pixel-level anomaly score map. At the same time, the global semantic difference score between the image to be detected and the reconstruction image to be detected is calculated through CLIP. Pixel-level anomaly localization results are obtained based on pixel-level anomaly score maps, and image-level anomaly detection results are obtained by combining the statistical values ​​of pixel-level anomaly score maps with global semantic difference scores.

Images