A Cross-Modal Industrial Anomaly Detection Method Based on Marginal Comparison Learning

Abstract

This invention discloses a cross-modal industrial anomaly detection method based on marginal contrastive learning. The method includes: establishing a cross-modal alignment relationship between industrial product images and semantic descriptions, endowing the model with defect semantic understanding capabilities; employing a marginal contrastive loss function based on cosine distance to construct explicit normal / abnormal decision boundaries in the feature space, improving the model's robustness to boundary samples. Simultaneously, it introduces joint training of reconstruction branches and contrastive learning, enabling the model to retain pixel-level details while achieving cross-modal semantic alignment, avoiding feature collapse, and ensuring sensitivity to fine-grained defects such as minor scratches and blemishes. This invention employs a two-stage training strategy: in the modal alignment stage, the encoder is frozen and only the projection layer is trained, preserving the pre-trained model's feature extraction capabilities; in the fine-tuning stage, the top layer of the encoder is unfrozen and domain adaptation is completed on industrial data, allowing the model to quickly adapt to new product detection requirements using zero-sample or few-sample methods.

Description

Technical Field

[0001] This invention belongs to the field of industrial visual inspection technology, specifically relating to a cross-modal industrial anomaly detection method based on marginal contrastive learning. Background Technology

[0002] Vision-based industrial anomaly detection, which analyzes images of industrial products to detect and locate anomalies and defects, is an important research area in industrial quality inspection and automation. Currently, industrial anomaly detection methods are mainly divided into two categories:

[0003] The first category is based on traditional deep neural network methods, including convolutional autoencoders, generative adversarial networks, and knowledge distillation networks. These methods enable the model to model the feature distribution of normal samples by learning from a large number of normal samples, and detect anomalies by quantifying the distance between test samples and the normal feature distribution. However, these methods have the following limitations: (1) they require a large number of normal samples to be collected for training for each product; (2) when the product type or production process changes, the model must be retrained; and (3) they lack the ability to understand the semantics of defects and have difficulty distinguishing different types of defects.

[0004] The second category is methods based on visual language models, such as WinCLIP and AnomalyCLIP. With the development of visual language models like CLIP, these models have demonstrated strong zero-shot generalization and cross-modal understanding capabilities in many fields, allowing for zero-shot detection of different products without the need to collect large amounts of samples and retrain the model for each new product. However, existing visual language model-based industrial anomaly detection methods still have the following problems:

[0005] First, existing visual-language pre-trained models lack generalization ability in industrial scenarios. On the one hand, these models are mainly pre-trained on general natural images and online text, and the feature distribution learned by their visual encoders differs significantly from that of industrial product images, making direct transfer to industrial inspection tasks difficult. On the other hand, when product type, production process, or inspection requirements change, the models need to collect a large number of samples and retrain to adapt to the new scenario, lacking the ability to quickly adapt to new products. Furthermore, when using visual and text encoders from different sources, their feature spaces are not aligned, making effective cross-modal matching impossible. Therefore, how to establish alignment between visual and text modalities while maintaining model generalization ability, and enable the model to quickly adapt to the inspection requirements of different industrial products, is a problem that urgently needs to be solved.

[0006] Secondly, existing visual language models based on contrastive learning tend to learn coarse-grained semantic alignment, which may lose fine-grained local details during the optimization process, resulting in a "feature collapse" phenomenon. This leads to insufficient sensitivity of the models to pixel-level defects such as tiny scratches and minor stains commonly found in industrial products, making it impossible to effectively detect these fine-grained anomalies.

[0007] Furthermore, existing methods lack explicit normal / abnormal decision boundaries when performing anomaly detection. Contrastive learning is trained only by bringing positive sample pairs closer together and pushing negative sample pairs further away, but it does not explicitly specify where normal and anomalous samples should be distributed in the feature space. This leads to unstable model decisions and a tendency to produce misclassifications when test samples are located in the boundary region between normal and anomalous samples.

[0008] In summary, existing visual language model-based industrial anomaly detection methods suffer from several problems, including insufficient generalization ability to adapt to new products, feature collapse leading to weak fine-grained defect detection capabilities, and the lack of explicit decision boundaries resulting in unstable boundary sample discrimination. Therefore, a cross-modal industrial anomaly detection method that can effectively address these issues is needed. Summary of the Invention

[0009] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide a cross-modal industrial anomaly detection method based on marginal contrastive learning. By establishing cross-modal alignment between images and semantics, the model acquires defect semantic understanding capabilities. It employs a marginal contrastive loss based on cosine distance, setting normal sample boundaries α and anomaly sample boundaries β to construct explicit decision boundaries in the feature space, thereby improving the robustness of boundary samples. Furthermore, it introduces joint training of reconstruction branches and contrastive learning to preserve pixel-level details, prevent feature collapse, and enhance sensitivity to fine-grained defects. This invention adopts a two-stage training strategy: in the modal alignment stage, the encoder is frozen, and only the projection layer is trained, preserving the pre-trained feature extraction capability; in the fine-tuning stage, the top layer of the encoder is unfrozen to complete domain adaptation, enabling the model to quickly adapt to new product detection with zero or few samples.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] In a first aspect, the present invention provides a cross-modal industrial anomaly detection method based on marginal contrastive learning, comprising the following steps:

[0012] S1. Construct an industrial anomaly detection dataset, use a multimodal large model to generate text descriptions for industrial product images, the text descriptions include normal product descriptions and abnormal product descriptions, and use a synthetic defect generation method to expand the abnormal samples;

[0013] S2. Modal alignment training is performed using a hybrid dataset, which includes normal sample images from a general image and text dataset and an industrial anomaly detection dataset. Under the condition that the parameters of the visual encoder and text encoder are completely frozen, only the visual projection layer and the text projection layer are trained to establish the cross-modal alignment relationship between visual features and text features.

[0014] S3. Fine-tuning is performed on the industrial anomaly dataset using the marginal contrast loss function. By setting the distance boundary α for normal samples and the distance boundary β for anomaly samples, an explicit normal / anomaly decision boundary is constructed in the feature space. At the same time, reconstruction loss and hard negative sample mining loss are introduced for joint training.

[0015] S4. Use the trained cross-modal model to detect anomalies in industrial products. Calculate the cosine distance between the visual features of the input image and the features of the normal descriptive text, and map the cosine distance to an anomaly score to complete the anomaly determination.

[0016] As a preferred technical solution, in step S1, the normal product description includes product appearance features, material features, color features and surface texture features;

[0017] The description of the abnormal product includes visual characteristics such as scratches, stains, damage, deformation, and color difference defects.

[0018] The synthetic defect generation adopts an image cutting and pasting method, which randomly cuts out image blocks from normal images and pastes them to the target position to simulate industrial surface patch anomalies.

[0019] As a preferred technical solution, in step S2, the general image and text dataset is used to establish basic cross-modal alignment capabilities, enabling the output features of the visual encoder and the text encoder to be effectively matched in a shared space.

[0020] The normal sample images are used to adapt the model to the visual feature distribution of industrial images.

[0021] As a preferred technical solution, in step S2, the modal alignment training specifically includes:

[0022] A complete freeze strategy is adopted, where all Transformer layer parameters of the visual encoder and text encoder are kept frozen, and only the visual projection layer and text projection layer are trained.

[0023] As a preferred technical solution, in step S2, the loss function of the modal alignment training phase includes contrast loss and reconstruction loss, and the formula is:

[0024] ;

[0025] in To compare the loss, cross-modal alignment is established by bringing paired visual-text features closer together and pushing unpaired features further apart; To mitigate the reconstruction loss, the model is forced to preserve image detail during the alignment process; The reconstruction loss weights, ranging from 0.1 to 1.0, are used to balance the contributions of the two loss terms.

[0026] As a preferred technical solution, in step S3, the marginal contrast loss function is implemented as follows:

[0027] Given visual features Corresponding text features cosine distance The total loss function during the fine-tuning phase is defined as:

[0028] ,

[0029] in Marginal contrast loss, positive sample loss Encourage the feature distance of normal samples to be less than the margin. Negative sample loss Encourage the feature distance of outlier samples to be greater than the margin. ,in , forming a width of The safety margin; the safety margin enhances the model's robustness to boundary samples, ensuring that samples in the boundary region can also be correctly classified; wherein To reconstruct the loss weights, Weights are assigned to the loss of difficult negative samples.

[0030] As a preferred technical solution, step S3 also includes a hard negative sample mining strategy:

[0031] For the visual features of each normal sample, find the text features of other normal samples that are most similar to it within the batch, force the model to distinguish these similar samples through InfoNCE loss, and use a lower temperature coefficient to make the model pay more attention to difficult samples, thereby refining the decision boundary.

[0032] As a preferred technical solution, the reconstruction loss in step S3 is implemented as follows:

[0033] A lightweight deconvolutional decoder is introduced, which consists of multiple transposed convolutional layers and BatchNorm layers, progressively upsampling the latent features to restore them to the original image size; the reconstruction loss is expressed as mean squared error. ,in The original input image, To reconstruct the image, the reconstruction decoder consists of multiple transposed convolutional layers and BatchNorm layers, which upsample the latent features to restore them to the original image size.

[0034] As a preferred technical solution, step S3 also employs a top-level unfreezing strategy in the fine-tuning stage:

[0035] During the fine-tuning phase, the top 2-4 Transformer modules of the visual encoder and the top 1-2 Transformer modules of the text encoder are unfrozen, while the underlying parameters remain frozen to preserve the general features learned on large-scale pre-trained data.

[0036] As a preferred technical solution, in step S4, the anomaly score is obtained by normalizing the cosine distance between the visual features and the normal descriptive text features. The larger the distance, the higher the anomaly score. The normal / abnormal binary classification is completed by setting a threshold.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] This invention provides a cross-modal industrial anomaly detection method based on marginal contrastive learning. First, it proposes a marginal contrastive loss function based on cosine distance. Through explicit margin constraints, it establishes clear normal / abnormal decision boundaries in the feature space, enhancing robustness to boundary samples compared to traditional contrastive learning methods. This enables the model to make stable and reliable judgments when facing samples in the normal / abnormal boundary region. Second, it proposes a training framework combining contrastive learning and reconstruction learning, effectively preventing feature collapse during the contrastive learning process. Combined with a multi-scale feature fusion mechanism, it significantly improves the model's detection capability for fine-grained defects commonly found in industrial products, such as minor scratches and blemishes. Finally, it proposes a two-stage training strategy: in the modal alignment stage, the encoder is completely frozen on a mixed dataset, and only the projection layer is trained to establish basic alignment capabilities; in the fine-tuning stage, a top-level unfreezing strategy is used to adapt the model to industrial data, effectively balancing the model's generalization ability and domain adaptability, enabling the model to adapt to the detection needs of new products with zero or few samples.

[0039] Compared with traditional industrial anomaly detection methods based on deep neural networks, this invention has the following advantages: First, by establishing the association between visual features and semantic descriptions through cross-modal learning, the model possesses the ability to understand the semantics of defects, distinguish different types of defects, and understand their causes; Second, based on the zero-shot / few-shot learning capability of the pre-trained visual language model, it eliminates the need to collect a large amount of training data for each new product, significantly improving the model's generalization ability and cross-domain transfer ability across different products; Third, by guiding detection through text descriptions, users can describe new defect types using natural language, allowing the model to quickly adapt to new detection requirements.

[0040] Compared with existing anomaly detection methods based on visual language models, this invention has the following advantages: First, it proposes a marginal contrast loss function based on cosine distance, by setting a boundary for normal samples. and the boundary of abnormal samples First, by establishing explicit normal / abnormal decision boundaries in the feature space, compared to the implicit approach of traditional contrastive learning that simply brings positive sample pairs closer and pushes negative sample pairs further away, the decision boundaries of this invention are clearer and more stable, and the judgment of samples in the boundary region is more reliable. Second, by introducing reconstruction branches and joint training with contrastive learning, the feature collapse phenomenon in the contrastive learning process is effectively prevented, and the model's sensitivity to fine-grained defects such as minor scratches and blemishes commonly found in industrial products is maintained. Third, by combining a multi-scale feature fusion mechanism and utilizing both shallow texture features and deep semantic features, the ability to identify large-scale structural anomalies is maintained while detecting fine-grained defects.

[0041] In summary, the cross-modal industrial anomaly detection method based on marginal contrastive learning provided by this invention not only improves the accuracy and robustness of industrial anomaly detection, but also enhances the model's sensitivity to minor defects and its adaptability to new products, thus having significant practical application value for quality inspection in industries such as discrete manufacturing. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of the cross-modal industrial anomaly detection method based on marginal contrastive learning in this embodiment.

[0044] Figure 2 This is an architecture diagram of the modality alignment stage in this embodiment, showing the module composition and data flow of the modality alignment stage, including a frozen visual encoder, a frozen text encoder, a trainable projection layer, a shared latent space, a contrastive learning module, and a visual reconstruction module.

[0045] Figure 3 This is an architecture diagram of the industrial data fine-tuning stage in this embodiment, showing the module composition and data flow of the fine-tuning stage, including a partially thawed visual encoder, a partially thawed text encoder, a multi-scale feature fusion module, a projection layer, a marginal contrast loss module, a reconstruction branch, and an anomaly score calculation module.

[0046] Figure 4This is a schematic diagram of the marginal contrast loss function in this embodiment, showing the boundary of normal samples in the feature space. Abnormal sample boundaries And the principles behind setting safety margins.

[0047] Figure 5 This is a flowchart of the weak category adaptive weighting mechanism in this embodiment, illustrating the dynamic adjustment process of category weights based on validation set performance. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0049] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0050] like Figures 1-3 As shown in the figure, this embodiment presents a cross-modal industrial anomaly detection method based on marginal contrastive learning, employing a visual encoder, a text encoder, a multi-scale feature fusion module, a visual projection layer, a text projection layer, a reconstruction decoder, and an anomaly scoring module. The visual encoder uses a pre-trained visual Transformer model based on the ViT architecture, the text encoder uses a pre-trained large language model, the visual projection layer and the text projection layer map features from different modalities to a shared latent space, and the reconstruction decoder restores the latent features to the original image to prevent feature collapse.

[0051] Furthermore, the visual encoder uses SigLIP-so400m-patch14-384, and the text encoder uses Qwen2.5-3B-Instruct. For the visual encoder, SigLIP uses the Sigmoid loss function instead of the traditional CLIP Softmax contrastive loss for pre-training, resulting in better performance on fine-grained visual recognition tasks and making it more suitable for scenarios requiring the perception of minute differences, such as surface defect detection in industrial products. The selected so400m-patch14-384 version uses an input resolution of 384×384 and a patch size of 14 pixels, providing sufficiently fine spatial resolution to capture pixel-level defects such as tiny scratches and minor stains. In addition, SigLIP is based on the ViT architecture, which has a clear hierarchical feature structure, facilitating the extraction of multi-layer features for multi-scale fusion. For text encoders, Qwen2.5-3B possesses stronger semantic understanding capabilities compared to traditional CLIP text encoders. It can accurately understand industrial terms such as "minor scratches," "surface oxidation," and "edge burrs," as well as diverse expressions in the Chinese context, providing high-quality semantic feature representations for cross-modal alignment. Furthermore, Qwen2.5 natively supports bilingual processing in Chinese and English, making it suitable for deployment needs in domestic industrial scenarios. This invention employs a combination of heterogeneous encoders from different pre-training tasks. Although their feature spaces are initially unaligned, through the modal alignment training proposed in this invention, an effective cross-modal correspondence can be established while preserving the advantageous features of each encoder, providing a richer feature representation space for the model compared to using homogeneous encoders.

[0052] like Figure 1 As shown in the figure, this embodiment provides a cross-modal industrial anomaly detection method based on marginal contrastive learning, which specifically includes the following steps:

[0053] S1. Construct an industrial anomaly detection dataset, use a multimodal large model to generate text descriptions for industrial product images, including descriptions of normal products and descriptions of abnormal products, and use a synthetic defect generation method to expand the anomaly samples.

[0054] Furthermore, the specific implementation methods for constructing the industrial anomaly detection dataset in step 1 include:

[0055] S11. Utilize multimodal large model analysis to generate detailed descriptions of normal and abnormal products. For normal product images, the generated description includes the product's appearance, material characteristics, color characteristics, and surface texture. For abnormal product images, the generated description includes visual characteristics of common defect types such as scratches, stains, damage, deformation, and color differences. The text description provides semantic supervision signals for cross-modal alignment, establishing a correspondence between visual features and semantic descriptions.

[0056] Furthermore, text description generation also includes the following augmentation methods:

[0057] S111. Product type expansion: Expand the prototype description to different variations of the same type of product;

[0058] S112. Abnormal situation expansion: The abnormal situation description is expanded to include attributes such as defect degree, location and size. S113. Synonym augmentation: Generates multiple synonyms for each description. The technical effect of text description augmentation is to enhance the model's robustness to different representations.

[0059] S12. To address the scarcity of anomalous samples in industrial scenarios, an image segmentation and pasting method is used to generate synthetic anomalous samples. Specifically, a cropped region is randomly selected from a normal image, with the area of ​​the cropped region accounting for 5% to 15% of the original image area. The cropped shape can be rectangular or an irregular polygon. The cropped region is then subjected to random transformations, including random rotation, random scaling, and color dithering. The transformed image patch is pasted onto random locations within the same image or other images to simulate common surface patch-like anomalies in industrial scenarios. Simultaneously, a corresponding anomalous text description is generated for each synthetic anomalous sample. The technical advantage of synthetic anomalous samples is that while maintaining a visual feature distribution similar to real defects, it can expand the number of anomalous samples, solving the problem of difficulty in collecting real anomalous samples.

[0060] S2. Modal alignment training is performed using a hybrid dataset. Only the projection layer is trained under the condition of completely freezing the encoder to establish the alignment relationship between visual and textual modalities.

[0061] Furthermore, such as Figure 2 , Figure 3 As shown, the specific implementation of modality alignment training in step S2 includes:

[0062] S21, the mixed dataset consists of two parts:

[0063] The first part is a general image-text dataset, which is used to establish basic cross-modal alignment capabilities, enabling the output features of the visual encoder and the text encoder to be effectively matched in a shared space;

[0064] The second part consists of normal samples from the industrial anomaly detection dataset paired with product description text generated in step S11, used to adapt the model to the visual feature distribution of industrial images. The design idea of ​​the hybrid dataset is to both leverage large-scale general data to establish the basic capability of cross-modal alignment and guide the model to focus on the specific visual features of industrial products through industrial data.

[0065] S22. In the modal alignment stage, a complete freeze strategy is adopted, and modal alignment training is performed using a general image and text dataset. Specifically, a visual encoder is used to extract image features, and a text encoder is used to extract text features. The extracted image and text features are then mapped to the same feature space through the visual projection layer and the text projection layer for training. The specific strategy is to keep all Transformer layer parameters of the visual encoder and the text encoder frozen, and only train the visual projection layer and the text projection layer.

[0066] The technical advantages of this strategy are: first, it preserves the powerful feature extraction capabilities learned by the pre-trained model on large-scale data; second, it avoids overfitting or forgetting pre-trained knowledge caused by training on relatively small industrial datasets; and third, it significantly reduces training costs and memory usage, making the training process more efficient.

[0067] S23. The loss function in the modal alignment stage includes contrast loss and reconstruction loss, as shown in equation (1):

[0068]

[0069] in To compare the loss, cross-modal alignment is established by bringing paired visual-text features closer together and pushing unpaired features further apart; To mitigate the reconstruction loss, the model is forced to preserve image detail during the alignment process; The reconstruction loss weights, ranging from 0.1 to 1.0, are used to balance the contributions of the two loss terms.

[0070] S3. Fine-tuning is performed on the industrial anomaly dataset using the marginal contrast loss function. By setting the distance boundary α for normal samples and the distance boundary β for anomaly samples, an explicit normal / anomaly decision boundary is constructed in the feature space. At the same time, reconstruction loss and hard negative sample mining loss are introduced for joint training.

[0071] Furthermore, such as Figure 3 and Figure 4 As shown, step S3 is the fine-tuning stage, where the top layer of the encoder is unfrozen for parameter optimization, using the marginal contrast loss function. The specific implementation includes:

[0072] S31. For each pair of visual features and text features The cosine distance is calculated as a similarity measure, as shown in equation (2):

[0073]

[0074] The cosine distance ranges from [0, 1]. The smaller the distance, the more similar the two vectors are, meaning the higher the matching degree between the image and the text description. The reason for choosing cosine distance instead of Euclidean distance is that cosine distance is not sensitive to the magnitude of the feature vectors, only considering the similarity of the vector directions, and is more suitable for measuring semantic similarity in high-dimensional feature spaces.

[0075] S32. Positive sample loss is used to constrain the feature distance of normal samples to be less than the margin. As shown in equation (3):

[0076]

[0077] in This represents the number of normal samples within the batch. For the first The cosine distance between the visual features of a normal sample and the corresponding normal descriptive text features. This is the threshold for the normal sample boundary. When the distance to the normal sample... Less than When the distance is zero, no further optimization is needed; when the distance is zero... Greater than At this time, a positive loss is generated, forcing the model to narrow the distance to that sample. Within. The value range is 0.1 to 0.3, with smaller values... The value implies that normal samples need to be highly aligned with the text description.

[0078] S33. Negative sample loss is used to constrain the feature distance of outlier samples to be greater than the margin. As shown in equation (4):

[0079]

[0080] in This represents the number of abnormal samples within the batch. For the first The cosine distance between the visual features of an anomalous sample and the features of the normal descriptive text. This is the threshold for the boundary of abnormal samples. When the distance to the abnormal sample... Greater than When the distance is zero, the loss is 0; when the distance is zero. Less than When this occurs, a positive loss is generated, forcing the model to push the sample away from the normal region. The value range is 0.5 to 0.8, with larger values... The value implies that outlier samples need to maintain a sufficient distance from normal descriptions.

[0081] S34. Marginal contrast loss is the weighted sum of positive sample loss and negative sample loss, as shown in equation (5):

[0082]

[0083] in This is the loss weight for negative samples, used to balance the contributions of positive and negative samples. It is set by... In the feature space, a width of The safety margin. The technical effect of this safety margin is that samples in the boundary region (at a distance of...) and The model is neither forced to move closer nor forced to move further away from the boundary samples, giving it a certain tolerance for errors and enhancing the robustness of decision-making.

[0084] S35. Perform hard negative sample mining within the batch, considering the visual features of each normal sample. The cosine similarity between the sample and the text features of all other normal samples in the batch is calculated, and the most similar sample is identified as the hard negative sample (i.e., the sample pair that is easily misclassified). The InfoNCE loss forces the model to distinguish these similar samples, as shown in Equation (6):

[0085]

[0086] Where τ is a temperature coefficient, a lower temperature value is used (e.g.) Setting the value to 0.07 makes the softmax distribution sharper, causing the model to focus more on difficult samples. The technical effect of difficult negative sample mining is that it forces the model to learn more refined feature representations, enabling the decision boundary to distinguish between products that look similar but belong to different categories.

[0087] Furthermore, step S3 introduces a reconstruction loss to prevent feature collapse, specifically implemented as follows:

[0088] S36. A lightweight deconvolutional decoder is used to restore the latent features to the original image. The decoder consists of 4 to 6 transposed convolutional layers, which are gradually upsampled to the original image size.

[0089] S37. Reconstruction loss is expressed in the form of mean square error: This forces the model to retain image detail information.

[0090] S38. In the fine-tuning stage, the marginal contrast loss, reconstruction loss, and hard negative sample loss are jointly optimized: The technical advantage of reconstruction loss lies in preventing feature collapse during contrastive learning and maintaining the model's sensitivity to fine-grained defects.

[0091] Step S3, the fine-tuning stage, also employs a top-level unfreezing strategy: unlike the modality alignment stage which completely freezes the encoder, the fine-tuning stage unfreezes the top 2-4 Transformer layers of the visual encoder and the top 1-2 Transformer layers of the text encoder, while keeping the bottom-level parameters frozen to preserve the general features learned on large-scale pre-trained data. The design philosophy behind this strategy is that the encoder's bottom layer learns task-independent general visual features (such as edges and textures), which have good transferability across different domains and should be kept frozen; the encoder's top layer learns task-related high-level semantic features, which need to be fine-tuned on industrial data to adapt to the specific needs of industrial anomaly detection tasks.

[0092] S4. Use the trained cross-modal model to detect anomalies in industrial products. Calculate the cosine distance between the visual features of the input image and the features of the normal descriptive text, and map the cosine distance to an anomaly score to complete the anomaly determination.

[0093] Furthermore, the specific implementation of anomaly scoring and detection in step 4 is as follows: input the image to be detected, and extract visual features through a visual encoder and a projection layer. Input normal descriptive text and extract text features. Calculate the cosine distance Mapped to anomaly scores, the greater the distance, the higher the degree of anomaly; through a threshold The scores are converted into binary classification results.

[0094] like Figure 5 As shown, this embodiment provides an implementation of a weak category adaptive weighting mechanism. For product categories with poor detection performance, a higher loss weight is assigned when calculating the marginal contrast loss. The specific implementation method is as follows:

[0095] After each training epoch, the detection performance of each product category on the validation set is calculated (e.g., For categories with below-average performance, their loss weights are increased in the next epoch, and the weight adjustment formula is shown in equation (7):

[0096]

[0097] in For category loss weights, Average of all categories , This is the adjustment coefficient. This mechanism makes the model pay more attention to difficult categories during training, thereby improving the model's detection performance on difficult categories.

[0098] This invention, through a two-stage training strategy and a marginal contrastive loss function, enables the application of visual language models to industrial anomaly detection while maintaining generalization ability and establishing explicit decision boundaries. By freezing the encoder and training only the projection layer during the modal alignment stage, the feature extraction capability of the pre-trained model is preserved, and cross-modal alignment relationships are established. The top-level unfreezing strategy during the fine-tuning stage allows the model to adapt to the specific distribution of industrial images. Simultaneously, by introducing a reconstruction branch and joint training with contrastive learning, the model is forced to retain pixel-level detail information while learning semantic alignment, making it sensitive to fine-grained defects such as minor scratches and blemishes. The marginal contrastive loss function sets the boundary for normal samples. and the boundary of abnormal samples By establishing explicit decision boundaries in the feature space, the model's stability in judging boundary samples is enhanced, thereby achieving better anomaly detection results.

[0099] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A cross-modal industrial anomaly detection method based on marginal contrastive learning, characterized in that, Includes the following steps: S1. Construct an industrial anomaly detection dataset, use a multimodal large model to generate text descriptions for industrial product images, the text descriptions include normal product descriptions and abnormal product descriptions, and use a synthetic defect generation method to expand the abnormal samples; S2. Modal alignment training is performed using a hybrid dataset, which includes normal sample images from a general image and text dataset and an industrial anomaly detection dataset. Under the condition that the parameters of the visual encoder and text encoder are completely frozen, only the visual projection layer and the text projection layer are trained to establish the cross-modal alignment relationship between visual features and text features. S3. Fine-tuning is performed on the industrial anomaly dataset using the marginal contrast loss function. By setting the distance boundary α for normal samples and the distance boundary β for anomaly samples, an explicit normal / anomaly decision boundary is constructed in the feature space. At the same time, reconstruction loss and hard negative sample mining loss are introduced for joint training. S4. Use the trained cross-modal model to detect anomalies in industrial products. Calculate the cosine distance between the visual features of the input image and the features of the normal descriptive text, and map the cosine distance to an anomaly score to complete the anomaly determination.

2. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S1, the normal product description includes product appearance features, material features, color features, and surface texture features; The description of the abnormal product includes visual characteristics such as scratches, stains, damage, deformation, and color difference defects. The synthetic defect generation adopts an image cutting and pasting method, which randomly cuts out image blocks from normal images and pastes them to the target position to simulate industrial surface patch anomalies.

3. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S2, the general image-text dataset is used to establish basic cross-modal alignment capabilities, enabling the output features of the visual encoder and the text encoder to be effectively matched in a shared space. The normal sample images are used to adapt the model to the visual feature distribution of industrial images.

4. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S2, the modal alignment training specifically includes: A complete freeze strategy is adopted, where all Transformer layer parameters of the visual encoder and text encoder are kept frozen, and only the visual projection layer and text projection layer are trained.

5. The cross-modal industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S2, the loss function for the modality alignment training phase includes contrastive loss and reconstruction loss, as shown in the formula: ； in To compare the loss, cross-modal alignment is established by bringing paired visual-text features closer together and pushing unpaired features further apart; To mitigate the reconstruction loss, the model is forced to preserve image detail during the alignment process; The reconstruction loss weights, ranging from 0.1 to 1.0, are used to balance the contributions of the two loss terms.

6. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S3, the marginal contrast loss function is implemented as follows: Given visual features Cosine distance to corresponding text features The total loss function of the fine-tuning stage is defined as:​ ， in Marginal contrast loss, positive sample loss Encourage the feature distance of normal samples to be less than the margin. Negative sample loss Encourage the feature distance of outlier samples to be greater than the margin. ,in , forming a width of The safety margin; the safety margin enhances the model's robustness to boundary samples, ensuring that samples in the boundary region can also be correctly classified; wherein To reconstruct the loss weights, Weights are assigned to the loss of difficult negative samples.

7. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, Step S3 also includes a hard negative sample mining strategy: For the visual features of each normal sample, find the text features of other normal samples that are most similar to it within the batch, force the model to distinguish these similar samples through InfoNCE loss, and use a lower temperature coefficient to make the model pay more attention to difficult samples, thereby refining the decision boundary.

8. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, The reconstruction loss in step S3 is implemented as follows: A lightweight deconvolutional decoder is introduced, which consists of multiple transposed convolutional layers and BatchNorm layers, progressively upsampling the latent features to restore them to the original image size; the reconstruction loss is expressed as mean squared error. ,in The original input image, To reconstruct the image, the reconstruction decoder consists of multiple transposed convolutional layers and BatchNorm layers, which upsample the latent features to restore them to the original image size.

9. A cross-modal industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S3, the fine-tuning phase also employs a top-level unfreezing strategy: During the fine-tuning phase, the top 2-4 Transformer modules of the visual encoder and the top 1-2 Transformer modules of the text encoder are unfrozen, while the underlying parameters remain frozen to preserve the general features learned on large-scale pre-trained data.

10. The cross-modality industrial anomaly detection method based on marginal contrastive learning according to claim 1, characterized in that, In step S4, the anomaly score is obtained by normalizing the cosine distance between the visual features and the normal descriptive text features. The larger the distance, the higher the anomaly score. The normal / anomaly binary classification is completed by setting a threshold.

Images