Text-guided zero-shot industrial defect detection method, equipment, and medium

Abstract

The present invention discloses a text-guided zero-shot industrial defect detection method, device, and medium, wherein the method comprises: obtaining images and text semantic labels of industrial products, constructing a training set and a test set; constructing a text-guided zero-shot industrial defect detection model, and training the model using the training set; inputting the test set into the trained model for testing; wherein, during the training phase, the model utilizes defect information from some types of industrial products to learn and expand to unified text prompts for other industrial products; during the testing phase, the model can be tested on industrial product types that have never been trained, detecting whether there are defects and locating the defects. The present invention fully utilizes the prior knowledge contained in the pre-trained multimodal large model CLIP, enabling the model to adaptively learn universal text prompts for different objects, achieving zero-shot defect detection for various industrial products, and can be widely applied in the field of computer vision technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a text-guided zero-sample industrial defect detection method, device, and medium. Background Art

[0002] Industrial defect detection is a key application of object detection. Industrial quality inspection technology utilizes advanced image processing and machine learning algorithms to identify and classify defects or anomalies during the production process, such as scratches, cracks, stains, and component defects. This ensures compliance with design specifications and helps production managers quickly determine the severity of the problem and address the issue. Existing zero-shot industrial defect detection models mostly use deep learning models to reconstruct normal industrial products and learn the characteristics of defect-free samples. During the testing phase, the input image is reconstructed and the difference between the reconstructed image and the input image is used to determine whether a defect exists. Because the training data consists only of defect-free samples, ideally, the model cannot reconstruct defects in the sample during the testing phase. However, if the input image during the testing phase contains defects, the difference between the reconstructed image and the input image in the defective area will be significant. The model then uses this to determine whether the input image contains defects.

[0003] However, the existing industrial defect detection methods have the following problems that need to be solved urgently: 1) Traditional methods and zero-shot industrial defect detection models based on deep neural networks are limited by actual application scenarios. They need to model one or several industrial products, and the models cannot detect defects in product types that are not in the training set. 2) The complex industrial production environment limits the adaptability of existing detection models. There are many types of industrial defects and few labeled samples. Therefore, the detection model needs stronger small sample utilization capabilities to adapt to the current situation of few labeled samples, and needs stronger scalability to detect unlabeled defect types. 3) Traditional methods and zero-shot industrial defect detection models based on deep neural networks require a lot of manual labeling (expert experience) and development costs. The cost of collecting industrial defects is high and the labeling difficulty is high. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a text-guided zero-sample industrial defect detection method, device and medium.

[0005] The first technical solution adopted by the present invention is:

[0006] A text-guided zero-shot industrial defect detection method comprises the following steps:

[0007] Obtain images and text semantic labels of industrial products to build training and test sets;

[0008] Constructing a text-guided zero-shot industrial defect detection model and training the zero-shot industrial defect detection model using a training set; wherein the input of the zero-shot industrial defect detection model is the image to be segmented, and the output is the defect segmentation result and classification result;

[0009] The test set is input into the trained zero-shot industrial defect detection model for testing;

[0010] Among them, during the training phase, the zero-shot industrial defect detection model uses defect information from some types of industrial products to learn and expand to unified text prompts for other industrial products; during the testing phase, the zero-shot industrial defect detection model can be tested on types of industrial products that have never been trained, to detect whether there are defects and locate the defects.

[0011] Furthermore, a zero-shot industrial defect detection model was built using the Python-based deep learning library pytorch1.2.

[0012] Furthermore, the acquisition of images and text semantic labels of industrial products and the construction of training sets and test sets include:

[0013] Get the preset data set and record the nth image in the data set as I n , through bilinear interpolation, the image I n The size is adjusted to C*H*W; where C, H, and W are the channels, height, and width of the image respectively;

[0014] Image I n The corresponding defect segmentation grayscale image is denoted as M n , M n The size is adjusted to 1*H*W;

[0015] For the defect-free samples in the data set, the corresponding defect segmentation grayscale image is recorded as a full-zero tensor of size 1*H*W;

[0016] For image I n The corresponding text semantic labels are recorded as Anomaly and Normal, and these label information are finally recorded as C n .

[0017] Furthermore, the MVTec and VisA test sets are specifically used, AUROC and AUPRO are used as segmentation indicators, and AUROC and AP are used as classification indicators to verify the zero-shot defect detection effect of the text-guided zero-shot industrial defect detection method.

[0018] Furthermore, the objects in the image are divided into two categories: defective objects and non-defective objects.

[0019] Furthermore, the bilinear interpolation method is calculated as follows:

[0020]

[0021] Where, (src x , src y ) represents the original coordinates, (des x ,des y ) represents the target coordinate after interpolation, (src w , src h ) represents the width and height of the original image, (des w ,des h ) represents the width and height of the interpolated target image.

[0022] Furthermore, the zero-shot industrial defect detection model includes an input layer, a hidden layer, and an output layer; the hidden layer includes a CLIP pre-training module, a learnable prompt module, an adaptive prompt module, an image-text feature interaction module, and a segmentation and classification task module;

[0023] The CLIP pre-training model module works as follows: the CLIP pre-trained image encoder and text encoder are respectively denoted as E I and E T ; Image I n Input image encoder E I , and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4} layer {F P1 ,F P2 ,F P3 ,F P4} as the image defect segmentation feature F P ;

[0024] The learnable prompt module includes multiple learnable text prompt tensors and CLIP pre-trained text encoder E T ; The learnable text hint tensor is divided into defect hint P A and non-defect hint P N , where P A ∈R na*d , P N ∈R nn*d , na and nn represent the number of defect cues and non-defect cues respectively, d represents the length of each learnable cue, and d is related to E T The length of the tensor after text embedding in P A and P N Parallel input E TAs the embedded tensor, we get the classification features F of multiple defect prompts A and the classification features F of non-defect hints N , where F A ∈R na*f , F N ∈R nn*f , f represents the length of each learnable prompt feature, f and text encoder E T The output feature dimensions are consistent;

[0025] The adaptive prompt module S consists of a Meta module and a multi-layer perceptron MLP; the input of the Meta module is the image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI , the prompt feature dimension before and after the module remains unchanged; the input of the multi-layer perceptron is the updated classification feature F of multiple defect prompts AI and the classification features F of non-defect hints NI , the output is the classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT , where F Ar ∈R 1*f , F NT ∈R 1*f ;

[0026] The input of the image-text feature interaction module C is the image classification feature F I , image defect segmentation feature F P , classification features of single defect prompt F AT and the classification features F of non-defect hints NT , the output is the reconstructed image classification feature F I ′ , image defect segmentation feature F P ′ , the feature dimension remains unchanged;

[0027] The segmentation and classification task module T consists of a segmentation task module and a classification task module; in the segmentation task module, the module input is the image defect segmentation feature F P ′ , classification features of single defect prompt F AT and the classification features F of non-defect hints NT , the output is the model segmentation result M corresponding to the defect prompt and non-defect prompt A and MN , M A ,M N ∈R C*H*W ; In the classification task module, the input of the module is the image classification feature F I ′ , classification features of single defect prompt F AT and the classification features F of non-defect hints NT , the output is the probability P of defects in the image A and the probability P that there is no defect in the image N , where P A ,P N ∈R 1*1 .

[0028] Furthermore, the zero-shot industrial defect detection model works as follows:

[0029] First, the original image is input into a pre-trained image encoder to obtain image features; the learnable prompt is input into a pre-trained text encoder to obtain multiple text features corresponding to defects and non-defects;

[0030] Input multiple text prompt features and image features into the adaptive prompt module to obtain text prompt features that are adapted to the input image, which are used as text prompt features for subsequent calculations in the model.

[0031] The text prompt features and image features are fed into the image-text feature interaction module to obtain the image features incorporating the prompt information, which are used as the image features for subsequent operations in the model.

[0032] By sending the image features and text prompt features into the classification task module and the segmentation task module, the predicted classification results and the predicted segmentation results can be obtained.

[0033] Furthermore, the learnable hint module simultaneously learns multiple defect hints and non-defect hints, which focus on the defects in the image but not the types of objects in the image. The hint feature dimensions are similar to the CLIP pre-trained text encoder E T The output feature dimensions are consistent.

[0034] Furthermore, the adaptive prompt module S is used to adaptively select the most suitable prompt feature for the image based on the features of the object and defect in the image, thereby improving the model's ability to detect different defects;

[0035] The Meta module includes a linear layer. The function of the Meta module is to integrate image information into the prompt features, thereby obtaining prompt features that match the input image to improve the effect of defect detection;

[0036] The multi-layer perceptron MLP includes three linear layers and an activation function;

[0037] The image-text feature interaction module C is composed of several cross-attention modules, and the output feature dimension is consistent with the input feature dimension; the main function of the image-text feature interaction module C is to further strengthen the connection between the prompt feature and the image feature, so as to facilitate the model to distinguish between defective and non-defective areas.

[0038] The segmentation and classification task module consists of a segmentation task module and a classification task module. The output of the segmentation task module is the predicted segmentation result, and the output of the classification task module is the predicted classification result.

[0039] Furthermore, in the process of training the zero-shot industrial defect detection model, the segmentation loss function adopts FocalLoss, Dice Loss and Cross Entropy Loss, and the classification loss function adopts Cross Entropy Loss;

[0040] For the segmentation loss, calculate the segmentation results M respectively A and M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M N ,M A ],M),L Dice (M A , M), L Dice (M N , 1-M), L CE ([M N ,M A ],M);

[0041] For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

[0042] Furthermore, the function loss value between the predicted segmentation result and the true segmentation result is calculated by using Focal Loss, Dice Loss, and cross entropy loss; the function loss value between the predicted classification result and the true classification result is calculated by using cross entropy loss.

[0043] Furthermore, the training of the zero-sample industrial defect detection model using a training set includes:

[0044] Repeat the training steps multiple times to obtain multiple loss values; find the minimum total loss value from the multiple loss values, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-shot industrial defect detection model, which are denoted by W best and b best ;Complete the training of zero-shot industrial defect detection model;

[0045] Inputting the test set into the trained zero-shot industrial defect detection model for testing includes:

[0046] The test set is input into the trained zero-shot industrial defect detection model, and the W best and b best Make predictions and obtain the defect segmentation image M corresponding to the test set p and defect classification C p , to achieve zero-sample industrial defect detection.

[0047] The second technical solution adopted by the present invention is:

[0048] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a text-guided zero-sample industrial defect detection method as described above.

[0049] The third technical solution adopted by the present invention is:

[0050] A computer-readable storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, at least one program, a code set, or an instruction set is loaded and executed by a processor to implement a text-guided zero-sample industrial defect detection method as described above.

[0051] The fourth technical solution adopted by the present invention is:

[0052] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above-mentioned text-guided zero-shot industrial defect detection method.

[0053] The beneficial effect of the present invention is that the present invention makes full use of the prior knowledge contained in the pre-trained multimodal large model CLIP, so that the model can adaptively learn common text prompts for different objects, thereby realizing zero-sample defect detection of various industrial products. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 1 is a flowchart of a text-guided zero-shot industrial defect detection method according to an embodiment of the present invention;

[0056] Figure 2 This is a structural block diagram of a text-guided zero-shot industrial defect detection model in an embodiment of the present invention;

[0057] Figure 3 2 is a schematic diagram of a framework of a text-guided zero-shot industrial defect detection model in an embodiment of the present invention;

[0058] Figure 4 This is a flowchart of the steps of a text-guided zero-shot industrial defect detection method in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0061] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0062] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0063] Explanation of terms:

[0064] CLIP: Contrastive Language–Image Pre-training, the CLIP model is a multimodal training model that learns the matching relationship between images and text by comparing their vector representations.

[0065] Example 1

[0066] like Figure 4 As shown, this embodiment provides a text-guided zero-sample industrial defect detection method, including the following steps:

[0067] S1. Obtain images and text semantic labels of industrial products and construct training and test sets;

[0068] S2. Build a text-guided zero-shot industrial defect detection model and train the zero-shot industrial defect detection model using a training set; wherein the input of the zero-shot industrial defect detection model is the image to be segmented, and the output is the defect segmentation result and the classification result;

[0069] S3. Input the test set into the trained zero-shot industrial defect detection model for testing.

[0070] The method of this embodiment is divided into two phases: training and testing. During the training phase, a zero-shot industrial defect detection model based on a pretrained multimodal model is first constructed. By analyzing a subset of existing industrial product categories and extracting defect information, the model learns unified textual prompts applicable to a wide range of industrial products. This process not only improves the model's ability to identify known defects but also focuses on capturing and generalizing defect characteristics, enabling it to be transferred to unseen product categories.

[0071] During the testing phase, the trained model is tested on a new category of industrial products. When new data is input, the model relies on the knowledge it has absorbed during the training phase to detect defects and accurately locate them. This zero-shot detection capability eliminates the need for large numbers of labeled samples as traditional methods do. Instead, the model can independently judge and analyze a completely new product category using previously learned textual cues and features.

[0072] In this way, the method in this embodiment not only improves the flexibility and adaptability of industrial defect detection, but also significantly reduces reliance on labeled data, lowering production and inspection costs and achieving more efficient quality control. The development of this technology will provide new solutions and technical support for the intelligent and automated industrial production, helping to improve overall product quality and production efficiency.

[0073] The above method is explained in detail below with reference to the accompanying drawings and specific implementation methods.

[0074] (1) Constructing training and test sets

[0075] See also Figure 1 During the training and testing process, the image dataset is first preprocessed. The dataset includes original images of different types of industrial products, corresponding defect segmentation grayscale images, and text semantic labels. For example, the mainstream industrial defect detection datasets MVTec and VisA are selected, and the industrial product types of the two datasets are not repeated. MVTec and VisA are used as image training sets and test sets respectively. The same preprocessing steps are performed on the two datasets. Taking the MVTec dataset as an example, the preprocessing steps of the dataset include image adjustment, defect segmentation map adjustment, and text semantic label processing.

[0076] Image adjustment: For each image in the dataset, let the nth image be I n , using bilinear interpolation, the image I n The image size is resized to C*H*W, where C represents the number of channels (e.g., RGB images have 3 channels), and H and W represent the height and width of the image, respectively. This step ensures that all images have consistent sizes, which facilitates subsequent model processing.

[0077] Defect segmentation map adjustment: For image I n The corresponding defect segmentation grayscale image is denoted as M n , similarly resized to 1*H*W using bilinear interpolation. Here, "1" represents a single channel of the grayscale image. This grayscale image represents the location of defects. If an image is a defect-free sample, its corresponding defect segmentation grayscale image is recorded as a 1*H*W tensor filled with all zeros, indicating the absence of any defects.

[0078] Text semantic label processing: For image I n The corresponding text semantic labels are recorded as Anomaly (defect) and Normal (normal). In order to facilitate model processing and mathematical calculations, the Anomaly label is recorded as 1 and the Normal label is recorded as 0. These label information are finally recorded as C n , will be used together with the image data for training and testing.

[0079] (2) Build and train a text-guided zero-shot industrial defect detection model

[0080] See also Figure 2 and Figure 3 In this embodiment, a text-guided zero-shot industrial defect detection model is constructed and trained. The model can effectively detect and locate defects in industrial products. The input of the model is the image to be segmented I n , the output is the defect segmentation result grayscale image M n ′ And the model classification result C n ′ The model structure includes an input layer, a hidden layer, and an output layer, wherein the input layer receives the image to be segmented I n , providing raw data for subsequent feature extraction and processing; the hidden layer is further subdivided into CLIP pre-training module, learnable prompt module, adaptive prompt module, image-text feature interaction module and segmentation and classification task module to achieve complex data processing and feature extraction. These modules work together to extract and fuse features from images and text to achieve accurate defect detection; the output layer outputs the grayscale image M of the defect segmentation result n ′ And the model classification result C n ′ , respectively represent the location and classification information of the defect.

[0081] In some embodiments, the CLIP pre-trained model module adopts the ViT-L / 14@336px version, including pre-trained image encoder and text encoder, respectively denoted as E I and E T . Image I n Input image encoder E I Encode and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4} layer {F P1 ,F P2 ,F P3 ,F P4} as the image defect segmentation feature FP .

[0082] In some embodiments, the learnable hint module includes a plurality of learnable text hint tensors and a CLIP pre-trained text encoder E T The learnable text hint tensor is divided into defect hint P A and non-defect hint P N , where P A ∈R na*d , P N ∈R nn *d , na and nn represent the number of defect cues and non-defect cues respectively, d represents the length of each learnable cue, and d is related to E T The length of the tensor after the text embedding in P A and P N Parallel input E T As the embedded tensor, we get the classification features F of multiple defect prompts A and the classification features F of non-defect hints N , where F A ∈R na*f , F N ∈R nn*f , f represents the length of each learnable prompt feature, f and text encoder E T The output feature dimensions are consistent.

[0083] In some embodiments, the adaptive hint module S consists of a Meta module and a multi-layer perceptron (MLP). It is primarily used to adaptively adjust hint features based on image features to improve defect detection. This module is designed to enable the model to better understand and identify defects in images by integrating image and hint features.

[0084] The input of the Meta module is the image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI, the dimension of the prompt features before and after the module remains unchanged. The main function of the Meta module is to integrate image information into the prompt features, so as to obtain prompt features that match the input image and improve the effect of defect detection. Specifically, the first step of the processing is to add the image features as biases (Bias) to the classification features of defect prompts and non-defect prompts. This process helps to emphasize the key features and contextual information of the image during feature fusion, so that the model can better capture various prompts related to the image. Next, these adjusted features will be sent to the linear layer to further fuse the image features and prompt features. This design ensures that the interactive relationship between the two features can be unfolded, so that the final output features can better reflect the actual situation and defects of the objects in the image.

[0085] The second part of the adaptive prompt module S is the multi-layer perceptron MLP, which receives the updated classification features F of multiple defect prompts AI and the classification features F of non-defect hints NI After that, further processing is performed. The main output of the MLP module is the classification feature F of a single defect prompt. AT and the classification features F of non-defect hints NT , where F AT ∈R 1*f , F NT ∈R 1*f Such an output structure enables the model to flexibly select the most suitable prompt features for the image based on the specific input image.

[0086] Multilayer Perceptron MLP is used to AT and F NT The combination and transformation of the MLP further optimizes defect-related prompt features. During implementation, the MLP processes and computes prompt features from multiple dimensions, enabling the model to demonstrate greater adaptability and recognition capabilities when faced with different types of defects. Ultimately, the design goal of the adaptive prompt module S is to effectively match prompt features, making the model not only more accurate in defect detection tasks but also more generalizable.

[0087] Through this adaptive adjustment, the adaptive prompt module S fully considers the actual object and defect characteristics in the image, improving the model's ability to detect different defects and ensuring high efficiency and accuracy in complex industrial environments. This design undoubtedly provides strong technical support for the model's intelligent defect detection.

[0088] In some embodiments, the image-text feature interaction module C is composed of multiple cross-attention modules, which are important components of the model and play a key bridge role. The input of this module includes the image classification feature F I, image defect segmentation feature F P , classification features of single defect prompt F AT and the classification features F of non-defect hints NT Together, these input features provide rich information, enabling the model to gain a deeper understanding of the context of defects in the image and their relationship with the cue features.

[0089] Specifically, the image classification feature F I Contains the overall understanding of the image to be detected, reflecting the basic features and background information of the objects in the image; and the image defect segmentation feature F P It focuses on the details in the image and provides detailed information about the specific location and shape of the defect. AT and the classification features F of non-defect hints NT It provides the model with additional semantic guidance on whether a defect exists or not, helping the model to consider the information carried by the text prompt when making judgments.

[0090] Through the interaction of these input features, the output of this module includes the reconstructed image classification features F I ′ and image defect segmentation feature F P ′ . The dimensions of these output features are consistent with the input features, which ensures the consistency of information transfer and conversion dimensions during feature processing, thereby avoiding unnecessary errors in subsequent module processing. The main function of this module is to strengthen the connection between prompt features and image features, and further enhance the model's understanding ability. In the defect detection task, the relationship between image and text is effectively incorporated, enabling the model to more accurately distinguish between defects and non-defective areas. This not only improves the model's sensitivity to defect characteristics, but also enhances its adaptability when facing complex backgrounds or diverse products. Through such a feature interaction mechanism, the model can more comprehensively capture the key information in the image, making the final defect detection results more accurate and reliable.

[0091] In some embodiments, the segmentation and classification task module T is composed of a segmentation task module and a classification task module. This module is responsible for performing detailed segmentation and accurate classification of defects in the input image to achieve comprehensive defect detection capabilities.

[0092] In the segmentation task module, the input includes image defect segmentation features F P ′ , classification features of single defect prompt F AT and the classification features F of non-defect hints NT The module analyzes these input features to identify defect areas in the image and perform corresponding segmentation. Specifically, the image defect segmentation feature FP ′ Responsible for providing regional features of potential defects in the image, while the classification features of single defect prompts F AT and the classification features F of non-defect hints NT The output of this module includes the model segmentation result M corresponding to the defect prompt. A The model segmentation result M corresponding to the non-defect prompt N , M A ,M N ∈R C*H*W , which represent the prediction results of defective and non-defective areas, respectively. According to the above structure, the segmentation task module can effectively separate defective and non-defective areas from the background through specialized feature processing, laying the foundation for subsequent classification and other processing tasks.

[0093] The classification task module is responsible for analyzing the image to determine whether it has defects and classify it. In this module, the input is the processed image classification feature F I ′ , classification features of single defect prompt F at and the classification features F of non-defect hints NT When these input features are combined, they can fully reveal the content and state of the image, allowing the possibility of individual defects to be evaluated. The output of the module is the probability P of a defect in the image. A and the probability P that there is no defect in the image N , reflects the model’s judgment of image defects. Specifically, P A and P N The output format is P A ,P N ∈R 1*1 , respectively representing the probability of a defect appearing in the image and the probability of no defect. By counting and calculating probabilities, the model can clearly distinguish whether there are defects in the image and perform targeted subsequent processing based on the output results.

[0094] As an optional implementation, the zero-shot industrial defect detection model works as follows:

[0095] First, the original image is input into a pre-trained image encoder to obtain image features; the learnable prompt is input into a pre-trained text encoder to obtain multiple text features corresponding to defects and non-defects;

[0096] Input multiple text prompt features and image features into the adaptive prompt module to obtain text prompt features that are adapted to the input image, which are used as text prompt features for subsequent calculations in the model.

[0097] The text prompt features and image features are fed into the image-text feature interaction module to obtain the image features incorporating the prompt information, which are used as the image features for subsequent operations in the model.

[0098] By sending the image features and text prompt features into the classification task module and the segmentation task module, the predicted classification results and the predicted segmentation results can be obtained.

[0099] (3) Loss function

[0100] During the model training process, defining and using appropriate loss functions is the key to ensuring comprehensive improvement of model performance. The segmentation loss function of the model uses Focal Loss, Dice Loss and Cross Entropy Loss. The combination of these loss functions can improve the model's processing ability for segmentation tasks from different angles and ensure accurate identification of defect areas. For segmentation loss, calculate M respectively. A , M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M N ,M A ],M),L Dice (M A , M), L Dice (M N , 1-M), L CE ([M N ,M A ],M).

[0101] Focal Loss was first applied to target detection tasks. It was proposed mainly to solve the problem of imbalance in the number of positive and negative samples and the imbalance of difficult and easy samples. Focal is defined as follows:

[0102] L Focal (p t )=-α t (1-p t ) γ log(p t )

[0103] Among them, p t Represents the probability of the model predicting a certain category, α t Used to balance the number of positive and negative samples, giving a smaller α to samples with a large number of samples t Value, the smaller the number of samples, the larger α is given t The value γ is used to adjust the imbalance problem of difficult and easy samples. Generally, γ≥1 is taken to reduce the loss of easy samples, so that the model pays more attention to difficult samples.

[0104] Dice Loss is a commonly used loss function that handles foreground-background imbalance. It reduces the influence of background areas by calculating the intersection ratio of predicted and true labels. Its mathematical expression is as follows:

[0105]

[0106] Here, |X∩U| represents the number of elements in the intersection of X and Y, and |X| and |Y| represent the number of elements in X and Y, respectively.

[0107] Cross entropy loss is often used in binary and multi-classification problems. This loss is sensitive to probability prediction results and can effectively encourage the predicted probability to be close to the true label. Its expression is as follows:

[0108]

[0109] in, represents the predicted label and y represents the true label.

[0110] The classification loss function uses the cross entropy loss function. For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

[0111] Repeat the training steps Q times to obtain the zero-shot industrial defect detection training model, and obtain a total of Q*N loss values. Find the minimum total loss value, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-shot industrial defect detection model, which are denoted by W best and b best ; That is, the training of the zero-sample industrial defect detection model is completed.

[0112] (4) Model testing

[0113] Input the test set into the trained model for testing. Input the preprocessed data into the trained zero-shot industrial defect detection model and use W best and b best Make predictions and obtain the defect segmentation image M corresponding to the test set p and defect classification C p , zero-shot industrial defect detection can be achieved. Optionally, AUROC and AUPRO are used as segmentation indicators, and AUROC and AP are used as classification indicators to verify the zero-shot defect detection effect of the text-guided zero-shot industrial defect detection method.

[0114] In summary, the method of the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0115] 1) Zero-shot learning capability. Traditional industrial defect detection models typically require large amounts of labeled data for training. However, this invention can accurately detect and classify new defects using only textual cues, without direct exposure to a specific sample or similar defect samples. This capability significantly reduces the model's data dependency and improves its applicability and flexibility.

[0116] 2) Efficient defect detection capabilities. This invention combines a pre-trained multimodal model with an efficient network structure to achieve efficient segmentation and accurate classification of defect areas. The model can capture the differences between model predictions and true labels from different perspectives and continuously optimize its performance during training. As a result, compared with existing technologies, this invention significantly improves both defect segmentation accuracy and classification precision.

[0117] 3) Strong generalization capabilities. This paper uses a text-based approach, allowing the model to learn common defect prompt features under the guidance of a pre-trained multimodal model and training set. This enables the model to demonstrate strong generalization capabilities when faced with complex and changing industrial scenarios. This means that even when encountering new defect types that differ from the training data in actual applications, the model can quickly adapt and provide accurate detection results.

[0118] 4) Reduce manual labeling and enhance algorithm universality. Existing industrial defect detection systems rely heavily on manual labeling, resulting in highly targeted algorithm designs but limited universality and the ability to dynamically adapt to different scenarios. This invention, by combining large pre-trained models with anomaly detection technology, will provide new insights for the development of defect detection systems.

[0119] 5) Efficiently utilize small samples to reduce promotion costs. The zero-sample / few-sample industrial defect detection framework will target common problems in actual application scenarios and can be promoted to various industrial product quality inspection processes at low cost and high efficiency.

[0120] 6) Enhance model migration capabilities and adapt to different scenarios. Another challenge in deploying detection models in real-world application scenarios is domain adaptation, which involves the discrepancy between training data and actual detection data. Therefore, research on model migration and scenario adaptation is essential.

[0121] 7) Enhance model generalization capabilities with the help of pre-trained large models. Large models have high prediction accuracy, strong transfer learning and domain adaptability, low dependence on labeled data, and strong multimodal processing capabilities, which will provide new possibilities for industrial defect detection.

[0122] Example 2

[0123] An embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following Figure 4 A text-guided zero-shot industrial defect detection method is shown.

[0124] It is understood that the memory may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the data storage area may store data created based on the use of the server, etc.

[0125] The processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the server. It executes various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory, as well as accessing data stored in memory. Optionally, the processor may be implemented using at least one of the following hardware forms: digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU) and a modem. The CPU primarily processes the operating system and application programs, while the modem handles wireless communications. It is understood that the modem may not be integrated into the processor and may be implemented separately via a single chip.

[0126] Since the electronic device is an electronic device corresponding to a text-guided zero-sample industrial defect detection method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0127] Example 3

[0128] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 4 A text-guided zero-shot industrial defect detection method is shown.

[0129] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0130] Since the storage medium is a storage medium corresponding to a text-guided zero-sample industrial defect detection method in an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0131] Example 4

[0132] In some possible implementations, various aspects of the methods of the embodiments of the present invention may also be implemented in the form of a program product, which includes program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform the steps of the text-guided zero-shot industrial defect detection method according to various exemplary embodiments of the present application described above in this specification. The executable computer program code or "code" used to perform the various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or various other programming languages.

[0133] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0135] The above embodiments are intended only to illustrate the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent changes or modifications made based on the essence of the present invention are intended to be covered by the scope of protection of the present invention.

Claims

1. A text-guided zero-shot industrial defect detection method, characterized in that: The following steps are involved: Obtain images and text semantic labels of industrial products to build training and test sets; Constructing a text-guided zero-shot industrial defect detection model and training the zero-shot industrial defect detection model using a training set; wherein the input of the zero-shot industrial defect detection model is the image to be segmented, and the output is the defect segmentation result and classification result; The test set is input into the trained zero-shot industrial defect detection model for testing; During the training phase, the zero-shot industrial defect detection model uses defect information from some types of industrial products to learn and expand unified text prompts for other industrial products. During the testing phase, the zero-shot industrial defect detection model can be tested on industrial product types that it has never been trained on, detecting whether defects exist and locating defects. The zero-shot industrial defect detection model includes an input layer, a hidden layer and an output layer; the hidden layer includes a CLIP pre-training module, a learnable prompt module, an adaptive prompt module, an image-text feature interaction module, and a segmentation and classification task module; the CLIP pre-training module works as follows: the CLIP pre-trained image encoder and text encoder are respectively denoted as E I and E T ; Image I n Input image encoder E I , and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4} layer {F P1 ,F P2 ,F P3 ,F P4 } as the image defect segmentation feature F P ; The learnable prompt module includes multiple learnable text prompt tensors and CLIP pre-trained text encoder E T ; The learnable text hint tensor is divided into defect hint P A and non-defect hint P N ;P A and P N Parallel input E T As the embedded tensor, Get the classification features F of multiple defect prompts A and the classification features F of non-defect hints N ; The adaptive prompt module consists of a Meta module and a multi-layer perceptron; the input of the Meta module is the image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI The input of the multi-layer perceptron is the updated classification features F of multiple defect prompts AI and the classification features F of non-defect hints NI , the output is the classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT ; The input of the image-text feature interaction module is the image classification feature F I , image defect segmentation feature F P , classification features of single defect prompt F AT and the classification features F of non-defect hints NT , the output is the reconstructed image classification feature F I ′, image defect segmentation feature F P '; The segmentation and classification task module consists of a segmentation task module and a classification task module; in the segmentation task module, the module input is the image defect segmentation feature F P ′, classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT , the output is the model segmentation result M corresponding to the defect prompt and non-defect prompt A and M N ; In the classification task module, the input of the module is the image classification feature F I ′, classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT , the output is the probability P of defects in the image A and the probability P that there is no defect in the image N .

2. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The process of obtaining images and text semantic labels of industrial products and constructing training and test sets includes: Get the preset data set and record the nth image in the data set as I n , through bilinear interpolation, the image I n The size is adjusted to C*H*W; where C, H, and W are the channel, height, and width of the image respectively; Image I n The corresponding defect segmentation grayscale image is denoted as M n , M n The size is adjusted to 1*H*W; For image I n The corresponding text semantic labels are recorded as Anomaly or Normal.

3. The text-guided zero-sample industrial defect detection method according to claim 2, characterized in that: The bilinear interpolation method is calculated as follows: Where, (src x , src y ) represents the original coordinates, (des x ,des y ) represents the target coordinate after interpolation, (src w , src h ) represents the width and height of the original image, (des w ,des h ) represents the width and height of the interpolated target image.

4. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The learnable hint module simultaneously learns multiple defect hints and non-defect hints, which focus on the defects in the image but not the types of objects in the image. The hint feature dimensions are similar to the CLIP pre-trained text encoder E T The output feature dimensions are consistent.

5. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The function of the adaptive prompt module is to adaptively select the prompt features that are most suitable for the image based on the object and defect features in the image, thereby improving the model's detection ability for different defects; The Meta module includes a linear layer. The function of the Meta module is to integrate image information into the prompt features, thereby obtaining prompt features that match the input image to improve the effect of defect detection; The multilayer perceptron includes three linear layers and an activation function; The image-text feature interaction module is composed of several cross-attention modules, and the output feature dimension is consistent with the input feature dimension. The function of the image-text feature interaction module is to further strengthen the connection between the prompt feature and the image feature to facilitate the model to distinguish between defective and non-defective areas.

6. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: In the process of training the zero-shot industrial defect detection model, the segmentation loss function adopts Focal Loss, Dice Loss and Cross Entropy Loss, and the classification loss function adopts Cross Entropy Loss; For the segmentation loss, calculate the segmentation results M respectively A and M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M N ,M A ],M)、L Dice (M A ,M)、L Dice (M N ,1-M)、L CE ([M N ,M A ],M); For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

7. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The adopting of the training set to train the zero-sample industrial defect detection model includes: Repeat the training steps multiple times to obtain multiple loss values; find the minimum total loss value from the multiple loss values, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-shot industrial defect detection model, which are denoted by W best and b best ;Complete the training of zero-shot industrial defect detection model; Inputting the test set into the trained zero-shot industrial defect detection model for testing includes: The test set is input into the trained zero-shot industrial defect detection model, and the W best and b best Make predictions and obtain the defect segmentation image M corresponding to the test set p and defect classification C p , to achieve zero-sample industrial defect detection.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

Images