Zero-sample industrial defect detection method and equipment based on text guidance and medium

Abstract

The invention discloses a zero-sample industrial defect detection method and device based on text guidance and a medium, and the method comprises the steps: obtaining an image and a text semantic tag of an industrial product, and constructing a training set and a test set; constructing a zero-sample industrial defect detection model based on text guidance, and training the model by adopting the training set; inputting the test set into the trained model for testing; wherein in the training stage, the model learns and extends to unified text prompts of other industrial products by utilizing defect information in part of types of industrial products; in the testing stage, the model can be used for testing the types of industrial products which are not trained, detecting whether the industrial products have defects or not and positioning the defects. According to the method, priori knowledge contained in the pre-trained multi-modal large model CLIP is fully utilized, so that the model can adaptively learn general text prompts for different objects, zero sample defect detection of various industrial products is realized, and the method can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a text-guided zero-sample industrial defect detection method, device and medium. Background Art

[0002] Industrial defect detection is an important application of target detection tasks. Industrial quality inspection technology involves the use of advanced image processing and machine learning algorithms to identify and classify defects or anomalies in the production process, such as scratches, cracks, stains, and component defects, to ensure compliance with design specifications and help production managers quickly determine the severity of the problem and treatment measures. Most of the existing industrial zero-shot industrial defect detection models reconstruct normal industrial products through deep learning models and learn the characteristics of defect-free samples; in the test phase, the input image is reconstructed and the difference between the reconstructed result and the input image is used to determine whether there is a defect. Since the training data only has defect-free samples, ideally, the model cannot reconstruct the defects in the sample during the test phase. If the input image during the test phase is defective, the difference between the reconstructed image and the input image in the defective part will be very large. The model uses this to determine whether there is a defect in the input image.

[0003] However, the existing industrial defect detection methods have the following problems that need to be solved urgently: 1) Traditional methods and zero-shot industrial defect detection models based on deep neural networks are limited by actual application scenarios and need to model one or several industrial products. The models cannot detect defects in product types that are not in the training set. 2) The complex industrial production environment limits the adaptability of existing detection models. There are many types of industrial defects and few labeled samples. Therefore, the detection model needs stronger small sample utilization capabilities to adapt to the current situation of few labeled samples, and stronger scalability to detect unlabeled defect types. 3) Traditional methods and zero-shot industrial defect detection models based on deep neural networks require a lot of manual labeling (expert experience) and development costs, and the cost of collecting industrial defects is high and the labeling difficulty is high. Summary of the invention

[0004] In order to solve at least one of the technical problems existing in the prior art to a certain extent, the present invention aims to provide a text-guided zero-sample industrial defect detection method, device and medium.

[0005] The first technical solution adopted by the present invention is:

[0006] A text-guided zero-shot industrial defect detection method comprises the following steps:

[0007] Obtain images and text semantic labels of industrial products and build training and test sets;

[0008] Constructing a text-guided zero-shot industrial defect detection model, and using a training set to train the zero-shot industrial defect detection model; wherein the input of the zero-shot industrial defect detection model is an image to be segmented, and the output is a defect segmentation result and a classification result;

[0009] The test set is input into the trained zero-shot industrial defect detection model for testing;

[0010] Among them, during the training stage, the zero-shot industrial defect detection model uses the defect information in some types of industrial products to learn and expand to unified text prompts for other industrial products; during the testing stage, the zero-shot industrial defect detection model can be tested on types of industrial products that have never been trained, to detect whether there are defects and locate the defects.

[0011] Furthermore, a zero-shot industrial defect detection model was built using the Python-based deep learning library pytorch1.2.

[0012] Furthermore, the acquisition of images and text semantic labels of industrial products and the construction of training sets and test sets include:

[0013] Get the preset data set and record the nth image in the data set as I n , through bilinear interpolation, the image I n The size is adjusted to C*H*W; where C, H, and W are the channel, height, and width of the image respectively;

[0014] Image I n The corresponding defect segmentation grayscale image is denoted as M n , M n The size is adjusted to 1*H*W;

[0015] For the defect-free samples in the data set, the corresponding defect segmentation grayscale image is recorded as a full-zero tensor of size 1*H*W;

[0016] For image I n The corresponding text semantic labels are recorded as Anomaly and Normal, and these label information are finally recorded as C n .

[0017] Furthermore, the MVTec and VisA test sets are specifically adopted, AUROC and AUPRO are adopted as segmentation indicators, and AUROC and AP are adopted as classification indicators to verify the zero-shot defect detection effect of the text-guided zero-shot industrial defect detection method.

[0018] Furthermore, the object categories in the image are divided into two categories: defective objects and non-defective objects.

[0019] Furthermore, the bilinear interpolation method is calculated as follows:

[0020]

[0021] In the formula, (src x , src y ) represents the original coordinates, (des x ,des y ) represents the target coordinates after interpolation, (src w , src h ) represents the width and height of the original image, (des w ,des h ) represents the width and height of the interpolated target image.

[0022] Furthermore, the zero-shot industrial defect detection model includes an input layer, a hidden layer and an output layer; the hidden layer includes a CLIP pre-training module, a learnable prompt module, an adaptive prompt module, an image-text feature interaction module, and a segmentation and classification task module;

[0023] The working method of the CLIP pre-trained model module is as follows: the pre-trained image encoder and text encoder of CLIP are respectively denoted as E I and E T ; Image I n Input image encoder E I , and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4}th layer {F P1 ,F P2 ,F P3 ,F P4} as the image defect segmentation feature F P ;

[0024] The learnable prompt module includes a plurality of learnable text prompt tensors and a CLIP pre-trained text encoder E T ; The learnable text prompt tensor is divided into defect prompt P A and non-defect hint P N , where P A ∈R na*d , P N ∈R nn*d , na and nn represent the number of defect cues and non-defect cues respectively, d represents the length of each learnable cue, and d is related to E T The length of the tensor after the text embedding in P A and P N Parallel Input E TAs the embedded tensor, we get the classification features F of multiple defect prompts. A and the classification features F of non-defect hints N , where F A ∈R na*f , F N ∈R nn*f , f represents the length of each learnable prompt feature, f and text encoder E T The output feature dimensions are consistent;

[0025] The adaptive prompt module S consists of a Meta module and a multi-layer perceptron MLP; the input of the Meta module is the image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI , the prompt feature dimension before and after the module remains unchanged; the input of the multi-layer perceptron is the updated classification feature F of multiple defect prompts AI and the classification features F of non-defect hints NI , the output is the classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT , where F Ar ∈R 1*f , F NT ∈R 1*f ;

[0026] The input of the image-text feature interaction module C is the image classification feature F I , image defect segmentation feature F P , classification features of a single defect hint F AT and the classification features F of non-defect hints NT , the output is the reconstructed image classification feature F I ′ , image defect segmentation feature F P ′ , the feature dimension remains unchanged;

[0027] The segmentation and classification task module T is composed of a segmentation task module and a classification task module; in the segmentation task module, the input of the module is the image defect segmentation feature F P ′ , classification features of a single defect hint F AT and the classification features F of non-defect hints NT The output is the model segmentation result M corresponding to the defect prompt and non-defect prompt A and MN , M A ,M N ∈R C*H*W ; In the classification task module, the input of the module is the image classification feature F I ′ , classification features of a single defect hint F AT and the classification features F of non-defect hints NT , the output is the probability P of defects in the image A and the probability P that there is no defect in the image N , where P A ,P N ∈R 1*1 .

[0028] Furthermore, the zero-shot industrial defect detection model works as follows:

[0029] First, the original image is input into a pre-trained image encoder to obtain image features; the learnable prompt is input into a pre-trained text encoder to obtain multiple text features corresponding to defects and non-defects;

[0030] Input multiple text prompt features and image features into the adaptive prompt module to obtain text prompt features that are adapted to the input image as text prompt features that participate in subsequent calculations of the model;

[0031] The text prompt features and image features are sent to the image-text feature interaction module to obtain the image features after the prompt information is integrated, which are used as the image features involved in the subsequent calculation of the model;

[0032] By sending the image features and text prompt features into the classification task module and the segmentation task module, the predicted classification results and the predicted segmentation results can be obtained.

[0033] Furthermore, the learnable hint module simultaneously learns multiple defect hints and non-defect hints, which focus on the defects in the image but not the types of objects in the image. The hint feature dimensions are similar to the CLIP pre-trained text encoder E T The output feature dimensions are consistent.

[0034] Furthermore, the function of the adaptive prompt module S is to adaptively select the prompt feature that best suits the image according to the features of the object and defect in the image, so as to improve the model's ability to detect different defects;

[0035] The Meta module includes a linear layer. The function of the Meta module is to integrate image information into the prompt feature, so as to obtain the prompt feature matching the input image to improve the effect of defect detection;

[0036] The multi-layer perceptron MLP includes three linear layers and an activation function;

[0037] The image-text feature interaction module C is composed of several cross-attention modules, and the output feature dimension is consistent with the input feature dimension; the main function of the image-text feature interaction module C is to further strengthen the connection between the prompt feature and the image feature, so as to facilitate the model to distinguish between defective and non-defective areas.

[0038] The segmentation and classification task module consists of a segmentation task module and a classification task module. The output of the segmentation task module is the predicted segmentation result, and the output of the classification task module is the predicted classification result.

[0039] Furthermore, in the process of training the zero-shot industrial defect detection model, the segmentation loss function uses FocalLoss, Dice Loss and Cross Entropy Loss, and the classification loss function uses Cross Entropy Loss;

[0040] For the segmentation loss, the segmentation results M are calculated separately. A and M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M N ,M A ],M),L Dice (M A , M), L Dice (M N , 1-M), L CE ([M N ,M A ],M);

[0041] For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

[0042] Furthermore, the function loss value between the predicted segmentation result and the true segmentation result is calculated by using Focal Loss, Dice Loss, and cross entropy loss; the function loss value between the predicted classification result and the true classification result is calculated by using cross entropy loss.

[0043] Furthermore, the adopting of a training set to train the zero-sample industrial defect detection model includes:

[0044] Repeat the training steps multiple times to obtain multiple loss values; find the minimum total loss value from the multiple loss values, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-sample industrial defect detection model, which are denoted by W best and b best ;Complete the training of zero-shot industrial defect detection model;

[0045] The step of inputting the test set into the trained zero-sample industrial defect detection model for testing includes:

[0046] The test set is input into the trained zero-shot industrial defect detection model, and the W best and b best Make a prediction and get the defect segmentation image M corresponding to the test set p and defect classification C p , to achieve zero-sample industrial defect detection.

[0047] The second technical solution adopted by the present invention is:

[0048] An electronic device comprises a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a text-guided zero-sample industrial defect detection method as described above.

[0049] The third technical solution adopted by the present invention is:

[0050] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a text-guided zero-sample industrial defect detection method as described above.

[0051] The fourth technical solution adopted by the present invention is:

[0052] A computer program product or a computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned text-guided zero-sample industrial defect detection method.

[0053] The beneficial effect of the present invention is that the present invention makes full use of the prior knowledge contained in the pre-trained multimodal large model CLIP, so that the model can adaptively learn common text prompts for different objects, thereby realizing zero-sample defect detection for various industrial products. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 It is a flowchart of a text-guided zero-sample industrial defect detection method in an embodiment of the present invention;

[0056] Figure 2 It is a structural block diagram of a zero-shot industrial defect detection model based on text guidance in an embodiment of the present invention;

[0057] Figure 3 is a schematic diagram of a framework of a text-guided zero-shot industrial defect detection model in an embodiment of the present invention;

[0058] Figure 4 It is a flowchart of the steps of a text-guided zero-sample industrial defect detection method in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.

[0061] In the description of the present invention, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed" etc. are understood as not including the number itself, and "above", "below", "within" etc. are understood as including the number itself. If there is a description of "first" or "second", it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0062] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0063] Terminology explanation:

[0064] CLIP: Contrastive Language–Image Pre-training, the CLIP model is a multimodal training model that learns the matching relationship between images and texts by comparing their vector representations.

[0065] Example 1

[0066] like Figure 4 As shown, this embodiment provides a text-guided zero-sample industrial defect detection method, comprising the following steps:

[0067] S1. Obtain images and text semantic labels of industrial products and build training and test sets;

[0068] S2. Construct a text-guided zero-shot industrial defect detection model, and use a training set to train the zero-shot industrial defect detection model; wherein the input of the zero-shot industrial defect detection model is an image to be segmented, and the output is a defect segmentation result and a classification result;

[0069] S3. Input the test set into the trained zero-shot industrial defect detection model for testing.

[0070] The process of the method of this embodiment is divided into two parts: training phase and testing phase. In the training phase, a zero-sample industrial defect detection model based on a pre-trained multimodal model is first constructed. By analyzing some existing types of industrial products and extracting their defect information, the model can learn unified text prompts applicable to a variety of industrial products. This process is not only aimed at improving the model's ability to identify known defects, but also focuses on capturing and summarizing defect features, so that it has the ability to migrate to unseen product categories.

[0071] During the testing phase, the trained model will be tested on industrial product categories that have never been seen before. When new data is input, the model will rely on the knowledge it has absorbed during the training phase to detect whether there are defects and accurately locate the location of the defects. This zero-sample detection capability means that the model no longer relies on the need for a large number of labeled samples in traditional methods, but can make independent judgments and analyses for a completely new product category based on previously learned text prompts and features.

[0072] In this way, the method of this embodiment can not only improve the flexibility and adaptability of industrial defect detection, but also significantly reduce the dependence on labeled data, reduce production and detection costs, and achieve more efficient quality control. The development of this technology will provide new solutions and technical support for the intelligent and automated industrial production, and help improve the overall quality and production efficiency of products.

[0073] The above method is explained in detail below with reference to the accompanying drawings and specific implementation methods.

[0074] (1) Constructing training and test sets

[0075] See also Figure 1 , during the training and testing process, the image dataset is first preprocessed. The dataset includes original images of different types of industrial products, corresponding defect segmentation grayscale images, and text semantic labels. Exemplarily, the mainstream industrial defect detection datasets MVTec and VisA are selected, and there is no duplication in the types of industrial products in the two datasets. MVTec and VisA are used as image training sets and test sets respectively. The same preprocessing steps are performed on the two datasets. Taking the MVTec dataset as an example, the preprocessing steps of the dataset include image adjustment, defect segmentation map adjustment, and text semantic label processing.

[0076] Image adjustment: For each image in the dataset, let’s assume that the nth image is I n , using bilinear interpolation, image I n The size of the image is resized to C*H*W, where C represents the number of channels of the image (such as RGB images have 3 channels), and H and W represent the height and width of the image respectively. This step ensures that all images have consistent sizes, which is convenient for subsequent model processing.

[0077] Defect segmentation map adjustment: For image I n The corresponding defect segmentation grayscale image is denoted as M n , and its size is also adjusted to 1*H*W by bilinear interpolation. Here "1" represents a single channel of the grayscale image. The grayscale image represents the location information of the defect. If an image is a defect-free sample, its corresponding defect segmentation grayscale image is recorded as a full-0 tensor of size 1*H*W, indicating that there is no defect.

[0078] Text semantic label processing: For image I n The corresponding text semantic labels are recorded as Anomaly (defect) and Normal (normal). In order to facilitate model processing and mathematical calculations, the Anomaly label is recorded as 1 and the Normal label is recorded as 0. These label information are finally recorded as C n , will be used together with the image data for training and testing.

[0079] (2) Build and train a text-guided zero-shot industrial defect detection model

[0080] See also Figure 2 and Figure 3 In this embodiment, a text-guided zero-shot industrial defect detection model is constructed and trained, which can effectively detect and locate defects in industrial products. The input of the model is the image to be segmented I n , the output is the defect segmentation result grayscale image M n ′ And the model classification result C n ′ The model structure includes an input layer, a hidden layer, and an output layer, wherein the input layer receives the image to be segmented I n , providing raw data for subsequent feature extraction and processing; the hidden layer is further subdivided into CLIP pre-training module, learnable prompt module, adaptive prompt module, image-text feature interaction module and segmentation and classification task module to achieve complex data processing and feature extraction. These modules work together to extract and fuse features from images and text to achieve accurate defect detection; the output layer outputs the defect segmentation result grayscale image M n ′ And the model classification result C n ′ , respectively representing the location and classification information of the defect.

[0081] In some embodiments, the CLIP pre-trained model module adopts the ViT-L / 14@336px version, including a pre-trained image encoder and a text encoder, respectively denoted as E I and E T . Image I n Input image encoder E I Encode and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4}th layer {F P1 ,F P2 ,F P3 ,F P4} as the image defect segmentation feature FP .

[0082] In some embodiments, the learnable hint module includes a plurality of learnable text hint tensors and a CLIP pre-trained text encoder E T The learnable text hint tensor is divided into defect hint P A and non-defect hint P N , where P A ∈R na*d , P N ∈R nn *d , na and nn represent the number of defect cues and non-defect cues respectively, d represents the length of each learnable cue, and d is related to E T The length of the tensor after the text embedding in P is consistent. A and P N Parallel Input E T As the embedded tensor, we get the classification features F of multiple defect prompts. A and the classification features F of non-defect hints N , where F A ∈R na*f , F N ∈R nn*f , f represents the length of each learnable prompt feature, f and text encoder E T The output feature dimensions are consistent.

[0083] In some embodiments, the adaptive prompt module S is composed of a Meta module and a multi-layer perceptron MLP, which is mainly used to adaptively adjust the prompt features according to the image features to improve the effect of defect detection. The design purpose of this module is to enable the model to better understand and identify defects in the image by integrating image features and prompt features.

[0084] The input of Meta module is image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI, the dimension of the prompt feature before and after the module remains unchanged. The main function of the Meta module is to integrate the image information into the prompt feature, so as to obtain the prompt feature that matches the input image and improve the defect detection effect. Specifically, the first step of the processing is to add the image features as biases to the classification features of defect prompts and non-defect prompts. This process helps to emphasize the key features and contextual information of the image during feature fusion, so that the model can better capture various prompts related to the image. Next, these adjusted features will be sent to the linear layer to further fuse the image features with the prompt features. This design ensures that the interactive relationship between the two features can be unfolded, so that the final output features can better reflect the actual situation and defects of the objects in the image.

[0085] The second part of the adaptive prompt module S is a multi-layer perceptron MLP, which receives the updated classification features F of multiple defect prompts. AI and the classification features F of non-defect hints NI After that, further processing is performed. The main output of the MLP module is the classification feature F of a single defect prompt. AT and the classification features F of non-defect hints NT , where F AT ∈R 1*f , F NT ∈R 1*f Such an output structure enables the model to flexibly select the most suitable prompt features for the image based on the specific input image.

[0086] The multi-layer perceptron MLP is constructed by AT and F NT The combination and conversion of further optimizes the prompt features related to defects. In the implementation process, MLP can process and calculate the prompt features from multiple dimensions, so that the model can show stronger adaptability and recognition ability when facing different types of defects. Ultimately, the design goal of the adaptive prompt module S is to effectively match the prompt features, so that the model is not only more accurate in defect detection tasks, but also has stronger generalization ability.

[0087] Through this adaptive adjustment, the adaptive prompt module S takes the actual objects and defect features in the image into full consideration, improving the model's ability to detect different defects and ensuring high efficiency and accuracy in complex industrial environments. This design undoubtedly provides strong technical support for the model's intelligent defect detection.

[0088] In some embodiments, the image-text feature interaction module C is composed of multiple cross-attention modules, which, as an important component of the model, plays a key bridge role. The input of this module includes the image classification feature F I, image defect segmentation feature F P , classification features of a single defect hint F AT and the classification features F of non-defect hints NT Together, these input features provide rich information, enabling the model to gain a deeper understanding of the context of defects in the image and their relationship with the cue features.

[0089] Specifically, the image classification feature F I It contains the overall understanding of the image to be detected, reflecting the basic features and background information of the objects in the image; and the image defect segmentation feature F P It focuses on the details in the image and provides detailed information about the specific location and shape of the defect. AT and the classification features F of non-defect hints NT This provides the model with additional semantic guidance on whether a defect exists or not, helping the model to take into account the information carried by the textual prompt when making judgments.

[0090] Through the interaction of these input features, the output of this module includes the reconstructed image classification features F I ′ and image defect segmentation feature F P ′ . The dimensions of these output features are consistent with the input features, which ensures the consistency of information transfer and conversion dimensions during feature processing, thereby avoiding unnecessary errors in subsequent module processing. The main function of this module is to strengthen the connection between prompt features and image features, and further enhance the model's understanding ability. In the defect detection task, the relationship between image and text is effectively integrated, so that the model can more accurately distinguish between defects and non-defective areas. This not only improves the model's sensitivity to defect characteristics, but also enhances its adaptability when facing complex backgrounds or diversified products. Through such a feature interaction mechanism, the model can capture the key information in the image more comprehensively, making the final defect detection results more accurate and reliable.

[0091] In some embodiments, the segmentation and classification task module T is composed of a segmentation task module and a classification task module. This module is responsible for performing detailed segmentation and accurate classification of defects in the input image to achieve a comprehensive defect detection function.

[0092] In the segmentation task module, the input includes image defect segmentation features F P ′ , classification features of a single defect hint F AT and the classification features F of non-defect hints NT The module analyzes these input features to identify defective areas in the image and perform corresponding segmentation. Specifically, the image defect segmentation feature FP ′ is responsible for providing regional features of potential defects in the image, while the classification features F AT and the classification features F of non-defect hints NT The output of this module includes the model segmentation result M corresponding to the defect prompt. A The model segmentation result M corresponding to the non-defect prompt N , M A ,M N ∈R C*H*W , respectively representing the prediction results of defective areas and non-defective areas. According to the above structure, the segmentation task module can effectively separate defective areas and non-defective areas from the background by special processing of features, laying the foundation for subsequent classification and other processing tasks.

[0093] The classification task module is responsible for analyzing the image to determine whether it has defects and classify it. In this module, the input is the processed image classification feature F I ′ , classification features of a single defect hint F at and the classification features F of non-defect hints NT The combination of these input features can fully reveal the content and state of the image, allowing the possibility of individual defects to be evaluated. The output of the module is the probability P of a defect in the image. A and the probability P that there is no defect in the image N , reflects the model’s judgment on image defects. Specifically, P A and P N The output format is P A ,P N ∈R 1*1 , respectively represent the probability of defects appearing in the image and the probability of defects not existing. By counting and calculating probabilities, the model can clearly distinguish whether there are defects in the image and perform subsequent processing in a targeted manner based on the output results.

[0094] As an optional implementation, the zero-shot industrial defect detection model works as follows:

[0095] First, the original image is input into a pre-trained image encoder to obtain image features; the learnable prompt is input into a pre-trained text encoder to obtain multiple text features corresponding to defects and non-defects;

[0096] Input multiple text prompt features and image features into the adaptive prompt module to obtain text prompt features that are adapted to the input image as text prompt features that participate in subsequent calculations of the model;

[0097] The text prompt features and image features are sent to the image-text feature interaction module to obtain the image features after the prompt information is integrated, which are used as the image features involved in the subsequent calculation of the model;

[0098] By sending the image features and text prompt features into the classification task module and the segmentation task module, the predicted classification results and the predicted segmentation results can be obtained.

[0099] (3) Loss Function

[0100] During the model training process, defining and using appropriate loss functions is the key to ensuring overall improvement of model performance. The segmentation loss function of the model uses Focal Loss, Dice Loss, and Cross Entropy Loss. The combination of these loss functions can improve the model's processing ability for segmentation tasks from different angles and ensure accurate identification of defect areas. For segmentation loss, M A , M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M N ,M A ],M),L Dice (M A , M), L Dice (M N , 1-M), L CE ([M N ,M A ],M).

[0101] Focal Loss was first applied to target detection tasks. It was proposed mainly to solve the problem of imbalance in the number of positive and negative samples and the imbalance in the number of difficult and easy samples. Focal is defined as follows:

[0102] L Focal (p t )=-α t (1-p t ) γ log(p t )

[0103] Among them, p t Represents the probability of the model predicting a certain category, α t Used to balance the number of positive and negative samples, giving a smaller α to samples with more samples t Value, the smaller the number of samples, the larger α is given t The value γ is used to adjust the imbalance problem of difficult and easy samples. Generally, γ≥1 is taken to reduce the loss of easy samples, so that the model pays more attention to difficult samples.

[0104] Dice Loss is a commonly used loss function that handles foreground-background imbalance. It reduces the impact of the background area by calculating the intersection ratio of the predicted and true labels. Its mathematical expression is as follows:

[0105]

[0106] Where |X∩U| represents the number of elements in the intersection of X and Y, and |X| and |Y| represent the number of elements in X and Y, respectively.

[0107] Cross entropy loss is often used in binary and multi-classification problems. This loss is sensitive to the probability prediction results and can effectively encourage the predicted probability to be close to the true label. Its expression is as follows:

[0108]

[0109] in, represents the predicted label and y represents the true label.

[0110] The classification loss function uses the cross entropy loss function. For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

[0111] Repeat the training steps Q times to obtain the zero-sample industrial defect detection training model, and obtain Q*N loss values ​​in total. Find the minimum total loss value, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-sample industrial defect detection model, which are recorded as W best and b best ; That is, the training of the zero-sample industrial defect detection model is completed.

[0112] (4) Model testing

[0113] The test set is input into the trained model for testing. The preprocessed data is input into the trained zero-shot industrial defect detection model. best and b best Make a prediction and get the defect segmentation image M corresponding to the test set p and defect classification C p , zero-shot industrial defect detection can be achieved. Optionally, AUROC and AUPRO are used as segmentation indicators, and AUROC and AP are used as classification indicators to verify the zero-shot defect detection effect of the text-guided zero-shot industrial defect detection method.

[0114] In summary, the method of the present invention has at least the following advantages and beneficial effects compared to the prior art:

[0115] 1) Zero-sample learning capability. Traditional industrial defect detection models usually require a large amount of labeled data for training, while the present invention can accurately detect and classify new defects based on text prompts without direct contact with a certain sample or similar defect samples. This capability greatly reduces the data dependence when applying the model and improves the applicability and flexibility of the model.

[0116] 2) Efficient defect detection capability. The present invention achieves efficient segmentation and accurate classification of defect areas by combining a pre-trained multimodal model with an efficient network structure. The model can capture the difference between the model prediction and the true label from different angles and continuously optimize its performance during the training process. Therefore, compared with the prior art, the present invention has significantly improved the accuracy of defect segmentation and the precision of classification.

[0117] 3) Strong generalization ability. This invention adopts a text-based prompt method, which allows the model to learn the general prompt features of defects under the guidance of the pre-trained multimodal model and training set, so that the model can show strong generalization ability when facing complex and changeable industrial scenarios. This means that even if new defect types that are somewhat different from the training data are encountered in actual applications, the model can quickly adapt and give accurate detection results.

[0118] 4) Reduce manual labeling and enhance algorithm universality. The development of existing industrial defect detection systems relies on a large amount of manual labeling. The algorithm design is highly targeted, has poor universality, and has weak ability to dynamically adapt to scenarios. This invention will provide new ideas for the development of defect detection systems by combining pre-trained large models with anomaly detection technology.

[0119] 5) Efficiently utilize a small number of samples to reduce promotion costs. The zero-sample / few-sample industrial defect detection framework will target common problems in actual application scenarios and can be promoted to a variety of industrial product quality inspection processes at a low cost and high efficiency.

[0120] 6) Enhance model migration capabilities and adapt to different scenarios. Another challenge in deploying detection models in actual application scenarios is the domain adaptation problem, that is, there is a deviation between training data and actual detection data. It is necessary to study model migration and scenario adaptation.

[0121] 7) Enhance the generalization ability of the model with the help of pre-trained large models. Large models have high prediction accuracy, strong transfer learning and domain adaptability, low dependence on labeled data, and strong multimodal processing capabilities, which will provide new possibilities for industrial defect detection.

[0122] Example 2

[0123] An embodiment of the present invention further provides an electronic device, the electronic device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the following Figure 4 A text-guided zero-shot industrial defect detection method is shown.

[0124] It is understood that the memory may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory includes a non-transitory computer-readable storage medium. The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the server, etc.

[0125] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect the various parts of the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor can be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor can integrate one or a combination of a central processing unit (CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a chip.

[0126] Since the electronic device is an electronic device corresponding to a text-guided zero-sample industrial defect detection method of an embodiment of the present invention, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0127] Example 3

[0128] The embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the following Figure 4 A text-guided zero-shot industrial defect detection method is shown.

[0129] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0130] Since the storage medium is a storage medium corresponding to a text-guided zero-sample industrial defect detection method of an embodiment of the present invention, and the principle of solving the problem by the storage medium is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0131] Example 4

[0132] In some possible implementations, various aspects of the method of the embodiment of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of a text-guided zero-sample industrial defect detection method according to various exemplary embodiments of the present application described above in this specification. Among them, the executable computer program code or "code" for executing various embodiments may be written in a high-level programming language such as C, C++, C#, Smalltalk, Java, JavaScript, Visual Basic, structured query language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0133] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0134] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0135] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable ordinary technicians in the field to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made based on the essence of the content of the present invention should be included in the protection scope of the present invention.

Claims

1. A text-guided zero-shot industrial defect detection method, characterized in that: The following steps are involved: Obtain images and text semantic labels of industrial products and build training and test sets; Constructing a text-guided zero-shot industrial defect detection model, and using a training set to train the zero-shot industrial defect detection model; wherein the input of the zero-shot industrial defect detection model is an image to be segmented, and the output is a defect segmentation result and a classification result; The test set is input into the trained zero-shot industrial defect detection model for testing; Among them, during the training stage, the zero-shot industrial defect detection model uses the defect information in some types of industrial products to learn and expand to unified text prompts for other industrial products; during the testing stage, the zero-shot industrial defect detection model can be tested on types of industrial products that have never been trained, to detect whether there are defects and locate the defects.

2. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The method of obtaining images and text semantic labels of industrial products and constructing training sets and test sets includes: Get the preset data set and record the nth image in the data set as I n , through bilinear interpolation, the image I n The size is adjusted to C*H*W; where C, H, and W are the channel, height, and width of the image respectively; Image I n The corresponding defect segmentation grayscale image is denoted as M n , M n The size is adjusted to 1*H*W; For image I n The corresponding text semantic labels are recorded as Anomaly or Normal.

3. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The bilinear interpolation method is calculated as follows: In the formula, (src x , src y ) represents the original coordinates, (des x ,des y ) represents the target coordinates after interpolation, (src w , src h ) represents the width and height of the original image, (des w ,des h ) represents the width and height of the interpolated target image.

4. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The zero-shot industrial defect detection model includes an input layer, a hidden layer and an output layer; the hidden layer includes a CLIP pre-training module, a learnable prompt module, an adaptive prompt module, an image-text feature interaction module, and a segmentation and classification task module; The working method of the CLIP pre-trained model module is as follows: the pre-trained image encoder and text encoder of CLIP are respectively denoted as E I and E T ; Image I n Input image encoder E I , and get the output feature F I As image classification features, from E I Get the intermediate layer feature list of the {l1,l2,l3,l4}th layer {F P1 ,F P2 ,F P3 ,F P4 } as the image defect segmentation feature F P ; The learnable prompt module includes a plurality of learnable text prompt tensors and a CLIP pre-trained text encoder E T ; The learnable text prompt tensor is divided into defect prompt P A and non-defect hint P N ; P A and P N Parallel Input E T As the embedded tensor, Get the classification features F of multiple defect prompts A and the classification features F of non-defect hints N ; The adaptive prompt module consists of a Meta module and a multi-layer perceptron; the input of the Meta module is the image feature F I , classification features of multiple defect prompts F A and the classification features F of non-defect hints N , the output is the classification feature F of multiple defect hints associated with the input image AI and the classification features F of non-defect hints NI ; The input of the multi-layer perceptron is the updated classification features F of multiple defect prompts AI and the classification features F of non-defect hints NI , the output is the classification feature F of a single defect prompt AT and the classification features F of non-defect hints NT ; The input of the image-text feature interaction module is the image classification feature F I , image defect segmentation feature F P , classification features of a single defect hint F AT and the classification features F of non-defect hints NT , the output is the reconstructed image classification feature F I ′ , image defect segmentation feature F P ′ ; The segmentation and classification task module is composed of a segmentation task module and a classification task module. In the segmentation task module, the input of the module is the image defect segmentation feature F P ′ , classification features of a single defect hint F AT and the classification features F of non-defect hints NT The output is the model segmentation result M corresponding to the defect prompt and non-defect prompt A and M N ; In the classification task module, the input of the module is the image classification feature F I ′ , classification features of a single defect hint F AT and the classification features F of non-defect hints NT , the output is the probability P of defects in the image A and the probability P that there is no defect in the image N .

5. The text-guided zero-sample industrial defect detection method according to claim 4, characterized in that: The learnable hint module simultaneously learns multiple defect hints and non-defect hints, which focus on the defects in the image but not the types of objects in the image. The hint feature dimensions are similar to the CLIP pre-trained text encoder E T The output feature dimensions are consistent.

6. The text-guided zero-sample industrial defect detection method according to claim 4, characterized in that: The function of the adaptive prompt module is to adaptively select the prompt feature that best suits the image based on the features of the object and defect in the image, so as to improve the model's ability to detect different defects; The Meta module includes a linear layer. The function of the Meta module is to integrate image information into the prompt feature, so as to obtain the prompt feature matching the input image to improve the effect of defect detection; The multilayer perceptron includes three linear layers and an activation function; The image-text feature interaction module is composed of several cross-attention modules, and the output feature dimension is consistent with the input feature dimension; the role of the image-text feature interaction module is to further strengthen the connection between the prompt feature and the image feature, so as to facilitate the model to distinguish between defective and non-defective areas.

7. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: In the process of training the zero-shot industrial defect detection model, the segmentation loss function uses Focal Loss, Dice Loss and Cross Entropy Loss, and the classification loss function uses Cross Entropy Loss; For the segmentation loss, the segmentation results M are calculated separately. A and M N The loss function value between the actual defect segmentation grayscale image M: L Focal ([M n ,M A ],M)、L Dice (M A ,M)、L Dice (M N ,1-M)、L CE ([M N ,M A ],M); For classification loss, calculate P A , P N The cross entropy loss value L directly with the true label C CE ([P N ,P A ],C).

8. The text-guided zero-sample industrial defect detection method according to claim 1, characterized in that: The adopting of the training set to train the zero-sample industrial defect detection model includes: Repeat the training steps multiple times to obtain multiple loss values; find the minimum total loss value from the multiple loss values, and use the weight and bias term corresponding to the minimum total loss value as the optimal weight vector and optimal bias term corresponding to the zero-sample industrial defect detection model, which are denoted by W best and b best ;Complete the training of zero-shot industrial defect detection model; The step of inputting the test set into the trained zero-sample industrial defect detection model for testing includes: The test set is input into the trained zero-shot industrial defect detection model, and the W best and b best Make a prediction and get the defect segmentation image M corresponding to the test set p and defect classification C p , to achieve zero-sample industrial defect detection.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

Images