Surface defect small sample detection method based on contrast learning

Abstract

The invention relates to a product surface defect small sample detection method based on comparative learning. The method comprises the following steps: collecting a defect image sample and carrying out image preprocessing; based on an improved image language comparison pre-training model CLIP, establishing a surface defect detection model; selecting a defect image sample to carry out finetune training on the model; and inputting a defect image to be detected into the trained surface defect detection model, and outputting a defect detection result. According to the method, the strong generalization ability of the image language comparison pre-training model CLIP is fully utilized, high detection accuracy can be achieved only through a small number of defect samples, and the problems that the number of the defect samples is small, and large-scale collection is difficult are solved. According to the method disclosed by the invention, the Q-K-VAttention in the Vision-Transform feature extraction network is improved into the V-VAttention, so that the fine-grained feature extraction capability is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of defect detection, and in particular to a surface defect small sample detection method based on contrast learning. Background Art

[0002] As a common quality problem, surface defects are widely present in products made of materials such as metals, ceramics, plastics and textiles, and are one of the key issues that need to be urgently addressed in product quality control.

[0003] At present, there are three main methods for the most common surface defect detection based on visual systems: ① Detection methods based on artificial vision, which mainly refer to experienced workers using their eyes to observe and detect defects during the production process. This method relies on professional knowledge or skilled experience, has low production efficiency and is subjective and arbitrary. ② Detection methods based on traditional image processing, which mainly rely on manually designed feature extractors, including texture features, color features, and shape features of images. This method requires professional knowledge and a complex parameter adjustment process, and each method is targeted at specific application scenarios, with poor generalization ability and robustness. ③ Detection methods based on deep learning, which mainly include supervised methods, unsupervised methods, and weakly supervised methods. Among them, only supervised methods can better meet the accuracy requirements of surface defect detection of industrial products, and are currently the most widely used methods. However, most of the deep learning algorithms based on supervised learning currently require a large amount of manually labeled data, which is a labor-intensive and financially intensive process, and the models trained in this way have weak generalization ability. Once new detection needs arise, data needs to be collected, labeled, and trained again. In addition, most production lines still produce mostly normal products, defective products are a minority in production, the number of samples is small, and it is difficult to collect a large amount of valid data. Summary of the invention

[0004] In view of this, it is necessary to provide a small sample detection method for surface defects based on contrastive learning to solve the above-mentioned defects of the prior art.

[0005] In order to solve the above problems, in a first aspect, an embodiment of the present invention provides a surface defect small sample detection method based on contrastive learning, comprising:

[0006] Step S1, collecting defect image samples, annotating defect detection mask images and defect categories, and performing image preprocessing on the annotated defect image samples to obtain a defect image data set;

[0007] Step S2, establishing a surface defect detection model based on the improved image language comparison pre-training model CLIP;

[0008] Step S3, selecting defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model;

[0009] Step S4, inputting the defect image to be detected into the trained surface defect detection model, and outputting the defect detection result.

[0010] Preferably, in step S1, the image preprocessing of the labeled defect image samples includes:

[0011] The collected defect image samples are scaled to a fixed size through quadratic interpolation;

[0012] A regularized data augmentation method is used to randomly perturb the brightness, contrast, and sharpness of the image to simulate the image style under the lighting changes and conveyor belt speed changes in the production workshop.

[0013] Preferably, in step S2, the surface defect detection model includes:

[0014] Vision-Transformer feature extraction network, where Transformer is used as a text feature generation network, and a VV attention mechanism is embedded in a specific layer of the Vision-Transformer feature extraction network to replace the original QKV attention mechanism of the CLIP model;

[0015] The text encoder uses a 12-layer Transformer block stack to accept fine-grained text generated by a preset domain expert module, combines the fine-grained text descriptions to generate tokens, and obtains the corresponding text feature embedding; where a token refers to a single discrete unit in a fine-grained text description;

[0016] The post-processing module includes a coarse detection unit and a fine detection unit. The coarse detection unit is used to calculate the image-level similarity score using the overall image features to determine whether the image has defects; the fine detection unit is used to compare the positive and negative properties of each pixel using the pixel-level similarity score of the fine-grained local features, generate a defect feature map, and realize defect area positioning and segmentation.

[0017] Preferably, the step S3 specifically includes:

[0018] Step S31, selecting defect image samples from the defect image dataset as fine-tuning input, using the multi-classification cross entropy loss function as the loss function of the rough detection process for training, and saving the model parameters;

[0019] Step S32: Select the target detection network for precise detection, use the defect image after image preprocessing as the input of the target detection network, train and save the model parameters; select the dynamically scaled cross entropy loss FocalLoss and the binary classification loss BinaryDiceLoss of image segmentation as the loss function of precise detection.

[0020] Preferably, in step S31, the expression of the multi-classification cross entropy loss function is:

[0021]

[0022] In the formula, Loss is the loss function, y is the discriminant probability distribution of the surface defect recognition network for the input image x, and label represents the actual defect category of the input image x;

[0023] In step S32, the expression of the dynamically scaled cross entropy loss FocalLoss is:

[0024] FocalLoss(p t )=-α t (1-p t ) y log(p t )

[0025] In the formula, FocalLoss(p t ) represents the dynamically scaled cross entropy loss function, p t is the model’s predicted probability for the true category, α t is a balancing factor used to adjust the importance of different categories, and the superscript y is a focusing parameter that controls the degree of attention paid to difficult and easy samples;

[0026] The expression of BinaryDiceLoss for image segmentation is:

[0027]

[0028] In the formula, X is the predicted probability value of each pixel, and Y is the true value.

[0029] Preferably, before step S4, the method further comprises:

[0030] The defect image to be detected is collected by an industrial camera in real time, and is scaled to a fixed size through quadratic interpolation as the input of the surface defect detection model.

[0031] Preferably, in step S4, the step of inputting the defect image to be detected into a trained surface defect detection model and outputting the defect detection result includes:

[0032] The defect image to be detected is input into the trained surface defect detection model, and the probability of the defect image belonging to each defect category is obtained through rough detection. It is judged whether the maximum value of the defect category probability is greater than the preset classification probability threshold. If not, it is judged as defect-free; if so, the defect image is input into the target detection network for fine detection;

[0033] The target detection network outputs the defect category confidence of the defect image and determines whether the defect category confidence is greater than the preset confidence threshold. If not, it is judged as defect-free; if so, the defect category and defect detection box are obtained.

[0034] In a second aspect, an embodiment of the present invention provides a surface defect small sample detection system based on contrastive learning, comprising:

[0035] The sample collection module is used to collect defect image samples, annotate defect detection mask images and defect categories, and perform image preprocessing on the annotated defect image samples to obtain a defect image data set;

[0036] Model building module, used to build a surface defect detection model based on the improved image language comparison pre-training model CLIP;

[0037] The model training module is used to select defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model;

[0038] The defect detection module is used to input the defect image to be detected into the trained surface defect detection model and output the defect detection result.

[0039] In a third aspect, the present invention further provides an electronic device, including a memory and a processor, wherein:

[0040] The memory is used to store programs;

[0041] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the small sample detection method for surface defects based on contrastive learning as described in the embodiment of the first aspect of the present invention.

[0042] In a fourth aspect, the present invention further provides a computer-readable storage medium for storing computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the surface defect small sample detection method based on contrast learning as described in the embodiment of the first aspect of the present invention.

[0043] Compared with the prior art, the surface defect small sample detection method based on contrastive learning provided by the present invention has at least the following beneficial effects:

[0044] ① The present invention establishes a surface defect detection model based on the improved image language comparison pre-training model CLIP. By making full use of the powerful generalization ability of the image language comparison pre-training model, only a small number of actual defect samples are needed to achieve a high detection accuracy, solving the problem of a small number of defect samples and difficulty in large-scale collection.

[0045] ② The present invention adopts a regularized data enhancement method to randomly jitter the brightness, contrast and saturation of the image respectively to simulate the possible interference under the operating conditions of the production line, so that the network model can obtain generalization and robustness under changes in illumination and speed after good learning.

[0046] ③The present invention is based on the CLIP network structure and improves the QK-V Attention in the original Vision-Transformer feature extraction network into V-V Attention, which solves the incorrect association between background redundant features and semantic features. At the cost of a small increase in parameters and a slight decrease in speed, it guides the network to focus on defect information and weaken irrelevant information, enhances the network's feature extraction capability for defect features, and improves the detection accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flow chart of a small sample surface defect detection method based on contrastive learning provided by the present invention;

[0048] Figure 2 A schematic diagram of a small sample surface defect detection method based on contrastive learning provided by the present invention;

[0049] Figure 3 A schematic diagram of the structure of the Vision-Transformer feature extraction network provided by the present invention;

[0050] Figure 4 A schematic diagram of a flow chart of providing prior information for the text encoder provided by the present invention;

[0051] Figure 5 A schematic diagram of the rough detection and fine detection process provided by the present invention;

[0052] Figure 6 A structural block diagram of a surface defect small sample detection system based on contrastive learning provided by the present invention;

[0053] Figure 7 This is a structural block diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0055] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] At present, in the industrial production process, most production lines still produce normal products, and defective products are a minority in production. The number of samples is small, making it difficult to collect a large amount of effective data. In addition, traditional defect detection methods based on supervised learning require a large amount of manually labeled data, which is time-consuming and labor-intensive.

[0057] In view of this, the present invention provides a surface defect small sample detection method based on contrastive learning, and establishes a surface defect detection model based on the improved image language contrast pre-training model CLIP. By making full use of the powerful generalization ability of the image language contrast pre-training model, the model can achieve higher accuracy in downstream defect detection tasks by fine-tuning with only a small number of actual defect samples. The following will be explained and introduced through multiple embodiments.

[0058] Figure 1 A flow chart of a small sample surface defect detection method based on contrastive learning provided by the present invention, Figure 2 The schematic diagram of the surface defect small sample detection method based on contrast learning provided by the present invention; Figure 1 As shown, the surface defect small sample detection method based on contrastive learning includes at least steps S1 to S4, wherein:

[0059] Step S1, collecting defect image samples, and marking defect detection mask images and defect categories, performing image preprocessing on the marked defect image samples to obtain a defect image data set.

[0060] In step S1, first, the collected defect image samples are scaled to a fixed size through quadratic interpolation. Then, the brightness, contrast and sharpness of the image are randomly perturbed using a regularized data enhancement method to simulate the image style under the lighting changes and conveyor belt speed changes in the production workshop.

[0061] In actual operation, it can be assumed that the size of the collected defect image sample is h*w. First, the original image is scaled to 336*336 by quadratic interpolation. Secondly, let the brightness factor, contrast factor and sharpness factor be b, c and s respectively. First, let b, c and s take random values ​​between [0.9, 1.1], and the values ​​of these three random numbers are independent of each other. For b, c and s, when the value is 1, it means the original image, when the value is less than 1, it means weakening, and when the value is greater than 1, it means enhancement, and the amplitude of the change is controlled by the numerical value. For example, for the defect image O, b greater than 1 means that O becomes brighter, and b less than 1 means that O becomes darker, and the contrast factor c and sharpness factor s have the same effect. Therefore, for the defect image O, b, c and s are used to slightly perturb its image properties. The way to achieve slight perturbation of image properties includes but is not limited to providing corresponding APIs by image processing libraries such as PIL and OpenCV-Python under Python language to achieve the adjustment of image properties.

[0062] The embodiment of the present invention adopts a regularized data enhancement method to randomly perturb the brightness, contrast and sharpness of the image, simulating the image style under the lighting changes in the production workshop and the conveyor belt speed changes, which can enhance the anti-interference ability of the network model, so that the network model can obtain generalization and robustness under lighting changes and speed changes after good learning.

[0063] Step S2, establishing a surface defect detection model based on the improved image language comparison pre-training model CLIP.

[0064] Among them, CLIP (Contrastive Language–Image Pre-training) is a powerful multimodal learning model that can learn rich visual and semantic information from massive image-text pairs. However, the original CLIP model may not be fully applicable to the surface defect detection task of specific industrial products (such as bottled products) because it is mainly trained based on overall image-level features, while the defects on the surface of bottled products are often regional-level features.

[0065] In view of this, the CLIP model is improved in this implementation to make it more suitable for the detection of surface defects of industrial products. Specifically, the surface defect detection model established in the embodiment of the present invention retains the basic structure of the image encoder and text encoder in the CLIP model, but optimizes the attention mechanism in the image encoder and introduces the VV attention module to better focus on fine-grained features and reduce the impact of background redundant information. At the same time, this embodiment also introduces a domain expert module for product surface defects in the text encoder, which provides the model with rich prior knowledge, such as the shape, area, color, etc. of the defects, to further improve the detection accuracy of the model.

[0066] In a preferred embodiment of the present invention, referring to Figure 2 to Figure 4 ,The surface defect detection model can be composed of the Vision-Transformer feature extraction network, text encoder and post-processing module.

[0067] First, refer to Figure 3 , the Vision-Transformer feature extraction network is described in detail:

[0068] Vision Transformer (ViT) is a deep learning model based on the Transformer architecture, which is specially used to process image data and performs well in tasks such as image recognition, object detection, and image segmentation. This example uses Vision Transformer (ViT) to extract image features of defect images. Figure 2 and Figure 3 , the Vision-Transformer feature extraction network is used as the image encoder. As an optional implementation, ResNet50 can also be used as the feature extraction network.

[0069] In this embodiment, Transformer is used as a text feature generation network. Figure 2 , the VV attention mechanism (Attention) is embedded in the specific layers (Layer6~Layer24) of the Vision-Transformer feature extraction network to replace the original QKV attention mechanism of the CLIP model. Figure 2 V-VAttention includes multiple V vectors as input, as well as a linear layer, a Scale layer, a Softmax layer, and a Dropout layer.

[0070] It can be understood that in the traditional QK-VAttention, Q (Query) represents the query vector, K (Key) is the key vector, and V (Value) is the value vector. In this embodiment, the innovative V-VAttention is used to replace the traditional QK-VAttention, which can significantly improve the ability to enhance fine-grained feature extraction. V-VAttention focuses on using only the V vector for attention calculation. This simplified process enables the model to focus more directly and finely on the fine-grained features in the image. By reducing the calculation of Q vectors and K vectors, V-VAttention can more effectively capture the correlation between image blocks, thereby extracting richer fine-grained features. At the same time, V-VAttention also significantly reduces the impact of background redundant features. The traditional QK-VAttention mechanism may introduce background noise due to the interaction of Q and K vectors, while V-VAttention effectively ignores this redundant information by focusing on the V vector, allowing the model to focus more clearly on the key fine-grained features in the image. In addition, V-VAttention also narrows the gap between image-level features and region-level features. By paying more attention to local fine-grained features in the image, V-VAttention helps the model better understand the relationship between the overall structure of the image and local details, thereby reducing the gap between features at different levels. This improvement enables the model to more comprehensively capture and understand the information in the image when processing image tasks.

[0071] The present invention is based on the CLIP network structure and improves the QK-V Attention in the original Vision-Transformer feature extraction network into V-V Attention, which solves the incorrect association between background redundant features and semantic features. At the cost of a small amount of parameter increase and a slight speed reduction, the network is guided to focus on defect information and weaken irrelevant information, thereby enhancing the network's feature extraction capability for defect features and improving the detection accuracy of the model.

[0072] Figure 4 A schematic diagram of a priori information flow is provided for the text encoder provided by the present invention, referring to Figure 4 The text encoder is composed of 12 layers of Transformer blocks, where Transformer is a powerful sequence model that can capture long-distance dependencies and has been widely used in natural language processing. Each layer of Transformer blocks contains self-attention mechanisms and feed-forward neural networks, which enable the encoder to understand and generate complex text representations.

[0073] In this embodiment, the text encoder accepts fine-grained text generated by a preset domain expert module, combines the fine-grained text descriptions to generate tokens, and obtains corresponding text feature embedding; wherein a token refers to a single discrete unit in a fine-grained text description.

[0074] The domain expert module is a module specifically designed to understand product surface defects. It generates fine-grained text descriptions based on the defect features (such as shape, area, color, etc.) in the preprocessed data. These text descriptions are closely related to the defects in the image, thus providing rich prior information for the text encoder. The fine-grained text descriptions generated by the domain expert model are converted into a series of tokens. In this embodiment, tokens can represent the specific attributes of the defects (such as "cracks", "dents", etc.).

[0075] The text encoder takes these tokens as input and processes them through its 12-layer stack of Transformer blocks. The result of the processing is a text feature embedding that corresponds to the input image features. This embedding is a high-dimensional vector that contains all the important information of the text description.

[0076] In subsequent contrastive learning, this text feature embedding is compared with the features extracted from the image. By calculating the similarity between these two features, the surface defect detection model is able to determine whether there are defects in the defect image that match the text description.

[0077] Figure 5 The schematic diagram of the rough detection and fine detection process provided by the present invention is shown in FIG. Figure 2 and Figure 5 The post-processing module may include a coarse detection unit and a fine detection unit, wherein the coarse detection unit is used to calculate the image-level similarity score using the overall image features to determine whether the image has defects; the fine detection unit is used to compare the positive and negative properties of each pixel using the pixel-level similarity score of the fine-grained local features, and generate a defect feature map to realize defect area positioning and segmentation.

[0078] The present invention embeds a domain expert module before the text encoder to provide rich prior information description, narrow the difference between the image-level feature domain and the region-level feature, thereby improving the model's ability to locate and segment defects.

[0079] Step S3, selecting defect image samples from the defect image dataset to finetune the surface defect detection model.

[0080] In a preferred embodiment of the present invention, step S3 may specifically include the following steps S31-S32:

[0081] Step S31, select defect image samples from the defect image dataset as fine-tune input, use the multi-classification cross entropy loss function as the loss function of the rough detection process for training, and save the model parameters.

[0082] Specifically, fine-tuning refers to a retraining process based on a pre-trained model. In the embodiment of the present invention, fine-tuning training refers to further training the surface defect detection model based on the improved CLIP network structure using a small amount of actual defect samples to improve the performance of the surface defect detection model in the product surface defect detection task.

[0083] In this embodiment, defect image samples are selected from the defect image dataset as finetune input. The selected small dataset is required to contain 50 images of each type of defect, and these images should cover defects with different characteristics to ensure the comprehensiveness and diversity of the training. The CLIP model is trained using the selected dataset as a training set. During the training process, a multi-classification cross entropy loss function is selected as the loss function of the rough detection process. This loss function is used to measure the difference between the probability distribution predicted by the model and the actual defect category, thereby guiding the model to update parameters. After the training is completed, the model parameters are saved for use in subsequent detection tasks.

[0084] The expression of the cross entropy loss function for multi-classification is:

[0085]

[0086] In the formula, Loss is the loss function, y is the discriminant probability distribution of the surface defect recognition network for the input image x, and label represents the actual defect category of the input image x.

[0087] Step S32: Select the target detection network for precise detection, use the defect image after image preprocessing as the input of the target detection network, train and save the model parameters; select the dynamically scaled cross entropy loss FocalLoss and the binary classification loss BinaryDiceLoss of image segmentation as the loss function of precise detection.

[0088] Specifically, in step S31, the CLIP model is fine-tuned to obtain a coarse detection model that can roughly determine whether an image has defects. In order to locate the defect area more accurately, we also need an object detection network for fine detection to perform fine-grained detection. Therefore, in step S32, an object detection network is selected and trained using the defect image after image preprocessing as input.

[0089] For the training of the target detection network, this embodiment selects the dynamically scaled cross entropy loss FocalLoss and the binary classification loss BinaryDiceLoss of image segmentation as the loss function for precise detection. Among them, FocalLoss is a loss function designed for class imbalance in classification problems. It can enhance the model's attention to difficult-to-classify samples by dynamically scaling the loss of difficult and easy-to-classify samples. BinaryDiceLoss is a loss function for image segmentation tasks, which calculates the similarity between the predicted image and the real image based on the Dice coefficient. This embodiment combines these two loss functions to help the model locate defect areas more accurately in the precise detection stage.

[0090] The expression of the dynamically scaled cross entropy loss FocalLoss is:

[0091] FocalLoss(p t )=-α t (1-p t ) y log(p t )

[0092] Where α is used to balance the importance of positive / negative examples. t ) represents the dynamically scaled cross entropy loss function, p t is the model’s predicted probability for the true category, α t is a balancing factor used to adjust the importance of different categories, and the superscript y is a focusing parameter that controls the degree of attention paid to difficult and easy samples.

[0093] The expression of BinaryDiceLoss for image segmentation is:

[0094]

[0095] In the formula, X is the predicted probability value of each pixel, and Y is the true value.

[0096] Step S4, inputting the defect image to be detected into the trained surface defect detection model, and outputting the defect detection result.

[0097] Specifically, before step S4, it is necessary to collect defect images to be detected. In this embodiment, the defect images to be detected can be collected in real time by an industrial camera, and the defect images to be detected can be scaled to a fixed size by quadratic interpolation as input of the surface defect detection model.

[0098] In a preferred embodiment of the present invention, referring to Figure 2 and Figure 5 ,The specific process of defect detection includes:

[0099] First, the defect image to be detected is input into the trained surface defect detection model. The defect image to be detected is processed by the surface defect detection model and a discriminant probability distribution is obtained after a rough detection. The discriminant probability distribution represents the probability that the defect image belongs to each defect category. In this embodiment, the probability that the defect image belongs to each defect category is referred to as the "defect category probability". The "defect category probability" is Figure 2 The “image-level defect classification probability” shown in . “Defect category probability” also refers to Figure 5 Image-level similarity score.

[0100] Next, it is determined whether the maximum defect category probability is greater than a preset classification probability threshold. Preferably, the classification probability threshold can be set to 0.3. If not, that is, the maximum defect category probability is less than or equal to 0.3, it is determined to be defect-free; if yes, that is, the maximum defect category probability is greater than 0.3, it indicates that there may be defects in the image, and the defect image is input into the target detection network for fine detection for further fine detection.

[0101] Furthermore, for images that are determined to have possible defects in the rough detection, this embodiment sends them to the target detection network for more in-depth analysis. The target detection network will carefully classify each pixel in the image to determine whether it belongs to the defective part. In this process, loss functions such as FocalLoss and BinaryDiceLoss are used to optimize the detection performance of the network to ensure accurate identification of defective areas.

[0102] The target detection network outputs the defect category confidence of the defect image, and determines whether the defect category confidence is greater than a preset confidence threshold. Preferably, the confidence threshold can be set to 0.7. If not, that is, the defect category confidence is not greater than 0.7, then it is determined that the input image does not have defects; if so, that is, the defect category confidence is greater than 0.7, then the surface defect detection model determines that the image has defects, and outputs a specific defect category and a defect detection frame, and accurately locates the defect area through the defect detection frame. Therefore, the defect detection result output by step S4 includes the determination result of whether the image has defects, and the defect category and defect area output when it is determined that there are defects.

[0103] In this embodiment, by combining the two-stage method of coarse detection and fine detection, and reasonably setting the classification and detection thresholds, defects on the product surface can be accurately and efficiently identified and located, while ensuring detection accuracy and improving detection efficiency.

[0104] Figure 6 This is a structural block diagram of the surface defect small sample detection system based on contrastive learning provided by the present invention, referring to Figure 6The surface defect small sample detection system 600 based on contrastive learning includes:

[0105] The sample collection module 601 is used to collect defect image samples, annotate defect detection mask images and defect categories, and perform image preprocessing on the annotated defect image samples to obtain a defect image data set;

[0106] A model building module 602 is used to build a surface defect detection model based on the improved image language comparison pre-training model CLIP;

[0107] A model training module 603 is used to select defect image samples from the defect image data set to perform fine-tuning training on the surface defect detection model;

[0108] The defect detection module 604 is used to input the defect image to be detected into the trained surface defect detection model and output the defect detection result.

[0109] The surface defect small sample detection system based on contrastive learning provided by the present invention executes the surface defect small sample detection method based on contrastive learning provided by the above-mentioned embodiments through the above-mentioned modules. The surface defect small sample detection method based on contrastive learning has been described in detail in the above-mentioned embodiments, which will not be repeated in this embodiment.

[0110] Figure 7 The structural block diagram of the electronic device provided by the present invention is as follows: Figure 7 As shown, the present invention also provides an electronic device, and the electronic device 700 can be a computing device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device 700 includes a processor 701 and a memory 702, wherein the memory 702 stores a surface defect small sample detection program 703 based on contrastive learning.

[0111] In some embodiments, the memory 702 may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the memory 702 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory 702 may also include both an internal storage unit of the computer device and an external storage device. The memory 702 is used to store application software and various types of data installed on the computer device, such as program codes for installing the computer device. The memory 702 may also be used to temporarily store data that has been output or is to be output. In one embodiment, when the surface defect small sample detection program 703 based on contrastive learning is executed by the processor 701, the following steps are implemented:

[0112] Step S1, collecting defect image samples, annotating defect detection mask images and defect categories, and performing image preprocessing on the annotated defect image samples to obtain a defect image data set;

[0113] Step S2, establishing a surface defect detection model based on the improved image language comparison pre-training model CLIP;

[0114] Step S3, selecting defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model;

[0115] Step S4, inputting the defect image to be detected into the trained surface defect detection model, and outputting the defect detection result.

[0116] In some embodiments, the processor 701 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 702, such as executing a small sample surface defect detection program based on contrast learning.

[0117] This embodiment further provides a computer-readable storage medium, on which a surface defect small sample detection program based on contrastive learning is stored. When the surface defect small sample detection program based on contrastive learning is executed by a processor, the following steps are implemented:

[0118] Step S1, collecting defect image samples, annotating defect detection mask images and defect categories, and performing image preprocessing on the annotated defect image samples to obtain a defect image data set;

[0119] Step S2, establishing a surface defect detection model based on the improved image language comparison pre-training model CLIP;

[0120] Step S3, selecting defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model;

[0121] Step S4, inputting the defect image to be detected into the trained surface defect detection model, and outputting the defect detection result.

[0122] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A small sample detection method for surface defects based on contrastive learning, It is characterized in that include: Step S1, collecting defect image samples, annotating defect detection mask images and defect categories, and performing image preprocessing on the annotated defect image samples to obtain a defect image data set; Step S2, establishing a surface defect detection model based on the improved image language comparison pre-training model CLIP; Step S3, selecting defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model; Step S4, inputting the defect image to be detected into the trained surface defect detection model, and outputting the defect detection result.

2. The surface defect small sample detection method based on contrastive learning according to claim 1, It is characterized in that In step S1, the image preprocessing of the labeled defect image samples includes: The collected defect image samples are scaled to a fixed size through quadratic interpolation; A regularized data augmentation method is used to randomly perturb the brightness, contrast, and sharpness of the image to simulate the image style under the lighting changes and conveyor belt speed changes in the production workshop.

3. The surface defect small sample detection method based on contrastive learning according to claim 1, It is characterized in that In step S2, the surface defect detection model includes: Vision-Transformer feature extraction network, where Transformer is used as a text feature generation network, and a VV attention mechanism is embedded in a specific layer of the Vision-Transformer feature extraction network to replace the original QKV attention mechanism of the CLIP model; The text encoder uses a 12-layer Transformer block stack to accept fine-grained text generated by a preset domain expert module, combines the fine-grained text descriptions to generate tokens, and obtains the corresponding text feature embedding; where a token refers to a single discrete unit in a fine-grained text description; The post-processing module includes a coarse detection unit and a fine detection unit. The coarse detection unit is used to calculate the image-level similarity score using the overall image features to determine whether the image has defects; the fine detection unit is used to compare the positive and negative properties of each pixel using the pixel-level similarity score of the fine-grained local features, generate a defect feature map, and realize defect area positioning and segmentation.

4. The surface defect small sample detection method based on contrastive learning according to claim 3, It is characterized in that The step S3 specifically includes: Step S31, selecting defect image samples from the defect image dataset as fine-tuning input, using the multi-classification cross entropy loss function as the loss function of the rough detection process for training, and saving the model parameters; Step S32: Select the target detection network for precise detection, use the defect image after image preprocessing as the input of the target detection network, train and save the model parameters; select the dynamically scaled cross entropy loss FocalLoss and the binary classification loss BinaryDiceLoss of image segmentation as the loss function of precise detection.

5. The surface defect small sample detection method based on contrastive learning according to claim 4, It is characterized in that In step S31, the expression of the cross entropy loss function of the multi-classification is: In the formula, Loss is the loss function, y is the discriminant probability distribution of the surface defect recognition network for the input image x, and label represents the actual defect category of the input image x; In step S32, the expression of the dynamically scaled cross entropy loss FocalLoss is: FocalLoss(p t )=-α t (1-p t ) y log(p t ) In the formula, FocalLoss(p t ) represents the dynamically scaled cross entropy loss function, p t is the model’s predicted probability for the true category, α t is a balancing factor used to adjust the importance of different categories, and the superscript y is a focusing parameter that controls the degree of attention paid to difficult and easy samples; The expression of BinaryDiceLoss for image segmentation is: In the formula, X is the predicted probability value of each pixel, and Y is the true value.

6. The surface defect small sample detection method based on contrastive learning according to claim 1, It is characterized in that Before step S4, the method further includes: The defect image to be detected is collected by an industrial camera in real time, and is scaled to a fixed size through quadratic interpolation as the input of the surface defect detection model.

7. The surface defect small sample detection method based on contrastive learning according to claim 4, It is characterized in that In step S4, the defect image to be detected is input into the trained surface defect detection model, and the defect detection result is output, including: The defect image to be detected is input into the trained surface defect detection model, and the probability of the defect image belonging to each defect category is obtained through rough detection. It is judged whether the maximum value of the defect category probability is greater than the preset classification probability threshold. If not, it is judged as defect-free; if so, the defect image is input into the target detection network for fine detection; The target detection network outputs the defect category confidence of the defect image and determines whether the defect category confidence is greater than the preset confidence threshold. If not, it is judged as defect-free; if so, the defect category and defect detection box are obtained.

8. A surface defect small sample detection system based on contrastive learning, It is characterized in that include: The sample collection module is used to collect defect image samples, annotate defect detection mask images and defect categories, and perform image preprocessing on the annotated defect image samples to obtain a defect image data set; Model building module, used to build a surface defect detection model based on the improved image language comparison pre-training model CLIP; The model training module is used to select defect image samples from the defect image dataset to perform fine-tuning training on the surface defect detection model; The defect detection module is used to input the defect image to be detected into the trained surface defect detection model and output the defect detection result.

9. An electronic device, It is characterized in that comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the surface defect small sample detection method based on contrastive learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps of the surface defect small sample detection method based on contrast learning as described in any one of claims 1 to 7.

Images