Small sample defect classification method based on retrieval enhancement and multi-modal large model

Abstract

The invention discloses a small sample defect classification method based on retrieval enhancement and a multi-modal large model, the method takes similar defect features in different products into consideration, a breakthrough retrieval mechanism is introduced, the mechanism accesses a cross-domain database containing various products and materials, and the retrieval efficiency is improved. In this way, the model learning range is effectively expanded, and the limitation of a single data set or field is exceeded; through the retrieval mechanism, effective training of the classification model can be realized by using small samples; besides, a BNADaster adapter module is designed in the classification model, effective transfer learning is realized through fine tuning of an additional feature adapter, the method is simple and efficient, the calculation cost is saved, and the classification accuracy can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial defect detection, and specifically is a small sample defect classification method based on retrieval enhancement and multimodal large model. Background Art

[0002] In the field of advanced manufacturing and quality control, accurate identification and classification of material and product defects are crucial to ensure high quality standards and reliability of industrial products. Defects are usually caused by uncontrollable factors in the manufacturing and transportation processes, resulting in the emergence of various defect forms. However, the difficulty in collecting enough defect image samples limits the application of deep learning in the field of industrial defect identification. Especially in small sample scenarios, such as limited production lines or early stages of product development, the challenge of defect classification is particularly prominent. Traditional defect classification methods rely on a large amount of labeled data to train deep learning models, but often do not work well under small sample conditions, seriously affecting the classification performance and generalization ability of the model. Therefore, innovative methods are urgently needed to effectively solve the challenges of small sample defect classification and ensure the robustness of the model when data is scarce.

[0003] In recent years, many methods have been proposed to solve the small sample problem, which can be roughly divided into two categories: data-based methods and model-based methods. Data-based methods increase the number of samples through various data augmentation techniques to enrich the dataset used to train the model. These methods usually involve geometric transformation, color space adjustment, or generate data through generative adversarial networks (GANs) to create more diverse and robust datasets, helping the model to improve generalization ability from limited original samples. On the other hand, model-based methods focus on optimizing learning algorithms to reduce the model's dependence on samples. Transfer learning is an effective method that pre-trains the model on a large dataset and then fine-tunes it using a small dataset for a specific task. In this method, small sample learning is particularly important, and the design of the model often requires the use of prior knowledge or structural information in the data to effectively learn under limited samples.

[0004] Although existing methods have made significant progress in solving the small sample problem, most studies focus on a single domain. In industrial defect classification, limited and homogeneous data in a single domain restricts the performance of the model. Summary of the invention

[0005] In view of the deficiencies of the prior art, the technical problem that the present invention intends to solve is to provide a small sample defect classification method based on retrieval enhancement and multimodal large models. The method takes into account the existence of similar defect features in different products (such as wood, strip steel, metal aluminum profiles and fabrics), such as scratches, oil spots and inclusions, and proposes a multimodal small sample retrieval enhanced defect classification method, which is achieved by systematically utilizing the retrieved external knowledge. The method of the present invention introduces a breakthrough retrieval mechanism that accesses a cross-domain database containing various products and materials. In this way, the scope of model learning is effectively expanded, surpassing the limitations of a single data set or field, so that defects can be identified and classified with higher accuracy and generalization performance.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows: a small sample defect classification method based on retrieval enhancement and multimodal large model, characterized in that the method comprises the following steps:

[0007] Step 1: Build a retrieval image library

[0008] Various types of defect data of different materials with backgrounds or illumination are obtained from open source defect data on the Internet, and the defect data form a retrieval image library, in which one data sample in the retrieval image library is one image;

[0009] Step 2: Build an image retrieval system

[0010] The image retrieval system includes a feature extractor, a feature memory and a retrieval module, wherein the feature extractor is a first CLIP model, and the feature memory is a FAISS vector database; the first CLIP model includes a visual encoder and a text encoder, both of which are open source pre-trained network models for extracting image and text features;

[0011] For the retrieval image library in step 1, a feature extractor is used to extract the pixel feature vector of each image in the retrieval image library. To ensure the consistency of the similarity measure, the extracted pixel feature vector is then normalized to unit length:

[0012]

[0013] Among them, ||f input || is the Euclidean norm of the eigenvector;

[0014] The normalized result of the extracted pixel feature vector is saved as a pixel feature embedding matrix, and is input into the pickle module to obtain a cache file, in which one of the data is the path of an image and the corresponding pixel feature embedding matrix; the cache file is input into the FAISS vector database, and index features are created through the FAISS vector database;

[0015] The image retrieval system supports two types of input: image and text. For the input image, the retrieval module inputs it into the visual encoder of the first CLIP model, extracts the pixel feature vector of the image, and normalizes the result to unit length to obtain the pixel query feature vector of the image; for the input text, the retrieval module inputs it into the text encoder of the first CLIP model, extracts the feature vector of the text, and normalizes the result to unit length to obtain the text query feature vector;

[0016] For a pixel query feature vector or a text query feature vector, the retrieval module calculates the similarity between the query feature vector and the index feature stored in the FAISS vector database. The similarity between the query feature vector and the index feature is calculated by cosine similarity, which is defined as:

[0017]

[0018] Where q is the query feature vector, f i is the index feature corresponding to the i-th image in the retrieval image library; since the feature values ​​have been normalized to unit length, the cosine similarity is simplified to the inner product;

[0019] According to the cosine similarity score sorting, the retrieval module selects the top n most relevant images retrieved from the retrieval image library whose similarity exceeds the set threshold as the output of the image retrieval system;

[0020] Step 3: Build a language-guided small sample defect classification model

[0021] The small sample defect classification model is a CLIP-BNAdapter model, which includes a second CLIP model and a BNAdapter adapter module; the second CLIP model is an open source pre-trained network model, and the input image and the corresponding defect description are processed by a visual encoder and a text encoder respectively, and the corresponding pixel feature vector f is output. vision (x) and the text feature vector t text (y);

[0022] The BNAdapter adapter module includes a first L-BN layer, a first ReLU activation function layer, a second L-BN layer, a second ReLU function layer, and a third L-BN layer; the first L-BN layer, the second L-BN layer, and the third L-BN layer each include a linear layer and a batch normalization layer, and the basic structure of the three modules is the same, and the parameters are not shared;

[0023] The pixel feature vector output by the second CLIP model is first processed by the first L-BN layer, the first ReLU activation function layer, the second L-BN layer, and the second ReLU function layer of the BNAdapter adapter module to obtain the first extracted feature; at the same time, the pixel feature vector input to the BNAdapter adapter module is processed by the third L-BN layer to obtain the second extracted feature; the first extracted feature and the second extracted feature are added to obtain the third extracted feature, which is expressed by the formula:

[0024] Ad(f)=ReLU(BN(W2·ReLU(BN(W1·f)))+BN(W r f)

[0025] Where f represents the pixel feature vector f vision (x), Ad(f) represents the third extracted features after processing by the BNAdapter adapter, W1, W2 and W r is the weight matrix, BN represents batch normalization operation, and ReLU represents ReLU activation function operation;

[0026] The third extracted feature Ad(f) is combined with the pixel feature vector f extracted by the feature extractor vision (x) is weighted and summed, λ is set to 0.2, and the fourth extracted feature f is obtained final ;

[0027] f final =λ·Ad(f)+f vision

[0028] Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix:

[0029] logits=logits_scale·Sim(f final ,t text (y))

[0030] Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores;

[0031] Step 4: Collect defect image datasets on the surface of the target object to be inspected

[0032] Obtain a physical image of each defect type on the surface of a target object to be inspected, and construct a small sample classification data set D1; the target object to be inspected and the corresponding defect type are included in the retrieval image library in the first step; a sample in the data set D1 includes a defect image and a corresponding defect category label; divide the data set D1 into a training set and a validation set in a ratio of 6:4 according to the number;

[0033] Step 5: Train the small sample defect classification model in step 3

[0034] Step 5.1: The defect images and defect category labels in the dataset D1 in the fourth step are converted into text descriptions, which are input into the image retrieval system in the second step respectively, and multiple images whose cosine similarity with each defect image and each defect category text description in the retrieval image library exceeds the threshold of 0.7 are screened out, and all the screened images are preprocessed to obtain the image set S1; the defect category text descriptions input into the image retrieval system are used as pseudo labels for the corresponding images in the image set S1, and the image set S1 is divided into a training set and a validation set in a ratio of 7:3; the dataset D1 is also preprocessed in the same way to obtain the dataset D2;

[0035] Step 5.2: Model parameter initialization

[0036] The learnable parameters of the BNAdapter adapter module are initialized by random assignment, and the second CLIP model uses open source pre-trained parameters trained on the WIT dataset; during the training process, SGD is used as the optimization algorithm, the initial learning rate is set to 0.002, the batch parameter is 32, and the maximum iteration round is set to 200; the second CLIP model uses ResNet-50 as the backbone network of the visual encoder, and uses BERT as the backbone network of the text encoder, and uses "a photo of [CLASS]" as the hard prompt template of the text encoder;

[0037] Step 5.3: Preliminary training of model parameters

[0038] The training set and validation set of the image set S1 in step 5.1 are input into the small sample defect classification model constructed in the third step, and set to fully supervised training, that is, all data in the training set participate in the training, and the validation set is used for verification in each round; the defect category corresponding to each image is converted into the encoding style of "a photo of [CLASS]" through the hard prompt template, and is sent to the text encoder and visual encoder of the second CLIP model with frozen parameters to extract features, and the pixel feature vectors f are obtained respectively. vision (x) and the text feature vector t text (y), where f vision(x) is then processed by the BNAdapter module to generate the third extracted feature vector Ad(f), which is compared with the pixel feature vector f extracted by the feature extractor vision (x) is weighted summed, λ is set to 0.2, and the fourth extracted feature f is obtained final ;

[0039] f final =λ·Ad(f)+f vision

[0040] Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix:

[0041] logits=logits_scale·Sim(f final ,t text (y))

[0042] Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores;

[0043] Then the cross entropy loss of logits and labels is calculated as the training loss, and the parameters of the BNAdapter adapter module are optimized by back propagation using a small batch stochastic gradient descent method based on the training loss; the cross entropy loss is defined as:

[0044]

[0045] In the above formula, N represents the total number of samples, L represents the average loss of the whole, and L i represents the cross entropy loss of the i-th sample, M represents the number of categories, and y ic is a sign function, taking values ​​of 0 or 1; if the true category of sample i is equal to c, then y ic =1, otherwise y ic =0; p ic is the predicted probability that the i-th sample belongs to defect category c, which is the result of converting the values ​​in the logits matrix into a probability distribution through softmax;

[0046] During the training process, the learning rate η is dynamically adjusted to ensure rapid convergence of the model. During the training process, a validation set evaluation is performed after each round. If the maximum round is reached or the classification accuracy of the validation set does not increase after 10 consecutive rounds, the training is stopped. The model parameters obtained after training are saved to obtain a model that has completed preliminary training.

[0047] Step 5.4: Deep training of model parameters

[0048] Input the training set in the data set D2 into the small sample defect classification model that has completed preliminary training. Different from the fully supervised training setting in step 5.3, the parameters random seed seed and num_shots are added at this time. Three trainings are performed under each num_shots setting. Seed is set to 1, 2, and 3 in turn as the mark of the three training tasks under each num_shots setting. Finally, the average of the three training results is taken as the final accuracy. Num_shots is set to 1, 2, 4, 8, 16, and 25 in turn. If it is set to 1, the model will extract 1 sample from each category in the training set for training before the training starts. There are 9 training samples for 9 categories, and so on. Input the extracted training samples into the model that has completed preliminary training in step 5.3 for training. The training process is the same as that in step 5.3. During the training process, the verification data for each round is the entire verification set, which will not change with the setting of the num_shots parameter. After 200 rounds of training or 10 consecutive rounds of verification set classification accuracy does not increase, save the model parameters of the round with the highest classification accuracy to obtain a deep training model.

[0049] Step 6: Classification of defect images on the surface of the target object to be inspected

[0050] Obtain a defect image of the unknown defect category of the surface of the target object to be inspected as in the fourth step, and a text description of all defect categories on the surface of the target object to be inspected, then adjust the size of the defect image according to the requirements of the small sample defect classification model, and convert all defect category text descriptions into the encoding style of "a photo of [CLASS]" through a hard prompt template; then input the defect image and the encoding style of each defect category text description into the deep training model in the fifth step respectively, and obtain the logits similarity matrix of the defect image; in the logits similarity matrix, each element value represents the model's unnormalized confidence that the input image belongs to a certain category, and from all logits similarity matrices, select the element with the largest value, and the defect category corresponding to the element with the largest value is the predicted category of the input image.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The multimodal small sample retrieval enhanced classification model of the present invention adopts the retrieval enhancement framework for the first time to obtain defect knowledge in other related fields. Different from the traditional data enhancement methods of geometry and color transformation, the retrieval enhancement method can obtain rich data such as a variety of materials, backgrounds, and different lighting, thereby improving the model's defect classification accuracy and generalization ability, and can achieve accurate classification of defects through a very small number of training samples, reducing the cost of manual labeling.

[0053] 2. In order to effectively construct external knowledge, a large-scale multimodal image retrieval system based on the multimodal contrastive learning model CLIP and the vector database FAISS was designed. The system supports bimodal input of images and language, supports fast and large-scale retrieval tasks, and does not require additional manual annotation.

[0054] 3. In order to quickly and effectively integrate external knowledge into the model, the BNAdapter adapter module is designed. By fine-tuning additional feature adapters, effective transfer learning is achieved. It is simple and efficient, saving computational costs and ensuring classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The figure is a schematic diagram of the working principle of an image retrieval system of an embodiment of a small sample defect classification method based on retrieval enhancement and multimodal large model of the present invention.

[0056] Figure 2 The diagram is a schematic diagram of the structure and principle of a small sample defect classification model of an embodiment of a small sample defect classification method based on retrieval enhancement and multimodal large model of the present invention. DETAILED DESCRIPTION

[0057] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods, but the protection scope of the present application is not limited thereto.

[0058] The present invention provides a small sample defect classification method based on retrieval enhancement and multimodal large model (hereinafter referred to as method), which comprises the following steps:

[0059] Step 1: Build a retrieval image library

[0060] Various types of defect data of different materials with backgrounds or illumination are obtained from open source defect data on the Internet. The materials include magnetic tiles, strip steel, metal aluminum profiles, fabrics, etc. The defects include scratches, stains, impurities, welds, etc. The defect data forms a retrieval image library, and a data sample in the retrieval image library is an image. The defect data used in this embodiment specifically includes 392 magnetic tiles, 4576 fabrics, 8230 strip steels, 372 metal aluminum profiles, and an open source DAGM data set, including 2200 artificially generated defect images, to form a retrieval image library.

[0061] Step 2: Build an image retrieval system

[0062] The image retrieval system includes a feature extractor, a feature memory and a retrieval module, wherein the feature extractor is a first CLIP model (Contrastive Language-Image Pre-training, i.e., a pre-trained visual language contrast learning model), and the feature memory is a FAISS vector database. The first CLIP model includes a visual encoder and a text encoder, wherein the backbone network of the visual encoder is ResNet50, and the backbone network of the text encoder is BERT, both of which are open source pre-trained network models for extracting image and text features.

[0063] For the retrieval image library in step 1, a feature extractor is used to extract the pixel feature vector of each image in the retrieval image library. To ensure the consistency of the similarity measure, the extracted pixel feature vector is then normalized to unit length:

[0064]

[0065] Among them, ||f input || is the Euclidean norm of the eigenvector.

[0066] The normalized result of the extracted pixel feature vector is saved as a pixel feature embedding matrix, and is input into the pickle module (a module in Python used to serialize and deserialize objects, here used to cache features) to obtain a cache file (extract_embeddings), in which one of the data is the path of an image and the corresponding pixel feature embedding matrix. The above cache file is input into the FAISS vector database, and index features (i.e., pixel features) are created through the FAISS vector database; so that images can be efficiently matched during retrieval. FAISS supports efficient large-scale similarity search, which is crucial for fast retrieval operations.

[0067] The image retrieval system supports two types of input: image and text. For the input image, the retrieval module inputs it into the visual encoder of the first CLIP model, extracts the pixel feature vector of the image, and normalizes the result to unit length to obtain the pixel query feature vector of the image; for the input text, the retrieval module inputs it into the text encoder of the first CLIP model, extracts the feature vector of the text, and normalizes the result to unit length to obtain the text query feature vector.

[0068] For a pixel query feature vector or a text query feature vector, the retrieval module calculates the similarity between the query feature vector and the index feature (i.e., pixel feature) stored in the FAISS vector database. The similarity between the query feature vector and the index feature is calculated using cosine similarity, which is defined as:

[0069]

[0070] Where q is the query feature vector, f i is the index feature corresponding to the i-th image in the retrieval image library. Since the feature values ​​have been normalized to unit length, the cosine similarity is simplified to the inner product.

[0071] According to the cosine similarity score sorting, the retrieval module selects the top n most relevant images retrieved from the retrieval image library whose similarity exceeds the set threshold as the output of the image retrieval system. The selected images serve as external knowledge of the classification model, effectively inputting industrial defect features into the model, thereby supporting small sample defect classification.

[0072] Step 3: Build a language-guided small sample defect classification model

[0073] The small sample defect classification model is the CLIP-BNAdapter model, and its structure is as follows: Figure 2 As shown in FIG. 1 , it includes the second CLIP model and the BNAdapter adapter module. The second CLIP model is an open source pre-trained network model. The input image and the corresponding defect description are processed by the visual encoder and the text encoder respectively, and the corresponding pixel feature vector f is output. vision (x) and the text feature vector t text (y).

[0074] The text encoder in the second CLIP model represents the semantic space of the task. In order to maintain the invariance of the semantic space of the large model, only the parameters of the feature adapter of the image branch are updated, and the language branch is frozen to lock the general task encoding knowledge.

[0075] The BNAdapter module includes a first L-BN layer, a first ReLU activation function layer, a second L-BN layer, a second ReLU function layer, and a third L-BN layer. The first L-BN layer, the second L-BN layer, and the third L-BN layer each include a linear layer and a batch normalization layer. The basic structure of the three modules is the same, and the parameters are not shared.

[0076] The first linear layer reduces the dimension of the input features, while the second linear layer maps these features back to the original dimension. This "bottleneck" structure helps compress and expand the features to capture deeper feature dependencies. The residual connection enables the input to bypass the backbone network and directly add the processed output of BNAdapter. This design helps alleviate the gradient vanishing or gradient exploding problems that occur in deeper networks while retaining the representational power of the network.

[0077] The pixel feature vector output by the second CLIP model is first processed by the first L-BN layer, the first ReLU activation function layer, the second L-BN layer, and the second ReLU function layer of the BNAdapter adapter module to obtain the first extracted feature; at the same time, the pixel feature vector input to the BNAdapter adapter module is processed by the third L-BN layer to obtain the second extracted feature. The first extracted feature and the second extracted feature are added to obtain the third extracted feature, which is expressed by the formula:

[0078] Ad(f)=ReLU(BN(W2·ReLU(BN(W1·f)))+BN(W r f)

[0079] Where f represents the pixel feature vector f vision (x), Ad(f) represents the third extracted features after processing by the BNAdapter adapter, W1, W2 and W r is the weight matrix of the linear layer in the trunk and residual branch, where W1 and W2 are responsible for transforming the input features in the main branch, and W r The dimensions in the residual branch are aligned so that they can be added correctly. BN represents the batch normalization operation and ReLU represents the ReLU activation function operation.

[0080] A batch normalization layer is added after each linear layer to normalize the output, which helps control internal covariate shift during training and enhances the generalization ability of the model. Batch normalization can be expressed as:

[0081]

[0082] Among them, μ and σ 2 are the mean and variance of the batch, ε is a very small constant to avoid division by zero, and γ and β are learnable parameters.

[0083] The third extracted feature Ad(f) is combined with the pixel feature vector f extracted by the feature extractor vision (x) is weighted summed, λ is set to 0.2, and the fourth extracted feature f is obtained final .

[0084] f final =λ·Ad(f)+f vision

[0085] Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix:

[0086] logits=logits_scale·Sim(ffinal ,t text (y))

[0087] Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores. The logits similarity matrix represents the unnormalized confidence scores of the model that the input image belongs to each category. After softmax transformation, the model's predicted probability of the input image belonging to each category can be obtained.

[0088] Step 4: Collect defect image datasets on the surface of the target object to be inspected

[0089] Obtain a physical image of each defect type on the surface of a target object to be inspected, preprocess it, and construct a small sample classification data set D1; the target object to be inspected and the corresponding defect type are covered in the retrieval image library in the first step; a sample in the data set D1 includes a defect image and the corresponding defect category label; divide the data set D1 into a training set and a validation set in a ratio of 6:4 according to the quantity.

[0090] The target object to be detected in this embodiment is strip steel, and physical images of surface defects of the strip steel are collected. When collecting physical images of surface defects of the strip steel, fixed camera position is used for shooting. The data set D1 includes defect images and defect category labels, including welding line, tar, water stains, shrinkage holes, scratches, pitting, oil spots, inclusions and gradients, a total of 9 types of defect images, 50 images of each type, a total of 450 images, and this data set is recorded as D1.

[0091] Step 5: Train the small sample defect classification model in step 3

[0092] Step 5.1: The defect images and defect category labels in the dataset D1 in the fourth step are converted into text descriptions (for example, an image with long and thin scratches), which are respectively input into the image retrieval system in the second step. The feature vectors of the input images or texts are extracted through the pre-trained first CLIP model and mapped to the same dimension to calculate the similarity. Finally, the cosine similarity is calculated with the corresponding index features in the FAISS vector database. Multiple images whose cosine similarity with each defect image and each defect category text description exceeds the threshold of 0.7 (similar visual representation) are selected from the retrieval image library. All the selected images are preprocessed to obtain image set S1. The defect category text descriptions input into the image retrieval system are used as pseudo labels for the corresponding images in image set S1, and image set S1 is divided into a training set and a validation set in a ratio of 7:3. The same preprocessing is performed on dataset D1 to obtain dataset D2.

[0093] The preprocessing first resizes the image according to the requirements of the small sample defect classification model, and scales the pixel values ​​of each image using the bicubic interpolation method. As an embodiment, the image size is adjusted to 224×224, the image pixel mean is scaled to [0.481, 0.458, 0.408], and the standard deviation is scaled to [0.269, 0.261, 0.276]. Then, three data augmentation transformation operations are performed on each image to expand the number of images; the three data augmentation transformation operations are random scaling and cropping, random flipping, and normalization.

[0094] Step 5.2: Model parameter initialization

[0095] The learnable parameters of the BNAdapter adapter module are initialized by random assignment, and the second CLIP model uses open source pre-trained parameters trained on the WIT (WebImageText) dataset. During the training process, SGD is used as the optimization algorithm, the initial learning rate is set to 0.002, the batch parameter (batch_size) is 32, and the maximum iteration round (max_epoch) is set to 200. The second CLIP model uses ResNet-50 as the backbone network of the CLIP visual branch (visual encoder), and uses BERT as the backbone network of the language branch (text encoder). "aphoto of [CLASS]" is used as the hard prompt template of the language branch.

[0096] Step 5.3: Preliminary training of model parameters

[0097] The training set and validation set of the image set S1 in step 5.1 are input into the small sample defect classification model constructed in the third step, and are set to fully supervised training, that is, all data in the training set participate in the training, and the validation set is used for each round of validation. The text description corresponding to each image is converted into the encoding style of "a photo of [CLASS]" through the hard prompt template, and is sent to the language encoder and visual encoder of the second CLIP model with frozen parameters to extract features, respectively, to obtain the pixel feature vectors f vision (x) and the text feature vector t text (y), where f vision (x) is then processed by the BNAdapter module to generate the third extracted feature vector Ad(f), which is compared with the pixel feature vector f extracted by the feature extractor vision (x) is weighted summed, λ is set to 0.2, and the fourth extracted feature f is obtained final .

[0098] f final =λ·Ad(f)+f vision

[0099] Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix:

[0100] logits=logits_scale·Sim(f final ,t text (y))

[0101] Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores.

[0102] Then the cross entropy loss of logits and labels is calculated as the training loss, and the parameters of the BNAdapter adapter module are optimized using the mini-batch stochastic gradient descent (SGD) back propagation method based on the training loss. The cross entropy loss is defined as:

[0103]

[0104] In the above formula, N represents the total number of samples, L represents the average loss of the whole, and L i represents the cross entropy loss of the i-th sample, M represents the number of categories, and y ic is a sign function, taking values ​​of 0 or 1. If the true category of sample i is equal to c, then y ic =1, otherwise y ic =0; p ic is the predicted probability that the ith sample belongs to defect category c, which is the result of converting the values ​​in the logits matrix into a probability distribution through softmax.

[0105] During the training process, the learning rate η is dynamically adjusted to ensure rapid convergence of the model. The termination condition of the training mainly depends on the maximum number of training rounds (max_epoch), which is 200. The training process will perform a validation set evaluation after each epoch. If the maximum number of rounds is reached or the classification accuracy of the validation set does not increase for 10 consecutive epochs, the training will be stopped. The model parameters obtained after training are saved to obtain a model that has completed preliminary training.

[0106] Step 5.4: Deep training of model parameters

[0107] Input the training set in the dataset D1 into the small sample defect classification model that has completed preliminary training. Different from the fully supervised training setting in step 5.3, the parameters random seed seed and num_shots are added at this time. Three trainings are performed under each num_shots setting. Seed is set to 1, 2, and 3 in turn as the mark of the three training tasks under each num_shots setting. Finally, the average of the three training results is taken as the final accuracy. Num_shots is set to 1, 2, 4, 8, 16, and 25 in turn. If it is set to 1, the model will extract 1 sample from each category in the training set for training before the training starts. There are 9 training samples for 9 categories, and so on. Input the extracted training samples into the model that has completed preliminary training in step 5.3 for training. The training process is the same as that in step 5.3. The verification data for each round of training is the entire verification set, which will not change with the setting of the num_shots parameter. After 200 epochs of training or 10 consecutive epochs, if the classification accuracy of the verification set does not increase, save the model parameters of the epoch with the highest classification accuracy to obtain a deep training model.

[0108] Step 6: Classification of defect images on the surface of the target object to be inspected

[0109] Obtain the same defect image of the unknown defect category on the surface of the target object to be inspected as in the fourth step, as well as all defect category text descriptions on the surface of the target object to be inspected, then adjust the size of the defect image according to the requirements of the small sample defect classification model, and convert all defect category text descriptions into the encoding style of "a photo of [CLASS]" through the hard prompt template; then input the defect image and the encoding style of each defect category text description into the deep training model in the fifth step respectively, and obtain the logits similarity matrix of the defect image. In the logits similarity matrix, each element value represents the model's unnormalized confidence that the input image belongs to a certain category. From all logits similarity matrices, select the element with the largest value, and the defect category corresponding to the element with the largest value is the predicted category of the input image.

[0110] Table 1 compares the classification results of the method of the present invention and the common small sample classification model based on the basic CLIP model on the data set D2.

[0111] Table 1 Comparison of detection results of different models on D2 dataset (accuracy / %)

[0112]

[0113] The results show that when the number of training samples per class is 2, the classification accuracy of the method of the present invention can reach about 80%. As the number of training samples increases, when the number of training samples per class increases to 8, the defect classification accuracy is increased to more than 90%. And under each small sample setting, the defect classification accuracy of the existing method is exceeded, which also verifies the effectiveness of the method of the present invention.

[0114] Any matters not described in the present invention are applicable to the prior art.

Claims

1. A small sample defect classification method based on retrieval enhancement and multimodal large model, characterized in that: The method comprises the following steps: Step 1: Build a retrieval image library Various types of defect data of different materials with backgrounds or illumination are obtained from open source defect data on the Internet, and the defect data form a retrieval image library, in which one data sample in the retrieval image library is one image; Step 2: Build an image retrieval system The image retrieval system includes a feature extractor, a feature memory and a retrieval module, wherein the feature extractor is a first CLIP model, and the feature memory is a FAISS vector database; the first CLIP model includes a visual encoder and a text encoder, both of which are open source pre-trained network models for extracting image and text features; For the retrieval image library in step 1, a feature extractor is used to extract the pixel feature vector of each image in the retrieval image library. To ensure the consistency of the similarity measure, the extracted pixel feature vector is then normalized to unit length: Among them, ||f input || is the Euclidean norm of the eigenvector; The normalized result of the extracted pixel feature vector is saved as a pixel feature embedding matrix, and is input into the pickle module to obtain a cache file, in which one of the data is the path of an image and the corresponding pixel feature embedding matrix; the cache file is input into the FAISS vector database, and index features are created through the FAISS vector database; The image retrieval system supports two types of input: image and text. For the input image, the retrieval module inputs it into the visual encoder of the first CLIP model, extracts the pixel feature vector of the image, and normalizes the result to unit length to obtain the pixel query feature vector of the image; for the input text, the retrieval module inputs it into the text encoder of the first CLIP model, extracts the feature vector of the text, and normalizes the result to unit length to obtain the text query feature vector; For a pixel query feature vector or a text query feature vector, the retrieval module calculates the similarity between the query feature vector and the index feature stored in the FAISS vector database. The similarity between the query feature vector and the index feature is calculated by cosine similarity, which is defined as: Where q is the query feature vector, f i is the index feature corresponding to the i-th image in the retrieval image library; since the feature values ​​have been normalized to unit length, the cosine similarity is simplified to the inner product; According to the cosine similarity score sorting, the retrieval module selects the top n most relevant images retrieved from the retrieval image library whose similarity exceeds the set threshold as the output of the image retrieval system; Step 3: Build a language-guided small sample defect classification model The small sample defect classification model is a CLIP-BNAdapter model, which includes a second CLIP model and a BNAdapter adapter module; the second CLIP model is an open source pre-trained network model, and the input image and the corresponding defect description are processed by a visual encoder and a text encoder respectively, and the corresponding pixel feature vector f is output. vision (x) and text feature vector t text (y); The BNAdapter adapter module includes a first L-BN layer, a first ReLU activation function layer, a second L-BN layer, a second ReLU function layer, and a third L-BN layer; the first L-BN layer, the second L-BN layer, and the third L-BN layer each include a linear layer and a batch normalization layer, and the basic structure of the three modules is the same, and the parameters are not shared; The pixel feature vector output by the second CLIP model is first processed by the first L-BN layer, the first ReLU activation function layer, the second L-BN layer, and the second ReLU function layer of the BNAdapter adapter module to obtain the first extracted feature; at the same time, the pixel feature vector input to the BNAdapter adapter module is processed by the third L-BN layer to obtain the second extracted feature; the first extracted feature and the second extracted feature are added to obtain the third extracted feature, which is expressed by the formula: Ad(f)=ReLU(BN(W2·ReLU(BN(W1·f)))+BN(W r f) Where f represents the pixel feature vector f vision (x), Ad(f) represents the third extracted features after processing by the BNAdapter adapter, W1, W2 and W r is the weight matrix, BN represents batch normalization operation, and ReLU represents ReLU activation function operation; The third extracted feature Ad(f) is combined with the pixel feature vector f extracted by the feature extractor vision (x) is weighted and summed, λ is set to 0.2, and the fourth extracted feature f is obtained final ; f final =λ·Ad(f)+f vision Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix: logits=logits_scale·Sim(f final ,t ext (y)) Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores; Step 4: Collect defect image datasets on the surface of the target object to be inspected Obtain a physical image of each defect type on the surface of a target object to be inspected, and construct a small sample classification data set D1; the target object to be inspected and the corresponding defect type are included in the retrieval image library in the first step; a sample in the data set D1 includes a defect image and a corresponding defect category label; divide the data set D1 into a training set and a validation set in a ratio of 6:4 according to the number; Step 5: Train the small sample defect classification model in step 3 Step 5.1: The defect images and defect category labels in the dataset D1 in the fourth step are converted into text descriptions, which are input into the image retrieval system in the second step respectively, and multiple images whose cosine similarity with each defect image and each defect category text description in the retrieval image library exceeds the threshold of 0.7 are screened out, and all the screened images are preprocessed to obtain the image set S1; the defect category text descriptions input into the image retrieval system are used as pseudo labels for the corresponding images in the image set S1, and the image set S1 is divided into a training set and a validation set in a ratio of 7:3; the dataset D1 is also preprocessed in the same way to obtain the dataset D2; Step 5.2: Model parameter initialization The learnable parameters of the BNAdapter adapter module are initialized by random assignment, and the second CLIP model uses open source pre-trained parameters trained on the WIT dataset; during the training process, SGD is used as the optimization algorithm, the initial learning rate is set to 0.002, the batch parameter is 32, and the maximum iteration round is set to 200; the second CLIP model uses ResNet-50 as the backbone network of the visual encoder, and uses BERT as the backbone network of the text encoder, and uses "a photo of [CLASS]" as the hard prompt template of the text encoder; Step 5.3: Preliminary training of model parameters The training set and validation set of the image set S1 in step 5.1 are input into the small sample defect classification model constructed in the third step, and set to fully supervised training, that is, all data in the training set participate in the training, and the validation set is used for each round of verification; the defect category corresponding to each image is converted into the encoding style of "a photo of [CLASS]" through the hard prompt template, and is sent to the text encoder and visual encoder of the second CLIP model with frozen parameters to extract features, and the pixel feature vectors f are obtained respectively. vision (x) and text feature vector t text (y), where f vision (x) is then processed by the BNAdapter module to generate the third extracted feature vector Ad(f), which is compared with the pixel feature vector f extracted by the feature extractor. vistion (x) is weighted summed, λ is set to 0.2, and the fourth extracted feature f is obtained final ; f final =λ·Ad(f)+f vision Calculate the fourth extracted feature f final With the text feature vector t text (y), and get the logits similarity matrix: logits=logits_scale·Sim(f final ,t text (y)) Where logits_scale is a trainable scaling factor used to adjust the range of similarity scores; Then the cross entropy loss of logits and labels is calculated as the training loss, and the parameters of the BNAdapter adapter module are optimized by back propagation using a small batch stochastic gradient descent method based on the training loss; the cross entropy loss is defined as: In the above formula, N represents the total number of samples, L represents the average loss of the whole, and L i represents the cross entropy loss of the i-th sample, M represents the number of categories, and y ic is a sign function, taking values ​​of 0 or 1; if the true category of sample i is equal to c, then y ic =1, otherwise y ic =0; p ic is the predicted probability that the i-th sample belongs to defect category c, which is the result of converting the values ​​in the logits matrix into a probability distribution through softmax; During the training process, the learning rate η is dynamically adjusted to ensure rapid convergence of the model. During the training process, a validation set evaluation is performed after each round. If the maximum round is reached or the classification accuracy of the validation set does not increase after 10 consecutive rounds, the training is stopped. The model parameters obtained after training are saved to obtain a model that has completed preliminary training. Step 5.4: Deep training of model parameters Input the training set in the data set D2 into the small sample defect classification model that has completed preliminary training. Different from the fully supervised training setting in step 5.3, the parameters random seed seed and num_shots are added at this time. Three trainings are performed under each num_shots setting. Seed is set to 1, 2, and 3 in turn as the mark of the three training tasks under each num_shots setting. Finally, the average of the three training results is taken as the final accuracy. Num_shots is set to 1, 2, 4, 8, 16, and 25 in turn. If it is set to 1, the model will extract 1 sample from each category in the training set for training before the training starts. There are 9 training samples for 9 categories, and so on. Input the extracted training samples into the model that has completed preliminary training in step 5.3 for training. The training process is the same as that in step 5.

3. During the training process, the verification data for each round is the entire verification set, which will not change with the setting of the num_shots parameter. After 200 rounds of training or 10 consecutive rounds of verification set classification accuracy does not increase, save the model parameters of the round with the highest classification accuracy to obtain a deep training model. Step 6: Classification of defect images on the surface of the target object to be inspected Obtain a defect image of the unknown defect category of the surface of the target object to be inspected as in the fourth step, and a text description of all defect categories on the surface of the target object to be inspected. Then adjust the size of the defect image according to the requirements of the small sample defect classification model, and convert all defect category text descriptions into the "a photo of [CLASS]" encoding style through a hard prompt template. Then, input the defect image and the encoding style of each defect category text description into the deep training model in the fifth step respectively to obtain the logits similarity matrix of the defect image. In the logits similarity matrix, each element value represents the model's unnormalized confidence that the input image belongs to a certain category. From all logits similarity matrices, select the element with the largest value, and the defect category corresponding to the element with the largest value is the predicted category of the input image.

2. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 1 is characterized in that: In the first step, the different materials specifically include at least two of magnetic tiles, steel strips, metal aluminum profiles, and fabrics.

3. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 1 is characterized in that: In the first step, the defect data used specifically include 392 physical defect images of magnetic tiles, 4576 fabrics, 8230 strip steels, 372 metal aluminum profiles, and the open source DAGM dataset, which contains 2200 artificially generated defect images, to form a retrieval image library.

4. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 1 is characterized in that: In the second step, the backbone network of the visual encoder of the first CLIP model is ResNet50, and the backbone network of the text encoder is BERT.

5. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 1 is characterized in that: In the fourth step, the target object to be detected is a steel strip.

6. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 5 is characterized in that: Defect category labels for strip steel include welds, tar, water stains, shrinkage cavities, scratches, pits, oil spots, inclusions, and gradients.

7. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 6 is characterized in that: 50 physical images are collected for each defect category of the strip.

8. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 1 is characterized in that: In step 5.1 of the fifth step, the preprocessing is first to resize according to the requirements of the small sample defect classification model, and scale the pixel values ​​of each image using the bicubic interpolation method, and then perform three data augmentation transformation operations on each image to expand the number of images.

9. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 8 is characterized in that: The image size is resized to 224×224, the image pixel mean is scaled to [0.481, 0.458, 0.408], and the standard deviation is scaled to [0.269, 0.261, 0.276].

10. The small sample defect classification method based on retrieval enhancement and multimodal large model according to claim 8, characterized in that: The three data enhancement transformation operations are random scaling and cropping, random flipping and normalization.

Images