Zero sample defect detection method based on large language model and self-scoring mechanism

Abstract

The invention belongs to the technical field of image recognition, and discloses a zero sample defect detection method based on a large language model and a self-scoring mechanism. The method is characterized by comprising the following steps: mapping a query image to a feature space through a feature extractor to obtain feature representation of the image; then, designing a dialogue template with the large language model, inputting a to-be-segmented image into the large language model, and inquiring corresponding attributes of the image to obtain a description text of the image; and then the feature map is expanded, and kmeans clustering is carried out on the expanded features. And respectively performing similarity calculation on the foreground and background features obtained on the text branches and the image branches and the image features to obtain a score graph based on text guidance and a score graph based on image guidance. And finally, obtaining a final score graph by weighting and combining the score graphs of the text guidance and the image guidance. And generating a prediction mask by comparing the numerical values of the fractional graph in the two dimensions so as to identify and segment a defect area. According to the method, the accuracy of defect detection is improved, efficient zero sample detection can be realized under the condition that defect samples are scarce, data requirements are effectively reduced, and the applicability and robustness of a detection model are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and relates to a zero-sample defect detection method based on a large language model and a self-scoring mechanism. Background Art

[0002] Defect detection plays a vital role in manufacturing, aerospace, electronics and other industries. With the development of technology and the advancement of production processes, the complexity of products continues to increase, and ensuring product quality and safety becomes increasingly important. Efficient defect detection can not only reduce production costs and improve product quality, but also improve customer satisfaction and reduce potential risks caused by defects. In these application scenarios, timely identification and repair of defects are necessary measures to ensure product performance and user safety.

[0003] Most of the current mainstream defect detection technologies rely on deep learning and computer vision technologies, which usually require a large number of labeled images for training. However, this data-dependent training method has several significant shortcomings:

[0004] 1. Scarcity of defect images: In many practical applications, the number of defect samples is often very limited, and it is difficult to obtain sufficient labeled data. This scarcity limits the learning ability of the model, resulting in poor performance of the model in real scenarios.

[0005] 2. Defect complexity: The appearance, form and nature of defects are often complex and changeable, and may vary from product to product. This makes it difficult for traditional defect detection methods to adapt to new defect types, resulting in a lower detection rate.

[0006] 3. High training cost: When new types of defects appear, traditional methods often require retraining the model, which increases the consumption of time and resources. This lack of flexibility does not meet the production needs of rapid iteration and updating.

[0007] In order to solve the above problems, zero-shot defect segmentation technology has gradually attracted attention in recent years. This technology mainly relies on the generalization ability of the image-language large model to identify and locate defect areas by designing text prompts. This method overcomes the problem of data scarcity to a certain extent and shows good application prospects. However, the existing text prompt design is usually coarse-grained and fails to effectively distinguish different images and defect types, resulting in limitations in detection performance.

[0008] In addition, there are still deficiencies in relying solely on textual cues for zero-shot detection. In many cases, it is not enough to rely solely on textual information for defect segmentation. In order to improve the accuracy and reliability of detection, it is particularly important to provide contrast features of normal images without increasing data requirements. The ICLR2024 paper "MUSC: ZERO-SHOTINDUSTRIAL ANOMALY CLASSIFICATION AND SEGMENTATION WITH MUTUAL SCORING OF THE UNLABELED IMAGES" first proposed a method of obtaining coarse-grained segmentation by comparing multiple input images, but the problem with this method is that it requires multiple input images, which limits the practical application of the method. Summary of the invention

[0009] The present invention aims to solve the shortcomings of the existing technology in zero-shot defect detection and proposes a zero-shot defect detection method based on a large language model and a self-scoring mechanism. The method uses zero-shot learning technology to reduce the dependence on large-scale data sets and achieve accurate segmentation of new defect categories, thereby overcoming the technical difficulty of rare defects that cannot be detected.

[0010] In order to solve the above problems, the present invention adopts the following technical solutions:

[0011] S1: Map the query image to the feature space through the feature extractor to obtain the feature representation of the image. In the feature space, calculate the adaptive average pooling values ​​of multiple image features at different scales in each encoding stage. Aggregate the average pooling features of three different scales in the same encoding stage through channel attention, and then aggregate the features of different stages by averaging.

[0012] S2: Design a dialogue template with the large language model ChatGPT, input the image to be segmented into the large language model, and ask the corresponding attributes of the image to obtain the description text of the image. Design a description that represents a flawless and perfect image. Design a lightweight network and input the feature map of the query image. Through this lightweight network, the feature map is mapped to an M-dimensional vector space for subsequent fusion processing with text features, and these features are input into the text encoder to generate text feature representation.

[0013] S3: Expand the feature map after step S1, perform kmeans clustering on the expanded features, and divide them into two categories in total. According to the characteristics of industrial defect images, the area of ​​the defect part is generally smaller than the background area. Therefore, the category with more clusters is the defect-free background feature, and the category with fewer clusters is the defect feature. Then calculate the score of each feature point through the intra-class and inter-class distances. The score is used as the weight for subsequent average calculations, and then all feature points in each category are weighted averaged to obtain the features of the defect-free background and defect part.

[0014] S4: Calculate the similarity between the foreground and background features obtained from the text branch and the image branch and the image features obtained from step S1 to obtain a score map based on text guidance and a score map based on image guidance. Finally, the final score map is obtained by weighted merging of the score maps based on text guidance and image guidance. A prediction mask is generated by comparing the numerical values ​​of the score maps in two dimensions to identify and segment the defect area.

[0015] Further features, in step S1, there are the following conditions:

[0016] The query image is mapped to the feature space, the query image is I, and it is mapped to the feature space by the feature extractor F to obtain the feature map: X l =F l (I); wherein H and W are the height and width of the feature map, and C is the number of feature channels. l is the lth stage, taken from 1, 2, 3, and 4. The feature patch at position (i, j) Represents the feature vector of the image at that position.

[0017] The calculation process of the multi-stage image feature multi-scale average pooling value is to extract multi-scale information on the feature map X l Perform average pooling operations at different scales. Assuming the scale parameter r is 1, 3, and 5, three average pooled features X can be obtained: L,1 ,X L,3 ,X L,5 , their dimensions remain unchanged, still H×W×C.

[0018] The channel attention aggregates the features after average pooling, first for X L,1 ,X L,3 ,X L,5 Perform global average pooling to obtain A L,1 ,A L,3 ,A L,5 , Then, we pass through a fully connected layer, an activation function layer, and a fully connected layer, and finally obtain the channel attention through the sigmoid function. Then, we combine the channel attention with the input vector XL,1 ,X L,3 ,X L,5 Multiply, and finally add the three results to obtain the aggregated feature X l Then calculate X l The average value of (l=1,2,3,4) obtains the features after multi-scale fusion of the image

[0019] Further features, in step S2, there are the following conditions:

[0020] The dialogue template with the large language model ChatGPT first gives the current dialogue scenario Env before the dialogue begins, then asks ChatGPT questions and expected results Q, and gives the query image I. ChatGPT outputs the description text of the image

[0021] T abnormal =ChatGPT(Env,Q,I);

[0022] The one described represents a flawless, perfect image description designed to read “This is a flawless industrial image.”

[0023] T normal = "This is a flawless industrial image."

[0024] These text messages T abnormal , T normal After the word segmentation, the foreground text and background text are processed into tensors of shape 1×77 respectively; since both are input into the text feature extractor as lists, a tensor T of shape 2×77 is finally obtained. 2×77 , where each row represents the features of foreground or background text.

[0025] Further features, a lightweight network is designed. It contains M learnable tensors, and includes a first linear layer, an activation layer, and a second linear layer; all text pairs correspond to the same learnable tensor; the first linear layer is used to perform a first linear change on the image feature to obtain a first linear change image feature; the activation layer is used to perform a nonlinear change on the first linear change image feature to obtain a nonlinear image feature; the second linear layer is used to perform a second linear change on the nonlinear image feature to obtain a second linear change image feature; the second linear change image feature is used as a bias to add the four learnable tensors corresponding to the foreground text and the background text respectively, to obtain four fused texts that fuse the query image. The fused text is only related to the query image input this time.

[0026] Query image feature map F QRepresents the global feature map of the query image, with a size of HxWxC, where H and W are the height and width of the feature map, and C is the number of channels. The defect feature block mapping network transforms the feature map F Q Mapped to M-dimensional vector space. The specific process is as follows: f Q It is a lightweight mapping network that queries the image feature map F Q Convert to an M-dimensional vector v Q .

[0027] In order to input these feature vectors into the text encoder, they need to be combined with the class token and padding tokens into a vector sequence that conforms to the text encoder input format. The class token is a tag used to identify that the current feature belongs to the query image, denoted by v class Padding tokens are used to ensure the fixed length of the input, denoted by v pad .

[0028] The complete query image input sequence V Q and the defect-free block image input sequence V P for:

[0029] V Q =[v Q ,v class ,v pad ,…,v pad ]

[0030] The combined input sequence V Q and T 2×77 Add them together and input them into the text encoder to generate a specific feature representation. Assume that the text encoder is the function TextEncoder(·), and its output is the encoded vector representation: E = TextEncoder(v Q +T 2×77 );

[0031] E is the encoded vector representation, which contains the feature information of the query image and the reference information of the text, providing support for subsequent similarity matching and segmentation prediction.

[0032] Further features, in step S3, there are the following conditions:

[0033] The self-scoring mechanism is to calculate the importance score of each block feature to the foreground (defective part) and background (non-defective part) to which it belongs. First, the feature X after multi-scale fusion of the image is expanded into a vector form with a size of HW×C. There are a total of HW patches. Then, k-means clustering is performed on the features of these HW patches, and the features are divided into two categories:

[0034] C1, C2 = KMeans(X); where C1 and C2 are two categories obtained by clustering algorithm. When |C1|>=|C2|, the former is the non-defective class and the latter is the defective class. When |C1|<|C2|, the latter is the non-defective class and the former is the defective class. By controlling the parameters of KMeans, the number of elements in the two categories can be kept as far apart as possible.

[0035] After being divided into two categories by kmeans, for each element, the distance D between it and the same type of elements is calculated. in,n , the distance D from non-same elements out,n , n represents the index of other elements. Design a in,n As the distance increases, D out,n The distance becomes smaller, and the score becomes smaller and smaller n , as the score of each element n.

[0036]

[0037] Calculate the score of each element n Finally, the weighted average of all elements in each category is calculated, and the weight is the score of the above design.

[0038]

[0039] When |C1|>=|C2|, x1 is the average feature of no defects, and x2 is the average feature of defects. When |C1|<|C2|, x2 is the average feature of no defects, and x1 is the average feature of defects. By controlling the parameters of KMeans, the number of elements in the two categories is kept as far apart as possible. The two are concatenated into x image .

[0040] Further features, in step S4, there are the following conditions:

[0041] The similarity between the foreground and background feature vectors and the query image feature X is calculated in the text guidance branch and the image guidance branch respectively. The method is to calculate the cosine similarity, and the formula is as follows:

[0042]

[0043] The score map S guided by the text textand the image-guided score map S image The weighted fusion is performed to obtain the final score map S. Then, a mask is generated based on the final score map through a thresholding operation. The final predicted mask can be used for defect detection tasks.

[0044] The present invention not only improves the accuracy of defect detection, but also realizes efficient zero-sample detection when defect samples are scarce, effectively reduces data requirements, improves the applicability and robustness of the detection model, and has good application prospects and practical value in the field of industrial defect detection. By adopting zero-sample learning technology, the dependence on large-scale training data sets is significantly reduced, so that new categories of defects can still be effectively detected in the absence of labeled data, solving the need for a large number of samples in traditional methods. Through the aggregation method of multi-scale feature extraction and channel attention, this method can effectively capture defect features at different scales, thereby improving the recognition accuracy of complex defects, especially in the detection of rare defects. Through the self-scoring mechanism and the text prompt design of large language model question and answer, it can flexibly adapt to new defect types without retraining the model. By mapping the query image features to vector spaces of different dimensions through a lightweight network, the diversity and expression ability of feature representation are improved, and the accuracy of subsequent processing is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart provided for an embodiment of the present invention.

[0046] Figure 2 This is a diagram of the overall network framework provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0048] A zero-shot defect detection method based on a large language model and a self-scoring mechanism, such as Figure 1 As shown, the following steps are included:

[0049] S1: Figure 2 As shown, a query image is encoded using a feature extractor to obtain a query image feature map;

[0050] In this embodiment, the query image is reshaped into a 1024x1024 image, and the image encoder uses the pre-trained ViT-L, which contains 24 transformer encoders. Each group of four is divided into a group, and the output of the last layer of transformer in each group (that is, the 6th, 12th, 18th, and 24th layers) is the feature vector of each stage. The image is divided into 64x64 16x16x3 patches, and the input is encoded to obtain a 768-dimensional vector containing a class token and 4096 768-dimensional vectors. That is, each input image will be processed to obtain 4 4096x768 feature vectors. The feature vector is reshaped into a 64x64x768 feature map, and then adaptive average pooling is calculated for each feature map, r=1, 3, 5, to ensure that the size of the pooling result remains unchanged. Then in the channel attention module, the three 64x64x768 feature maps corresponding to different r are globally averaged and pooled into a 1x1x768 vector. Then, after two fully connected layers, a probability vector is obtained through sigmoid, and the size is still 1x1x768. This vector is multiplied by the 64x64x768 feature map to obtain a feature map after channel attention calculation, with a size of 64x64x768. Then the average feature map under the four encoding stages is calculated.

[0051] S2: Figure 2 As shown, design a dialogue scenario description "We are conducting industrial defect detection. As an expert in industrial defect detection, you need to assist me in making an objective and accurate description of the defects. I will give you an industrial background image with defects. I need to know the background material, defect type, and material and defect feature description in the image. For example: [Me: An image] [Expert: This is an image of a wooden board surface with hole defects. The surface of the wooden board presents uniform wood grain, and the holes are irregular in shape and vary in size. They are usually concentrated in certain areas and may affect the integrity of the board.]". Then you only need to give ChatGPT one image each time to get a text description. The text description given by ChatGPT and the designed defect-free image text description are passed through the word segmenter to obtain two feature vectors of size 2x77. Design a lightweight network and input the feature map of the query image. Through this network, the feature map is mapped to the M-dimensional vector space for subsequent fusion processing with text features, and these features are input into the text encoder to generate text feature representation;

[0052] In this embodiment, M is 4, which is a 4-dimensional learnable vector. Taking into account the category token and padding, the lightweight network outputs a 77x1 word vector. The two word vectors are input into the text encoder to obtain a 768-dimensional text embedding.

[0053] S3: The kmeans class is 2, initialized by init = 'random', increased tol = 1e-2 convergence tolerance, and set n_init = 1. The element feature corresponding to each patch is 1x768. The final average feature size of the foreground and background is also 1x768.

[0054] S4: Perform weighted fusion of the text-guided score map and the image-guided score map to obtain the final score map S. Average weighting is used here. Then, a thresholding operation is performed to generate a mask based on the final score map. The final predicted mask can be used for defect detection tasks.

[0055] In this embodiment, the feature vectors guided by text and the feature vectors guided by image are both 2x768. Then multiply them with the query image feature 4096x768 to obtain a score map of 2x4096, which is reshaped to 64x64x2. The average of the score map guided by text and the score map guided by image is calculated, and then upsampled and reshaped into a tensor of 1024x1024x2. Finally, the maximum value index of each pixel position on the first dimension is calculated by the argmax operation, where 0 represents non-defect and 1 represents defect, which is the final prediction mask.

Claims

1. A zero-shot defect detection method based on a large language model and self-scoring mechanism, It is characterized by the following steps: S1: Map the query image to the feature space through the feature extractor to obtain the feature representation of the image; In the feature space, the adaptive average pooling values ​​of multiple image features at different scales in each encoding stage are calculated; the three average pooling features of different scales in the same encoding stage are aggregated by channel attention, and then the features of different stages are aggregated by averaging; S2: Design a dialogue template with the large language model ChatGPT, input the image to be segmented into the large language model, and ask the corresponding attributes of the image to obtain a description text of the image; design a description representing a flawless and perfect image; design a lightweight network and input the feature map of the query image; map the feature map to an M-dimensional vector space through this lightweight network for subsequent fusion processing with text features, and input these features into the text encoder to generate a text feature representation; S3: Expand the feature map after step S1, perform kmeans clustering on the expanded features, and divide them into two categories in total. According to the characteristics of industrial defect images, the area of ​​the defect part is generally smaller than the background area; therefore, the category with more clusters is the defect-free background feature, and the category with fewer clusters is the defect feature; then calculate the score of each feature point through the intra-class and inter-class distances, and use the score as the weight for subsequent average calculations, and then perform weighted average on all feature points in each category to obtain the features of the defect-free background and the defect part; S4: Calculate the similarity between the foreground and background features obtained on the text branch and the image branch and the image features obtained in step S1 to obtain a text-guided score map and an image-guided score map. Finally, the final score map is obtained by weighted merging of the text-guided and image-guided score maps. A prediction mask is generated by comparing the numerical values ​​of the score maps in two dimensions to identify and segment the defect area.

2. The zero-sample defect detection method based on a large language model and a self-scoring mechanism according to claim 1, further characterized by: In step S1, The query image is mapped to the feature space, the query image is I, and it is mapped to the feature space by the feature extractor F to obtain the feature map: X l =F l (I); wherein H and W are the height and width of the feature map, C is the number of feature channels; l is the first stage, taken from 1, 2, 3, 4; the feature patch at position (i, j) Represents the feature vector of the image at that position; The calculation process of the multi-stage image feature multi-scale average pooling value is to extract multi-scale information on the feature map X l Perform average pooling operations at different scales; assuming that the scale parameter r takes values ​​of 1, 3, and 5, three average pooled features X are obtained L,1 ,X L,3 ,X L,5 , their dimensions remain unchanged, still H×W×C; The channel attention aggregates the features after average pooling, first for X L,1 ,X L,3 ,X L,5 Perform global average pooling to obtain A L,1 ,A L,3 ,A L,5 , Then, we pass through a fully connected layer, an activation function layer, and a fully connected layer, and finally obtain the channel attention through the sigmoid function. Then, we combine the channel attention with the input vector X L,1 ,X L,3 ,X L,5 Multiply them, and finally add the three results to obtain the aggregated feature X l ; Then calculate X l The average value of (l=1,2,3,4) obtains the features after multi-scale fusion of the image 3. The zero-sample defect detection method based on a large language model and a self-scoring mechanism according to claim 1, further characterized by: In step S2, The dialogue template with the large language model ChatGPT first gives the current dialogue scenario Env before the dialogue begins, then asks ChatGPT questions and expected results Q, and gives the query image I. ChatGPT outputs the description text of the image T abnormal =ChatGPT(Env,Q,I); The one represents a flawless, perfect image description, designed as "This is a flawless industrial image;" T normal = "This is a flawless industrial image;" These text messages T abnormal , T normal After the word segmentation, the foreground text and background text are processed into tensors of shape 1×77 respectively; since both are input into the text feature extractor as lists, a tensor T of shape 2×77 is finally obtained. 2×77 , where each row represents the features of foreground or background text; Query image feature map F Q Represents the global feature map of the query image, with a size of HxWxC, where H and W are the height and width of the feature map, and C is the number of channels; the defect feature block mapping network transforms the feature map F Q Mapped to M-dimensional vector space; the specific process is as follows: f Q It is a lightweight mapping network that queries the image feature map F Q Convert to an M-dimensional vector v Q ; In order to input these feature vectors into the text encoder, they need to be combined with class tokens and padding tokens into a vector sequence that conforms to the text encoder input format; the class token is a token used to identify that the current feature belongs to the query image, denoted by v class ; Padding tokens are used to ensure the fixed length of the input, denoted by v pad ; Therefore, the complete query image input sequence V Q and the defect-free block image input sequence V P for: V Q =[v Q ,v class ,v pad ,…,v pad ] The combined input sequence V Q and T 2×77 Add and input into the text encoder to generate a specific feature representation; Assume that the text encoder is a function TextEncoder(·), whose output is the encoded vector representation: E = TextEncoder(v Q +T 2×77 ); E is the encoded vector representation, which contains the feature information of the query image and the reference information of the text, providing support for subsequent similarity matching and segmentation prediction.

4. The zero-sample defect detection method based on a large language model and a self-scoring mechanism according to claim 3 is further characterized by: A lightweight network is designed; it contains M learnable tensors, and includes a first linear layer, an activation layer, and a second linear layer; all text pairs correspond to the same learnable tensor; the first linear layer is used to perform a first linear change on the image features to obtain a first linear change image feature; The activation layer is used to perform a nonlinear change on the first linear change image feature to obtain a nonlinear image feature; the second linear layer is used to perform a second linear change on the nonlinear image feature to obtain a second linear change image feature; The second linear variation image feature is used as a bias to add the four learnable tensors corresponding to the foreground text and the background text respectively, so as to obtain four fused texts fused with the query image; the fused text is only related to the query image input this time.

5. The zero-sample defect detection method based on a large language model and a self-scoring mechanism according to claim 1, further characterized by: In step S3, The self-scoring mechanism is to calculate the importance score of each block feature to the foreground (defective part) and background (non-defective part) to which it belongs; first, the feature X after multi-scale fusion of the image is expanded into a vector form with a size of HW×C; there are a total of HW patches; then, the features of these HW patches are clustered by k-means to divide the features into two categories; C1,C2=KMeans(X) Among them, C1 and C2 are two categories obtained by clustering algorithm; When |C1|>=|C2|, the former is the non-defective class and the latter is the defective class; when |C1|<|C2|, the latter is the non-defective class and the former is the defective class; by controlling the parameters of KMeans, the number of elements in the two classes is kept as far apart as possible; After being divided into two categories by kmeans, for each element, the distance D between it and the same type of elements is calculated. in,n , the distance D from non-same elements out,n , n represents the index of other elements; design a in,n As the distance increases, D out,n The distance becomes smaller, and the score becomes smaller and smaller n , as the score of each element n; Calculate the score of each element n After that, the weighted average of all elements in each category is calculated, and the weight is the score of the above design; When |C1|>=|C2|, x1 is the average feature of no defects, and x2 is the average feature of defects; when |C1|<|C2|, x2 is the average feature of no defects, and x1 is the average feature of defects; by controlling the parameters of KMeans, the number of elements in the two categories is kept as far apart as possible; the two are concatenated into x image .

6. The zero-sample defect detection method based on a large language model and a self-scoring mechanism according to claim 1, further characterized by: In step S4, The similarity between the foreground and background feature vectors and the query image feature X in the text guidance branch and the image guidance branch is calculated respectively; the method is to calculate the cosine similarity, and the formula is as follows: The score map S guided by the text text and the image-guided score map S image Weighted fusion is performed to obtain the final score map S; then a mask is generated based on the final score map through a thresholding operation; the final predicted mask is used for defect detection tasks.

Images