Multi-modal zero sample defect detection method based on search engine and human guidance

Abstract

According to the multi-modal zero-sample defect detection method based on the search engine and human guidance, the exogenous background image is automatically acquired through the search engine based on the keyword, and the acquisition process of the background image is greatly simplified; meanwhile, a textual feature extractor is enhanced by introducing a texture element network, so that the textual feature extractor can be combined with texture information in an exogenous image, accurate segmentation of the defect is realized when an unseen material or an unseen defect category is processed, and the limitation of the prior art in facing rare defects is overcome; that is to say, the problem that a traditional defect detection algorithm depends on a large-scale data set is solved by introducing a multi-modal zero sample learning method and combining feature alignment of the text and the exogenous texture image.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine learning, and in particular relates to a multimodal zero-sample defect detection method based on search engine and human guidance. Background Art

[0002] In the manufacturing industry, complex process flows can lead to various defects on the surface of products, so defect segmentation plays a vital role in ensuring product quality. Visual defect segmentation is a non-contact defect detection technology with the advantages of flexibility, accuracy, and density. In addition, deep learning has significantly improved the performance of this task. Existing deep learning methods have achieved remarkable results in defect segmentation tasks, but they are heavily dependent on large-scale annotated datasets, and their performance often drops sharply when applied to unseen defect categories. Unseen defect segmentation tasks can be divided into two categories: unseen defect segmentation in known backgrounds and unseen defect segmentation in unseen backgrounds. Unseen defect segmentation tasks face the problems of insufficient defect samples, scale changes, texture shift, and high similarity between foreground and background in images.

[0003] Few-shot learning techniques can be used to solve the reliance of sample defect detection algorithms on large-scale datasets. However, traditional few-shot defect segmentation methods require a small number of labeled defect samples, which are difficult to collect and annotate in advance in many industrial fields, making these methods inapplicable. In addition, although existing zero-shot methods do not require labeled samples, they rely on the acquisition of high-quality background images, which is also challenging to obtain in some industrial scenarios.

[0004] The journal paper "IEEE Transactions on Instrumentation and Measurement, vol.73, id.3341105" proposed two methods to segment unseen defects. The first is a defect segmentation method based on background image guidance. By cropping the segmented image in each segmentation, a high-quality background image is obtained as guidance information, and the features of the background image and the segmented image are fused and decoded, good segmentation results can be obtained on unseen backgrounds. The second method proposed in the paper is a defect segmentation method based on text guidance. By describing the foreground and background of the segmented image, the text and image features are aligned with the help of a multimodal large language model, and finally the segmentation result is obtained through the similarity map method.

[0005] That is to say, the traditional method based on background image guidance needs to ensure the clarity and homology of the background image, which is difficult to obtain in some industrial scenarios. Although the method based on text guidance greatly reduces the difficulty of obtaining guidance information, the effect of text guidance method on unseen backgrounds is very poor. Summary of the invention

[0006] In order to solve the problem of traditional defect detection algorithms' dependence on large-scale data sets, the present invention provides a multimodal zero-sample defect detection method based on search engine and human guidance, which can not only improve the defect segmentation performance on unseen backgrounds, but also reduce the difficulty of data acquisition.

[0007] A multimodal zero-shot defect detection method based on search engine and human guidance adopts a defect detection large model to perform defect detection on a query image, wherein the defect detection large model includes an image feature extractor, a prompt fine-tuner, a text feature extractor, and a multi-head attention mechanism, and the training method of the large model includes the following steps:

[0008] Inputting material keywords related to the query image into a keyword-driven search engine, and obtaining a material image most similar to the query image as an external texture image;

[0009] The query image and the exogenous texture image are respectively input into the image feature extractor, and query image features and exogenous texture image features are correspondingly obtained;

[0010] The foreground text and background text in the text guidance content are respectively combined with the exogenous texture image features and input into the prompt fine tuner, so as to obtain the foreground fusion text and background fusion text fused with the exogenous texture image features; wherein the foreground text contains the defect category information to be extracted;

[0011] Input the foreground fused text and the background fused text into the text feature extractor respectively to obtain primary foreground text features and primary background text features;

[0012] A multi-head self-attention mechanism is used to enhance the primary background text features and the primary foreground text features to obtain the final background text features and foreground text features.

[0013] The result of multiplying the foreground text feature and the query image feature is used as the foreground feature;

[0014] The result of averaging the first dimension of the exogenous texture image features is used as the background feature;

[0015] The foreground features and background features are concatenated according to the channel dimension to obtain multimodal features containing visual information and text information;

[0016] Perform feature value comparison operation on multimodal features to obtain a prediction mask containing only two values ​​0 and 1, where 0 represents background and 1 represents defect. When the background feature value at the same position of different channels in the multimodal features is greater than the foreground feature value, the corresponding position of the prediction mask is 0, otherwise, the corresponding position of the prediction mask is 1;

[0017] The cosine similarity between background text features and background features is obtained as the loss function, and the detection model is back-propagated according to the gradient direction of the loss function to change the network parameters of the image feature extractor, prompt fine-tuner, text feature extractor, and multi-head attention mechanism until the set number of training iterations is reached.

[0018] Further, the prompt fine-tuner includes a texture element network and four learnable tensors, and the texture element network includes a first linear layer, an activation layer, and a second linear layer; wherein all text guidance contents correspond to the same learnable tensor;

[0019] The first linear layer is used to perform a first linear change on the exogenous texture image feature to obtain a first linearly changed texture image feature;

[0020] The activation layer is used to perform nonlinear changes on the first linearly changing texture image feature to obtain a nonlinear texture image feature;

[0021] The second linear layer is used to perform a second linear change on the nonlinear texture image feature to obtain a second linearly changed texture image feature;

[0022] The second linearly varying texture image feature is used as a bias and added to four learnable tensors corresponding to the text guidance content to obtain four fused texts that incorporate exogenous texture image features, wherein the fused text includes a foreground fused text and a background fused text.

[0023] Furthermore, the method for obtaining the external texture image is:

[0024] After inputting material keywords related to the query image into the keyword-driven search engine, the similarity between the first five images in the search results given by the search engine and the query image is calculated through the image similarity algorithm, and the similarity scores of the five images are provided for reference by human experts, who then select one from the five images as the final exogenous texture image.

[0025] Furthermore, after the exogenous texture image is obtained, the exogenous texture image is processed by a color migration algorithm, and then the processed exogenous texture image whose difference with the query image has been reduced is subjected to subsequent steps.

[0026] Furthermore, the template of the material keyword is: "HD {material type} material", wherein HD means that the resolution of the image is at least 1280×720P.

[0027] Furthermore, material types include: metal, plastic, ceramic, silicone, glass, wood, fabric, leather, paint, ink, coating, and composite materials.

[0028] Furthermore, the template of the foreground text is: "{material type} with {defect category} defect", and the template of the background text is: "other".

[0029] Furthermore, the defect categories include color defects, hole defects, crack defects, pore defects, inclusion defects, structural unevenness defects, and folding defects.

[0030] Furthermore, when using the defect detection large model to perform defect detection on the query image, if the material type of the query image to be detected belongs to the material type that has appeared during the training of the defect detection large model, the foreground text and background text in the text guidance content corresponding to the query image to be detected do not need to enter the prompt fine-tuner, but are directly input into the text feature extractor to be obtained as primary background text features and primary foreground text features.

[0031] Beneficial effects:

[0032] 1. The present invention provides a multimodal zero-shot defect detection method based on search engine and human guidance. The exogenous background image is automatically acquired through a keyword-based search engine, which greatly simplifies the background image acquisition process. At the same time, the present invention introduces a texture element network to enhance the text feature extractor, so that it can combine the texture information in the exogenous image. When processing unseen materials or unseen defect categories, it can achieve accurate segmentation of defects, thereby overcoming the limitations of the prior art when facing rare defects. In other words, the present invention solves the problem of traditional defect detection algorithms relying on large-scale data sets by introducing a multimodal zero-shot learning method and combining the feature alignment of text and exogenous texture images.

[0033] 2. The present invention provides a multimodal zero-sample defect detection method based on search engine and human guidance. When the keyword-based search engine automatically obtains the external background image, it also uses the image similarity algorithm and manual decision-making to select the most matching material image, which greatly simplifies the background image acquisition process and is particularly suitable for complex or scarce background scenes;

[0034] 3. The present invention provides a multimodal zero-shot defect detection method based on search engine and human guidance. In the feature fusion of the model, the alignment of text features and image features is strengthened through the multi-head self-attention mechanism, thereby ensuring the deep integration of the information of the two in the multimodal fusion process; multi-scale feature upsampling and fusion not only enhance the ability to restore details, but also improve the robustness of the model to different scenarios and defect categories; in practical applications, the present invention realizes accurate detection of defects in complex industrial scenarios, especially can process unprecedented defect types and materials, and provides a reliable solution for the automated detection of rare defects in industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a multimodal zero-sample defect detection method based on search engine and human guidance provided by the present invention;

[0036] Figure 2 A schematic diagram of the multimodal feature acquisition process provided by the present invention;

[0037] Figure 3 A schematic diagram of the process of obtaining an external background image provided by the present invention;

[0038] Figure 4 The functional block diagram of the prompt fine-tuner provided by the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0040] A multimodal zero-shot defect detection method based on search engine and human guidance uses a defect detection large model to perform defect detection on a query image, wherein the defect detection large model includes an image feature extractor, a hint fine-tuner, a text feature extractor, a multi-head attention mechanism, such as Figure 1 As shown, the training method of the large model includes the following steps:

[0041] S1: Input material keywords related to the query image into a keyword-driven search engine, and obtain the material image most similar to the query image as an external texture image;

[0042] The method for obtaining the external texture image is:

[0043] After inputting material keywords related to the query image into the keyword-driven search engine, the similarity between the first five images in the search results given by the search engine and the query image is calculated through the image similarity algorithm, and the similarity scores of the five images are provided for reference by human experts, who then select one from the five images as the final exogenous texture image.

[0044] Furthermore, after the exogenous texture image is obtained, the exogenous texture image is processed by a color migration algorithm, and then the processed exogenous texture image whose difference with the query image has been reduced is subjected to subsequent steps.

[0045] For example, in the present invention, Figure 3As shown in the figure, the template for designing keyword query is "HD {material type} material", and HD means that the resolution of the image is at least 1280×720P, such as "HD wood material". The material types that can be input are judged by human experts based on common sense and professional knowledge, including metal, plastic, ceramic, silicone, glass, wood, fabric, leather, paint, ink, coating, composite material, etc. The query obtains the first five relevant images, and then the queried images and the input images are converted from three-channel images to binary images, and the average of the image similarity in the three-channel mode and the image similarity in the binary mode is calculated as the final similarity evaluation score. Human experts combine professional knowledge and algorithm evaluation results to finally decide which external source image to select as guidance information.

[0046] It can be seen that the present invention obtains material images through a keyword-driven search engine, which effectively simplifies the process of obtaining high-quality background images; the operator only needs to input the corresponding keywords, and the system will automatically collect material images similar to the background of the target segmented image from the network; then, the system performs a preliminary evaluation on the first five images searched out through an image similarity algorithm, and returns the evaluation results to human experts, who finally determine the appropriate exogenous texture image in combination with manual judgment; finally, the color migration algorithm is used to narrow the gap between the input query image and the exogenous texture image.

[0047] S2: Input the query image and the exogenous texture image into the image feature extractor respectively, and obtain the query image features and the exogenous texture image features accordingly;

[0048] like Figure 2 As shown in the figure, the input image is preprocessed into a tensor of 160×160×3, and a feature extractor with ResNet101 as the backbone is selected to extract features from the tensor. Then, the image features are gradually abstracted into multi-scale feature maps. As the network depth increases, the feature map size gradually decreases, and the number of channels increases from 256 to 2048, gradually extracting more advanced semantic information. Specifically, the size of the first layer feature map obtained by ResNet101 is 256×80×80. 512 convolution kernels are used to downsample the first layer feature map to a 512×40×40 feature map of the second layer. Similar downsampling operations are used to obtain a 1024×20×20 feature map of the third layer and a 2048×10×10 feature map of the fourth layer. After this process, the image features are connected through a multi-scale network, the high-dimensional features of the deep layer are upsampled and added to the tensor of the previous layer, and the feature maps of different depths of 512×160×160 are fused to obtain the final output result of the backbone network.

[0049] It can be seen that the multi-scale feature fusion of the present invention enables the image feature extractor to retain both low-level detail information and high-level semantic information when generating the final image features, and can better restore the details of the image, especially when dealing with defect detection tasks in complex backgrounds; by aligning and fusing feature maps of different resolutions, the image feature extractor can simultaneously utilize the rich semantic information in the low-resolution feature map and the spatial details of the high-resolution feature map to achieve a more comprehensive feature representation.

[0050] S3: Combining the foreground text and background text in the text guidance content with the exogenous texture image features and inputting them into the prompt fine tuner, thereby obtaining the foreground fused text and background fused text fused with the exogenous texture image features; wherein the foreground text contains the defect category information to be extracted;

[0051] It should be noted that the pattern of the foreground text of the present invention is "{material type} with {defect category} defect", wherein the defect categories include color defects, hole defects, crack defects, pore defects, inclusion defects, structural uneven defects, folding defects, etc.; the background text is more general, usually expressed as "others".

[0052] For example, Figure 3 As shown in the figure, the foreground text can be written as "wood with color defects", while the background text is "other". These text information is segmented by the tokenizer, and the foreground text and background text are processed into tensors of shape 1×77 respectively; since both are input into the text feature extractor as lists, a tensor of shape 2×77 is finally obtained, in which each row represents the features of the foreground or background text.

[0053] Furthermore, if Figure 4 As shown, the prompt fine-tuner includes a texture element network and four learnable tensors, and the texture element network includes a first linear layer, an activation layer, and a second linear layer; all text guidance contents correspond to the same learnable tensor;

[0054] The first linear layer is used to perform a first linear change on the exogenous texture image feature to obtain a first linearly changed texture image feature;

[0055] The activation layer is used to perform nonlinear changes on the first linearly changing texture image feature to obtain a nonlinear texture image feature;

[0056] The second linear layer is used to perform a second linear change on the nonlinear texture image feature to obtain a second linearly changed texture image feature;

[0057] The second linearly varying texture image feature is used as a bias and added to the four learnable tensors corresponding to the text guidance content to obtain four fused texts that are fused with the exogenous texture image, wherein the fused text includes a foreground fused text and a background fused text; it should be noted that the fused text is only related to the exogenous texture image input this time.

[0058] It should be noted that for material types and material defects that the defect detection large model has not seen, the present invention proposes a texture element network solution, which provides more relevant context information for the text feature extractor by combining the features of the exogenous texture image, helping the defect detection large model to obtain more accurate feature expressions in new materials and scenes. The exogenous texture image features are input into the texture element network and added with the text features to obtain a new tensor. Secondly, a multi-head self-attention mechanism is introduced after the text feature extractor to process the obtained 2×512 text features and further enhance them into features of the same dimension. However, when the defect detection large model is used to perform defect detection on the query image, if the material type of the query image to be detected belongs to the material type that has appeared during the training of the defect detection large model, the foreground text and background text in the text guidance content corresponding to the query image to be detected do not need to enter the prompt fine-tuner, but are directly input into the text feature extractor to obtain primary background text features and primary foreground text features.

[0059] It should be noted that V1~VM is a learnable tensor with a custom length. Here we set the length to M=4. The dimension of each position is determined by the selected backbone network. The size of each learnable tensor is (512, 1), so the total size of V1~VM is (512, 4); Class is the token of each given class, which is also a 512-dimensional tensor. This tensor comes from the manually input foreground text and background text, and the token is calculated by the index value of Class in the given list; the exogenous texture image is obtained by the image feature extractor. A tensor of (512, 160, 160) is obtained. This tensor is obtained by the texture element network. A tensor of (512, 4) is obtained, which is the bias. Adding the bias to V1~VM, the text content will contain the information of the texture image, realizing the defect segmentation of unseen categories and materials.

[0060] S4: inputting the foreground fused text and the background fused text into the text feature extractor respectively to obtain primary foreground text features and primary background text features;

[0061] S5: Use a multi-head self-attention mechanism to enhance the primary background text features and the primary foreground text features to obtain the final background text features and foreground text features;

[0062] S6: multiplying the foreground text feature by the query image feature as the foreground feature;

[0063] S7: The result of averaging the first dimension of the exogenous texture image features is used as the background feature;

[0064] S8: Concatenate foreground features and background features according to the channel dimension to obtain multimodal features containing visual information and text information;

[0065] For example, the foreground text feature (512, 1) is transposed to (1, 512) and multiplied with the query image feature (512, 160, 160) to obtain the foreground feature (1, 160, 160). The exogenous background image feature (512, 160, 160) is averaged along the first dimension to obtain the background feature (1, 160, 160). The two are concatenated according to the channel dimension in order to integrate information from different modalities into a common feature space, forming a multimodal feature containing visual and textual information with a shape of (2, 160, 160).

[0066] S9: performing a feature value comparison operation on the multimodal feature to obtain a prediction mask containing only two values ​​0 and 1, wherein 0 represents background and 1 represents a defect. When the background feature value at the same position of different channels in the multimodal feature is greater than the foreground feature value, the corresponding position of the prediction mask is 0, otherwise, the corresponding position of the prediction mask is 1;

[0067] S10: Obtain the cosine similarity between the background text features and the background features as the loss function, and back-propagate the detection model according to the gradient direction of the loss function to change the network parameters of the image feature extractor, prompt fine-tuner, text feature extractor, and multi-head attention mechanism until the set number of training iterations is reached.

[0068] The experimental results of the present invention on the MVTec-Seen, MVTec-Unseen and Defect-4i datasets show excellent zero-sample segmentation performance, especially in unseen categories and scenes. Specifically, in the MVTec-Seen dataset, the present invention surpasses the most advanced algorithms in multiple segmentation evaluation indicators (such as mIoU and FB-IoU), and the segmentation accuracy is improved by 3.42% to 7.39%. Compared with the pure text-guided segmentation method, the present invention combines exogenous texture images and multimodal fusion to make the segmentation results more accurate in edge sharpness and complex scene processing, thereby enhancing the robustness and adaptability of the model. In the MVTec-Unseen dataset, although the effect of text-guided segmentation is slightly inferior to that of image-guided segmentation, the performance of the present invention in unseen defects and scenes is still close to the classic few-sample segmentation algorithm HSNet, demonstrating its advantages under sparse information conditions. Especially in the Defect-4i dataset, the present invention demonstrates excellent segmentation capabilities when faced with complex images of metal and non-metal surfaces, surpassing the pure text-guided method, and even being on par with or better than the image-guided method in some scenarios.

[0069] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal zero-shot defect detection method based on search engine and human guidance, characterized in that: A defect detection large model is used to perform defect detection on the query image, wherein the defect detection large model includes an image feature extractor, a prompt fine-tuner, a text feature extractor, and a multi-head attention mechanism, and the training method of the large model includes the following steps: Inputting material keywords related to the query image into a keyword-driven search engine, and obtaining a material image most similar to the query image as an external texture image; The query image and the exogenous texture image are respectively input into the image feature extractor, and query image features and exogenous texture image features are correspondingly obtained; The foreground text and background text in the text guidance content are respectively combined with the exogenous texture image features and input into the prompt fine tuner, so as to obtain the foreground fusion text and background fusion text fused with the exogenous texture image features; wherein the foreground text contains the defect category information to be extracted; Input the foreground fused text and the background fused text into the text feature extractor respectively to obtain primary foreground text features and primary background text features; A multi-head self-attention mechanism is used to enhance the primary background text features and the primary foreground text features to obtain the final background text features and foreground text features. The result of multiplying the foreground text feature and the query image feature is used as the foreground feature; The result of averaging the first dimension of the exogenous texture image features is used as the background feature; The foreground features and background features are concatenated according to the channel dimension to obtain multimodal features containing visual information and text information; Perform feature value comparison operation on multimodal features to obtain a prediction mask containing only two values ​​0 and 1, where 0 represents background and 1 represents defect. When the background feature value at the same position of different channels in the multimodal features is greater than the foreground feature value, the corresponding position of the prediction mask is 0, otherwise, the corresponding position of the prediction mask is 1; The cosine similarity between background text features and background features is obtained as the loss function, and the detection model is back-propagated according to the gradient direction of the loss function to change the network parameters of the image feature extractor, prompt fine-tuner, text feature extractor, and multi-head attention mechanism until the set number of training iterations is reached.

2. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 1, characterized in that: The prompt fine-tuner includes a texture element network and four learnable tensors, and the texture element network includes a first linear layer, an activation layer, and a second linear layer; wherein all text guidance contents correspond to the same learnable tensor; The first linear layer is used to perform a first linear change on the exogenous texture image feature to obtain a first linearly changed texture image feature; The activation layer is used to perform nonlinear changes on the first linearly changing texture image feature to obtain a nonlinear texture image feature; The second linear layer is used to perform a second linear change on the nonlinear texture image feature to obtain a second linearly changed texture image feature; The second linearly varying texture image feature is used as a bias and added to four learnable tensors corresponding to the text guidance content to obtain four fused texts that incorporate exogenous texture image features, wherein the fused text includes a foreground fused text and a background fused text.

3. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 1, characterized in that: The method for obtaining the external texture image is: After inputting material keywords related to the query image into the keyword-driven search engine, the similarity between the first five images in the search results given by the search engine and the query image is calculated through the image similarity algorithm, and the similarity scores of the five images are provided for reference by human experts, who then select one from the five images as the final exogenous texture image.

4. The multimodal zero-shot defect detection method based on search engine and human guidance according to claim 1, characterized in that: After the exogenous texture image is acquired, the exogenous texture image is processed by a color migration algorithm, and then the processed exogenous texture image whose difference with the query image is reduced is used to perform subsequent steps.

5. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 1, characterized in that: The template of the material keyword is: "HD {material type} material", where HD means that the resolution of the image is at least 1280×720P.

6. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 5, characterized in that: Material types include: metal, plastic, ceramic, silicone, glass, wood, fabric, leather, paint, ink, coating, composite materials.

7. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 1, characterized in that: The template for the foreground text is: "{material type} with {defect category} defect", and the template for the background text is: "Other".

8. The multimodal zero-sample defect detection method based on search engine and human guidance according to claim 7, characterized in that: Defect categories include color defects, hole defects, crack defects, pore defects, inclusion defects, structural uneven defects, and folding defects.

9. The multimodal zero-shot defect detection method based on search engine and human guidance according to claim 1, characterized in that: When using the defect detection large model to perform defect detection on a query image, if the material type of the query image to be detected belongs to the material type that has appeared during the training of the defect detection large model, the foreground text and background text in the text guidance content corresponding to the query image to be detected do not need to enter the prompt fine-tuner, but can be directly input into the text feature extractor to be obtained as primary background text features and primary foreground text features.

Images