A defect detection method and apparatus thereof

Abstract

The application provides a defect detection method and device. The defect detection method comprises: obtaining an image to be detected and corresponding preset prompt text; the preset prompt text is a description of a category of a detection target of interest in the image to be detected; the preset prompt text is converted into structured text; and the category and / or position of the detection target of interest in the image to be detected are predicted by taking the structured text as a guide. The method can solve the problem of detecting defects of known categories and / or unknown categories in the image to be detected, and improve the detection performance of the defects. The device comprises a text structuring module for structuring the preset prompt text to obtain structured text; an image acquisition and prompt module for obtaining the image to be detected and the corresponding preset prompt text; and a data processing module for predicting the category and / or position of the detection target by taking the structured text as a guide.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a defect detection method and apparatus. Background Technology

[0002] Currently, image defect detection methods typically rely on manually collected large amounts of data, which are then categorized and labeled accordingly. A closed set of category data—that is, a set of known defect types obtained through the labeling—is then used to train a defect detection model, thereby enabling the model to detect defects in the image. However, in actual industrial production, due to the diversity and unknown nature of defect categories, the defect detection model trained using a closed set of category data performs poorly in identifying open set data—that is, a set of categories with unknown defect types.

[0003] The aforementioned diversity primarily refers to the wide variety of defects in images, encompassing different shapes, sizes, locations, colors, and textures. For example, the surface of a mobile phone case may exhibit different types of defects such as cracks, dents, and indentations, with varying degrees of intensity, resulting in significant diversity. The aforementioned unknown nature stems from the fact that, in industrial production, it is impossible for experts to collect all possible defects in industrial products. Therefore, the defect detection model trained using a closed set of category data cannot effectively detect unknown defects in industrial products. Furthermore, the lack of clear, missing, or incomplete labeling of some defect categories in industrial products further increases the difficulty for the aforementioned defect detection model in identifying unknown defects. High-precision defect detection models require comprehensive and sufficient training samples to ensure performance. However, because the occurrence of unknown defect categories in images is uncertain, defect detection models are prone to missed or false detections.

[0004] Therefore, it is necessary to improve upon the shortcomings of the existing technology. Summary of the Invention

[0005] This invention provides a defect detection method and apparatus, which mainly solves the problem of detecting defects of known and / or unknown categories in images to be detected, and improves the detection performance of the above-mentioned defects.

[0006] According to a first aspect, one embodiment provides a defect detection method. The defect detection method includes:

[0007] Acquire the image to be detected and the corresponding preset prompt text; wherein, the preset prompt text is a description of the category of the target of interest in the image to be detected;

[0008] Convert the preset prompt text into structured text;

[0009] Guided by the structured text, the category and / or location of the target of interest in the image x to be detected are predicted, thereby obtaining the category information and / or location information of the target of interest.

[0010] In one embodiment, converting the preset prompt text into structured text includes: performing structuring processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

[0011] In one embodiment, the step of using the structured text as guidance to predict the category and / or location of the target of interest in the image to be detected, thereby obtaining the category information and / or location information of the target of interest, includes:

[0012] Text features are obtained by text encoding the structured text; multi-scale feature extraction is performed on the image to be detected to obtain an n-layer first feature map v. i Where i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the more likely n ≥ 2;

[0013] The n-layer first feature map v i Each feature is fused with the text features to obtain an n-layer fused feature map;

[0014] The category information of the detected target in the image to be detected is obtained by predicting the category of the detected target using the text features and the n-layer fused feature map; and / or, the position information of the detected target in the image to be detected is obtained by predicting the position of the detected target using the n-layer fused feature map.

[0015] In one embodiment, the n-layer first feature map v i The text features are fused to obtain an n-layer fused feature map, including:

[0016] An n-layer text attention network is used to process the first feature map v in the nth layer. i The text features are fused to obtain an n-layer fused feature map; wherein, for the nth layer text attention network, the first feature map of the nth layer is used as the first input feature map, the text features are used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain the output feature map, and the obtained output feature map is used as the nth layer fused feature map;

[0017] For the text attention network at layer k, a first convolution operation and an upsampling operation are performed on the fused feature map at layer (k+1) to obtain a corresponding upsampled feature map, wherein the resolution of the upsampled feature map is the same as the resolution of the first feature map at layer k; the first feature map at layer k and the corresponding upsampled feature map are added element-wise to obtain an element-wise added feature map; the element-wise added feature map is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as the fused feature map at layer k, where k = n-1, ..., 1;

[0018] The text attention network is used to enhance the first feature map v. i Location information of the detected target;

[0019] In one embodiment, the step of inputting the first input feature map and the second input feature map into the text attention network to obtain the output feature map includes:

[0020] The first input feature map d and the second input feature map e, which are input into the text attention network, are used to calculate the first text-region similarity matrix using the following expression:

[0021]

[0022] Wherein, Conv(d) represents the second convolutional feature map obtained by performing a second convolution operation on the first input feature map d, Norm(e) represents the feature map obtained by performing a first normalization process on the text features, μ and δ are respectively a learnable first scaling factor and a learnable first translation factor, and the... This represents the matrix multiplication operation;

[0023] The first text and region similarity matrix s1 is maximized to obtain the first text and region most similarity matrix.

[0024] The first nonlinear activation process is applied to the first text-region most similarity matrix to obtain a text-region attention map;

[0025] An attention-enhanced feature map is obtained by performing an element-wise multiplication operation on the text-region attention map and the first input feature map d input to the text attention network;

[0026] A third convolution operation is performed on the attention enhancement feature map to obtain the corresponding fused feature map, and the corresponding fused feature map is used as the output feature map of the text attention network.

[0027] In one embodiment, the step of predicting the location of the detected target in the image to be detected using the n-layer fused feature map to obtain the location information of the detected target includes:

[0028] The n-layer fused feature map is subjected to a fourth convolution operation to output a prediction box corresponding to the fused feature map; wherein the prediction box is used to represent the position information of the detected target.

[0029] In one embodiment, the step of predicting the category of the detected target using the text features and the n-layer fused feature map to obtain the category information of the detected target in the fused feature map includes:

[0030] The fusion feature map f with the j-th layer is calculated using the text features, the fused feature map, and the following expression. j The corresponding second text-region similarity matrix:

[0031]

[0032] Wherein, Norm(e) represents the feature map obtained after the second normalization process on the text features, α and β are the learnable second scaling factor and learnable second translation factor, respectively. This represents the matrix multiplication operation;

[0033] The fused feature map f with the j-th layer is calculated using the second text-region similarity matrix and the following expression. j Corresponding category probability feature map:

[0034] p j = sigmoid(s2);

[0035] Wherein, sigmoid represents the second nonlinear activation process, and the fused feature map f with the j-th layer... j The corresponding category probability feature map p j f is used to characterize the fusion feature map with the j-th layer. j The probability that each pixel belongs to the detected target of the category.

[0036] In one embodiment, the defect detection method is implemented based on a defect detection model, which includes a data processing module. The step of using the structured text as guidance to predict the category and / or location of the target of interest in the image x to be detected, thereby obtaining the category information and / or location information of the target of interest, includes: the data processing module using the structured text as guidance to predict the category and / or location of the target of interest in the image x to be detected, thereby obtaining the category information and / or location information of the target of interest.

[0037] In one embodiment, the data processing module includes a text encoding submodule, an image encoding submodule, and a feature fusion submodule. The data processing module also includes a localization submodule and / or a classification submodule.

[0038] The step of obtaining text features by text encoding the structured text includes: using the text encoding submodule to perform text encoding on the structured text to obtain text features;

[0039] Specifically, the process of performing multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v i ,include:

[0040] The image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v. i ; wherein, the first feature map v of the nth layer i The features are fused with the text features to obtain an n-layer fused feature map f. j This includes: using the feature fusion submodule to process the n-layer first feature map v i The features are fused with the text features to obtain an n-layer fused feature map f. j ; wherein, the use of the text features and the n-layer fused feature map f j Predicting the category of the detected target to obtain category information of the detected target in the image to be detected includes:

[0041] The classification submodule is used to analyze the text features and the n-layer fused feature map f. j The process involves predicting the category of the detected target to obtain category information of the detected target in the image to be detected; wherein, the n-layer fused feature map f is used... j Predicting the location of the target to obtain the location information of the target in the image to be detected includes:

[0042] The localization submodule is used to process the n-layer fused feature map f jThe process is performed to predict the location of the detected target and obtain the location information of the detected target in the image to be detected.

[0043] In one embodiment, the defect detection model further includes a text structuring module; the step of converting the preset prompt text into structured text includes:

[0044] The structured text is obtained by using the text structuring module to perform structuring processing on the preset prompt text.

[0045] In one embodiment, the defect detection model is trained through the following training process:

[0046] Acquire training sample images, as well as preset training prompt text and annotation data corresponding to the training sample images. The annotation data includes the ground truth bounding boxes of training detection targets of interest in the training sample images. The ground truth bounding boxes correspond to the preset training prompt text. The preset training prompt text is a description of the category of the training detection targets of interest in the training sample images.

[0047] The text structuring module converts the preset training prompt text into training structured text;

[0048] The data processing module uses the training structured text as a guide to predict the category and / or location of the target of interest in the training sample image, thereby obtaining a training category probability feature map and / or a training prediction box for the target of interest; wherein, the training category probability feature map is used to characterize the predicted probability of each pixel in the training sample image belonging to the category of the target of interest; and the training prediction box is used to characterize the location information of the target of interest.

[0049] The defect detection model is trained according to the total loss function to obtain the trained defect detection model; wherein, the total loss function is determined by a first loss function and a second loss function; the first loss function is determined based on the training prediction box and the ground truth box corresponding to the training prediction box; the second loss function is determined based on the ground truth class label value of the category of the training detection target and the predicted probability, wherein the ground truth class label value is obtained based on the preset training prompt text.

[0050] In one embodiment, the data processing module uses the training structured text as guidance to predict the category and / or location of the target of interest in the training sample image, thereby obtaining a training category probability feature map and / or training prediction box for the target of interest, including:

[0051] The training text features are obtained by using the text encoding submodule to encode the training structured text.

[0052] The image encoding submodule is used to perform multi-scale feature extraction on the training sample images to obtain an n-layer first training feature map v. i ;

[0053] The feature fusion submodule is used to combine the n-layer first training feature map v i Each feature is fused with the features of the training text to obtain an n-layer training fusion feature map;

[0054] The step of using the training text features and the n-layer training fusion feature map to predict the category of the training detection target to obtain the category information of the training detection target in the training sample image includes:

[0055] The classification submodule uses the training text features and the n-layer training fusion feature map to predict the category of the training detection target, thereby obtaining the category information of the training detection target in the training sample image;

[0056] The step of using the n-layer training fusion feature map to predict the location of the training detection target to obtain the location information of the training detection target in the training sample image includes:

[0057] The localization submodule uses the n-layer training fusion feature map to predict the position of the training detection target to obtain the position information of the training detection target in the training sample image;

[0058] The text encoding submodule is pre-trained, and its weight parameters are not updated during the training process described above.

[0059] In one embodiment, the feature fusion submodule includes an n-layer text attention network; wherein, the feature fusion submodule is used to combine the n-layer first training feature map v i The n-layer training fusion feature map is obtained by fusing the features of the training text with the features of the training text, including:

[0060] An n-layer text attention network is used to process the first training feature map v of the n layers. i Each feature map is fused with the features of the training text to obtain an n-layer training fusion feature map; wherein...

[0061] For the text attention network of the nth layer, the first training feature map of the nth layer is used as the first training input feature map, the training text features are used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain the training output feature map, and the obtained training output feature map is used as the training fusion feature map of the nth layer.

[0062] For the text attention network at layer k, a first convolution operation and upsampling operation are performed on the training fusion feature map at layer k+1 to obtain the corresponding training upsampled feature map, wherein the resolution of the training upsampled feature map is the same as the resolution of the first training feature map at layer k; the first training feature map at layer k and the corresponding training upsampled feature map are added element-wise to obtain the training element-wise added feature map; the training element-wise added feature map is used as the first training input feature map, the training text features are used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain the training output feature map, and the obtained training output feature map is used as the training fusion feature map at layer k, where k = n-1, ..., 1.

[0063] In one embodiment, the image encoding submodule is obtained by performing low-rank fine-tuning on a pre-trained multi-scale feature extraction submodel; wherein, the low-rank fine-tuning process includes:

[0064] For each convolutional layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to that convolutional layer is constructed to form a new multi-scale feature extraction sub-model; this new multi-scale feature extraction sub-model is used as the feature extraction sub-model; wherein, the expression for the weight parameters of each convolutional layer in the new multi-scale feature extraction sub-model is:

[0065] W = W0 + ΔW = W0 + BA;

[0066] Wherein, W0 represents the weight parameters of each convolutional layer in the pre-trained multi-scale feature extraction sub-model, ΔW represents the weight parameters of the low-rank parameter update branch corresponding to the convolutional layer, and B and A represent trainable parameters; wherein, in the above training process, the weight parameters of all convolutional layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

[0067] In one embodiment, the expression for the total loss function of the above-mentioned defect detection model or defect detection network is:

[0068] Loss=δ1L r +δ2L c ;

[0069] Wherein, the L r Let L represent the first loss function. c The second loss function is represented by δ1 and δ2, which are the preset weight coefficients of the first loss function and the second loss function, respectively.

[0070] In one embodiment, the expression for the first loss function is:

[0071]

[0072] Wherein, p r The training prediction box is represented by t. r This represents the ground truth bounding box corresponding to the trained prediction box. p represents r The center point, the Indicates the t r The center point, the Indicates the and stated The Euclidean distance between them, where η represents the bounding distance of p. r and the t r The length of the diagonal of the smallest outer rectangle, where IoU represents the operation of finding the intersection-union ratio.

[0073] In one embodiment, the expression for the second loss function is:

[0074]

[0075] Among them, the The training fusion feature map f represents the training layer j. j The loss of the training category probability feature map of the target being detected, as described in the text, where N represents the total number of layers in the training category probability feature map. This represents the loss of the k-th pixel in the corresponding training category probability feature map, where K represents the total number of pixels in the training category probability feature map;

[0076] Among them, the The expression is:

[0077]

[0078] Wherein, the S t ∑S represents the predicted probability / confidence of the category of the trained detection target in the k-th pixel. u This represents the sum of the predicted probabilities / confidences of the categories of all non-trained detected targets in the k-th pixel.

[0079] According to a second aspect, one embodiment provides a defect detection device. The defect detection device includes:

[0080] A camera is used to capture images of industrial products to be inspected and form an image of the product to be inspected.

[0081] A processor, connected to the camera, is used to process the image to be detected according to the defect detection method described in any embodiment of this application, to obtain the location information and / or category information of the target of interest in the image to be detected.

[0082] In one embodiment, the processor includes: an image acquisition and prompting module, configured to acquire an image to be detected and a corresponding preset prompting text; wherein the preset prompting text is a description of the category of the detected target; and a data processing module, configured to use the structured text as guidance to predict the category and / or location of the target of interest in the image to be detected x, thereby obtaining the category information and / or location information of the target of interest.

[0083] In one embodiment, the processor further includes: a text structuring module for performing structuring processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

[0084] In one embodiment, the data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; the data processing module further includes: a localization submodule and / or a classification submodule; wherein, the text encoding submodule is used to perform text encoding on structured text to obtain text features; the image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v. i Where i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the greater the n ≥ 2; the feature fusion submodule is used to process the n-layer first feature map v i The features are fused with the text features to obtain an n-layer fused feature map f. j The classification submodule is used to utilize the text features and the n-layer fused feature map f j The category of the detected target is predicted to obtain the category information of the detected target in the image to be detected; the localization submodule is used to utilize the n-layer fused feature map f j The location information of the detected target is obtained by predicting the location of the detected target.

[0085] According to a third aspect, one embodiment provides a computer program product including a computer program and / or instructions. When executed by a processor, the computer program and / or instructions implement the defect detection method described in any embodiment of this application.

[0086] According to a fourth aspect, one embodiment provides a computer-readable storage medium, characterized in that it includes a program that can be executed by a processor to implement the defect detection method as described in any embodiment of this application.

[0087] The beneficial effects of this application are:

[0088] The defect detection method of this application includes: acquiring an image to be detected and a corresponding preset prompt text; wherein the preset prompt text is a description of the category of the target of interest in the image to be detected; converting the preset prompt text into structured text; using the structured text as a guide, predicting the category and / or location of the target of interest in the image to be detected, thereby obtaining the category information and / or location information of the target of interest; that is, this application obtains the preset prompt text corresponding to the image to be detected, then performs text encoding on the preset prompt text or structured text to obtain text features, uses the text features to guide and indicate the target in the image to be detected (such as the category and / or location of the target of interest mentioned above), and performs alignment / matching between the image to be detected and the text features, thereby achieving accurate detection of defects of unknown categories. Attached Figure Description

[0089] Figure 1 This is a flowchart illustrating a defect detection method according to one embodiment;

[0090] Figure 2 This is a schematic diagram of the processing flow of a defect detection model or defect detection network according to one embodiment;

[0091] Figure 3 This is a schematic diagram illustrating the process of obtaining category information and / or location information of a target of interest in one embodiment.

[0092] Figure 4 This is a schematic diagram illustrating the data processing procedure of a feature fusion submodule in one embodiment.

[0093] Figure 5 This is a schematic diagram illustrating the process of inputting a first input feature map and a second input feature map into a text attention network to obtain an output feature map, according to one embodiment.

[0094] Figure 6 This is a schematic diagram illustrating the training process of a defect detection model according to one embodiment.

[0095] Figure 7 A flowchart illustrating the training of a defect detection model for one embodiment;

[0096] Figure 8 This is a schematic diagram illustrating how a defect detection model predicts an image to be detected, according to one embodiment. Detailed Implementation

[0097] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0098] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0099] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. Unless otherwise specified, the terms "connection" and "linkage" used in this application include both direct and indirect connections.

[0100] In the field of machine vision inspection, commonly used methods for detecting unknown categories of defects in images typically employ the pre-defined vector query method. This method involves first inputting image features, corresponding text features, and a pre-defined query vector into a multimodal decoder to obtain a predicted query vector. Then, based on the predicted query vector and the aforementioned image and text features, a bounding box and target text are determined. Finally, the target text corresponding to the predicted target vector is used as the predicted category to detect unknown categories of defects. However, the inventors have discovered that: First, the pre-defined vector query method lacks template-based adaptation to the descriptive text of the detection target, resulting in highly arbitrary input content, weak scene adaptability, and poor generalization performance. Second, obtaining the pre-defined query vector requires a pre-defined threshold, which necessitates manual parameter tuning, is time-consuming and labor-intensive, and typically requires highly specialized personnel, thus raising the user threshold and increasing the production burden on enterprises. Third, the network structure of the multimodal decoder that decodes image, text features, and the pre-defined query vector is usually complex, with a large number of network parameters and poor real-time inference performance. These three factors combined lead to a decline in detection performance.

[0101] To address the problems existing in the prior art, this application proposes a defect detection method. One objective of this method is to train a defect detection model or network using only data of known categories for training, thereby solving the problem of detecting defects of known and / or unknown categories in images and improving the detection performance. The defect detection network can be a network capable of performing essentially the same functions as the defect detection model.

[0102] The technical concept of this defect detection method is as follows: based on the text information added by the user, such as the preset prompt text below, the defect detection model or defect detection network is trained on target samples of known categories in an end-to-end manner, thereby achieving accurate detection of defects of known and / or unknown categories in the image.

[0103] This defect detection method, guided by user-added text information, achieves accurate detection of defects, especially those of unknown categories, based on the following principle: Through a pre-trained text encoder network (as described in the text encoding submodule below) and a visual network (as described in the image encoding submodule below), the method learns the matching relationship between target regions and text information (as described in the preset prompt text or structured text below) in the training data (as described in the training sample images below). This text information guides and / or enhances the target information in image features, aligning image and text features to achieve open-set target detection, i.e., accurate detection of defects of unknown categories. Essentially, it involves a series of computational processes (detailed below) to detect the similarity or matching between text features and visual features, such as fused feature maps, ultimately achieving open-set detection, i.e., accurate detection of unknown defects.

[0104] The technical solution of this application will be described in detail below with reference to the embodiments.

[0105] Please refer to Figure 1 This application provides a defect detection method. The defect detection method includes:

[0106] Step S100: Obtain the image to be detected and the preset prompt text;

[0107] Step S200: Convert the preset prompt text into structured text;

[0108] Step S300: Predict the target of interest in the image to be detected.

[0109] Specifically, in step S100, the preset prompt text is a description of the category of the target of interest in the image to be detected.

[0110] Specifically, in step S300, structured text is used as a guide to predict the category and / or location of the target of interest in the image to be detected x, thereby obtaining the category information and / or location information of the target of interest.

[0111] It should be noted that in step S100, the categories of some or all of the detected targets in the image to be detected may be unknown. That is to say, the category information contained in the detected targets may be known and / or unknown.

[0112] In some embodiments, in step S100 above, those skilled in the art can select the image to be detected according to actual needs. For example, the image to be detected may be an image obtained by photographing industrial products or other objects. Users can determine the "target of interest in the image to be detected" according to actual needs.

[0113] In some embodiments, in step S200, please refer to... Figure 2 Convert the preset prompt text into structured text t, including:

[0114] The preset prompt text is processed into structured text t; the structured text t has a fixed format.

[0115] It should be noted that the purpose of "structuring the preset prompt text to obtain structured text t" is to constrain and adapt various preset prompt texts through templates, so that the structured text t can facilitate the identification of effective features of defects in the image to be detected by the defect detection model or defect detection network, thereby enhancing the adaptability of the defect detection model or defect detection network to different scenarios, and ultimately improving the detection performance of this defect detection method. The "constraint adaptation" refers to converting the user's language input, such as the preset prompt text, into a fixed-format template, thereby standardizing and normalizing the user's language input. This facilitates the text encoder in the defect detection model to accurately extract the corresponding effective features from the "structured text," thus enabling the defect detection model or defect detection network to better understand the user's intention expressed by the preset prompt text. For example, in some embodiments, the target object of interest to the user in the image to be detected is "dog". Different users will have different language prompts for it, which is quite arbitrary. For example, descriptions such as "dog in the image to be detected", "detect target dog", "dog located in the upper left position of the image to be detected" are diverse and arbitrary. By performing the above-mentioned "constraint adaptation" through template, the noun of the detected target is extracted as "dog", and a preset template formula, such as "detected object is {xxx}", can be used to template the preset prompt text into "detected object is dog", that is, the corresponding structured text.

[0116] In some embodiments, please refer to Figure 2 and Figure 3 In step S300, guided by structured text, the category and / or location of the target of interest in the image x to be detected are predicted, thereby obtaining the category information and / or location information of the target of interest, including:

[0117] Step S310: Text encoding is performed on the structured text t to obtain text features e; multi-scale feature extraction is performed on the image to be detected x to obtain the first feature map v of the nth layer;

[0118] Step S320: Transfer the first feature map v of layer n i Each feature is fused with the text feature e to obtain an n-layer fused feature map f;

[0119] Step S330: Predict the category of the detected target in the image by using text features e and the fused feature map f from n layers. pj ; and / or, using n-layer fused feature maps f to predict the category of the detected target to obtain the location information box of the detected target in the image to be detected.

[0120] In some embodiments, in step S310, those skilled in the art can determine the specific expression of the above-mentioned text features according to actual needs. For example, the dimensional shape of the above-mentioned text feature e can be represented as [n, m], where n is the number of categories of the target of interest to the user in the image to be detected, and m is the length of the preset feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network during the training phase can be different from n in the defect detection model or defect detection network during the inference phase.

[0121] It should be noted that the above text features are essentially feature vectors. Those skilled in the art can determine the specific form of the above text features according to their actual needs.

[0122] In some embodiments, in step S310, those skilled in the art can determine the number of layers of the first feature map according to actual needs. For example, the number of the first feature map can be 2 or 4 layers, etc., and those skilled in the art generally determine the number of layers of the first feature map to be 3 layers. Each layer of the first feature map v i Each represents a scale. Here, i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the greater n ≥ 2.

[0123] In some embodiments, step S320, fusing the n-layer first feature map v with text features to obtain an n-layer fused feature map f, includes:

[0124] An n-layer text attention network is used to fuse the first feature map v of the n-layer network with text features to obtain an n-layer fused feature map f, where,

[0125] For the text attention network of the nth layer, the first feature map of the nth layer is used as the first input feature map, and the text features are used as the second input feature map. The first input feature map and the second input feature map are input into the text attention network to obtain the output feature map, and the obtained output feature map is used as the fusion feature map f of the nth layer.

[0126] For the text attention network at layer k, the first convolution operation and upsampling operation are performed on the fused feature map at layer (k+1) to obtain the corresponding upsampled feature map, where the resolution of the upsampled feature map is the same as the resolution of the first feature map at layer k. The first feature map at layer k and the corresponding upsampled feature map are then added element-wise to obtain the element-wise added feature map. The element-wise added feature map is used as the first input feature map, and the text features are used as the second input feature map. The first input feature map and the second input feature map are input into the text attention network to obtain the output feature map. The obtained output feature map is used as the fused feature map f at layer k, where k = n-1, ..., 1.

[0127] It should be noted that the text attention network is used to guide and enhance the location information of the detected target in the first feature map v.

[0128] In some embodiments, please refer to Figure 4 At this point, n=3. A 3-layer text attention network Attn is used to fuse the first feature maps of the three layers (e.g., v1, v2, and v3) with the text feature e to obtain a 3-layer fused feature map f (Note: f includes f1, v2, and v3). 1、 (f2 and f3). For the third-layer text attention network Attn, the first feature map v3 of the third layer is used as the first input feature map, and the text feature e is used as the second input feature map. The first and second input feature maps are input into the third-layer text attention network Attn to obtain the output feature map, and the obtained output feature map is used as the third-layer fusion feature map f3. For the second-layer text attention network Attn, the third-layer fusion feature map f3 is subjected to the first convolution operation Conv and the upsampling operation Up to obtain the corresponding upsampled feature map. The resolution of the upsampled feature map is the same as that of the second-layer first feature map v2. The second-layer first feature map v2 and the corresponding upsampled feature map are added element-wise by the Add operation to obtain the element-wise added feature map. The element-wise added feature map is used as the first input feature map, and the text feature e is used as the second input feature map. The first and second input feature maps are input into the second-layer text attention network Attn to obtain the output feature map, and the obtained output feature map is used as the second-layer fusion feature map f2. Since the processes for obtaining the first-layer fused feature map f1 and the second-layer fused feature map f2 are basically the same, they will not be described again here.

[0129] It should be noted that the "first convolution operation," "upsampling operation," and "element-by-element addition operation" in step S320 above are all conventional techniques in the art. For example, the "first convolution operation" can use a convolutional layer with a 1×1 kernel. Those skilled in the art can determine the specific process of each of the above operations themselves, and it will not be elaborated here. The purpose of the first convolution operation is to make the output channel of the feature map obtained by the first convolution operation consistent with the first feature map of the second layer, such as v2. The purpose of the upsampling operation Up is to make the resolution of the "corresponding upsampled feature map" consistent with the resolution of the first feature map of the second layer, v2, so that the "corresponding upsampled feature map" can be fused with the first feature map of the second layer, v2. The above fusion is achieved through the element-by-element addition operation Add. The element-by-element addition operation Add adds the elements at corresponding positions of the feature maps.

[0130] It should be noted that the role of the text attention network Attn is to realize the interaction and fusion of text features e with multi-scale features (such as v1, v2 and v3), which is essentially to complete the multimodal feature fusion of text information and visual features.

[0131] In some embodiments, please refer to Figure 5 The above-mentioned input of the first input feature map and the second input feature map into the text attention network to obtain the output feature map includes:

[0132] Step 10: Calculate the first text-region similarity matrix s1 using the first input feature map d and the second input feature map e of the input text attention network Attn through the following expression:

[0133]

[0134] Wherein, the first text-region similarity matrix is ​​used to characterize the similarity between each pixel in the first input feature map d and each category in the second input feature map e, Conv(d) represents the second convolutional feature map obtained by performing a second convolution operation on the first input feature map d, the second input feature map e is the text feature e, Norm(e) represents the first normalized feature map obtained by performing a first normalization process on the text feature e, and μ and δ are the first learnable scaling factor and the first learnable translation factor, respectively. This represents the matrix multiplication operation;

[0135] Step 20: Maximize the similarity matrix s1 between the first text and the region to obtain the most similar matrix between the first text and the region;

[0136] Step 30: Perform a first non-linear activation process on the first text-region most similarity matrix to obtain a text-region attention map;

[0137] Step 40: Perform element-wise multiplication on the text-region attention map and the first input feature map d of the input text attention network to obtain the attention enhancement feature map;

[0138] Step 50: Perform a third convolution operation on the attention-enhanced feature map to obtain the corresponding fused feature map f. j The corresponding fused feature map f j As the output feature map of the text attention network.

[0139] It should be noted that in step 10 above, μ and δ are not preset parameters, but learnable parameters. Essentially, μ and δ are the learnable first scaling factor and the learnable first translation factor, respectively. That is to say, μ and δ can be adaptively adjusted / updated during the training process of the defect detection model or the defect detection network, and their function is to allow the defect detection model or the defect detection network to better adapt to the training task and data.

[0140] It should be noted that the process of calculating the "first text-region similarity matrix s1" in step 10 above is an intermediate calculation process for obtaining the text-region attention map. The purpose of performing a second convolution operation on the first input feature map d to obtain the second convolutional feature map is to ensure that the number of channels in the second convolutional feature map is equal to the length of the second input feature map e. Essentially, the matrix multiplication operation in step 10 above... What is calculated is the pixel-by-pixel multi-class similarity between the text features of each category and the fused features (such as Conv(d) above).

[0141] It should be noted that the purpose of step 20 above is to obtain the first text-region most similarity matrix by taking the maximum value of the first text-region similarity matrix s1 along the channel dimension. This first text-region most similarity matrix represents the maximum similarity between each pixel in the first input feature map d and each category in the text feature e.

[0142] It should be noted that in step 30 above, a first non-linear activation process is performed on the first text and region most similarity matrix to obtain a text-region attention map. The function of the text-region attention map is to integrate information from the language prompt, i.e., the preset prompt text such as "prompt", with image features (such as the n-layer first feature map v). iBy combining these elements, the importance of the foreground region can be obtained, thus indicating the spatial importance of image features to the target. For example, if the preset prompt text is "dog," the text-region attention map indicates the presence of the target of interest, "dog," in the spatial representation of image features, i.e., the importance of the pixel position in the foreground. Essentially, the text-region attention map is a pixel-by-pixel probability map. The numerical value of each element in the text-region attention map represents the probability that the pixel corresponding to that element in the first input feature map d belongs to a certain category.

[0143] It should be noted that in step 40 above, the “attention-enhanced feature map” represents the multimodal fusion information of text features and visual features. Its function is to align text features with image features such as the multi-scale features mentioned above, guide and enhance the spatial importance of image features through text features, and thus help the defect detection model or defect detection network to enhance its perception of the location information of the area where the above-mentioned detection target is located.

[0144] It should be noted that step 50 above, which involves performing a third convolution operation on the attention-enhancing feature map, aims to ensure that its number of channels is equal to the length of the text features, thereby obtaining the corresponding fused feature map f. j .

[0145] It should be noted that the "second convolution operation," "first normalization processing," "maximum value operation," "first nonlinear activation processing," "element-by-element multiplication operation," and "third convolution operation" in steps 10 to 50 above are all conventional techniques in this field. For example, the "second convolution operation" can use a 1×1 convolution, where a 1×1 convolution is a special type of convolution operation with a kernel size of 1×1; the "first nonlinear activation processing" can use the Sigmoid activation function for nonlinear activation processing, etc. Those skilled in the art can determine the specific process of the above operations themselves, and it will not be elaborated here.

[0146] It should be noted that "applying text attention to image features based on text guidance, injecting linguistic information into image features, and realizing the interaction and fusion of text and visual information" refers to fusing text features with the aforementioned multi-scale features to obtain the desired result. In other words, the entire process of fusing text features with the aforementioned multi-scale features is used to "realize the interaction and fusion of text and visual information."

[0147] In some embodiments, in step S330, the n-layer fused feature map f is used. j The location information of the target in the image to be detected is obtained by predicting the location of the target, including: fusing n layers of feature maps f j The fourth convolution operation is performed separately to output and fuse the feature map f. jThe corresponding prediction bounding box; where the prediction bounding box is used to represent the location information of the detected target.

[0148] It should be noted that the "fourth convolution operation" in step S330 above is a conventional technique in this field, so its specific process will not be elaborated here. The purpose of this "fourth convolution operation" is to output and fuse the feature map f. j The corresponding prediction box. The specific process of “performing the fourth convolution operation on each of the n-layer fused feature maps and outputting the prediction box corresponding to the fused feature map” is also a conventional technique in this field. For example, the dimension and shape of any prediction box can be [b, 4, h, w], where b represents the batch number, 4 represents the position information of the prediction box, and h and w are the height and width of the fused feature map of the current detection layer, respectively.

[0149] In some embodiments, in step S330, text features and n-layer fused feature maps f are utilized. j The category of the detected target is predicted to obtain the fused feature map f. j The category information of the detected target includes:

[0150] Step S331: Utilize text feature e and fuse feature map f j The following expression is used to calculate and fuse the feature map f. j The corresponding second text-region similarity matrix:

[0151]

[0152] The second text-region similarity matrix is ​​used to represent the fused feature map f of the j-th layer. j The similarity of each pixel in the text feature e to each category is given by Norm(e). Norm(e) represents the feature map obtained after the second normalization process on the text feature e. α and β are the learnable second scaling factor and learnable second translation factor, respectively. This represents the matrix multiplication operation;

[0153] Step S332: Calculate the fusion feature map f with the j-th layer using the second text-region similarity matrix and the following expression. j Corresponding category probability feature map:

[0154] p j = sigmoid(s2);

[0155] Wherein, sigmoid represents the second nonlinear activation process, and is fused with the feature map f of the j-th layer. j The corresponding category probability feature map p j The fusion feature map f used to characterize the j-th layer jThe probability that each pixel belongs to the above-mentioned categories of detected targets.

[0156] It should be noted that the "fourth convolution operation," "second normalization processing," and "second nonlinear activation processing" in steps S331 and S332 are all conventional techniques in the art. For example, the "second nonlinear activation processing" can use the Sigmoid activation function for nonlinear activation processing. Those skilled in the art can determine the specific process of the above operations themselves, and it will not be elaborated here.

[0157] It should be noted that α and β in step S331 above are not preset parameters, but learnable parameters. Essentially, α and β are learnable second scaling factor and learnable second translation factor, respectively, which can adaptively adjust / update during the training process of the defect detection model or the defect detection network. Their function is to allow the defect detection model or the defect detection network to better adapt to the training task and data.

[0158] In some embodiments, the above-described defect detection method can also be implemented based on a defect detection network. The defect detection network can be a network capable of performing essentially the same functions as a defect detection model.

[0159] In some embodiments, please refer to Figure 2 The above defect detection method is based on a defect detection model G, which includes a data processing module. Step S300 involves using structured text t as a guide to predict the category and / or location of the target of interest in the image x to be detected, thereby obtaining the category information and / or location information of the target of interest, including:

[0160] The data processing module uses structured text t as a guide to predict the category and / or location of the target of interest in the image x to be detected, thereby obtaining the category information p of the target of interest. j And / or location information box.

[0161] In some embodiments, please refer to Figure 2 The data processing module includes a text encoding submodule T, an image encoding submodule V, and a feature fusion submodule F. The data processing module also includes a localization submodule L and / or a classification submodule C; wherein, in step S310, text encoding is performed on the structured text t to obtain text features e, including:

[0162] Text features e are obtained by text encoding the structured text t using the text encoding submodule T;

[0163] Step S310: Performing multi-scale feature extraction on the image to be detected x to obtain an n-layer first feature map v includes: using the image coding submodule V to perform multi-scale feature extraction on the image to be detected x to obtain an n-layer first feature map v;

[0164] Step S320: fusing the first feature map v of the nth layer with the text feature e to obtain the fused feature map f of the nth layer includes: using the feature fusion submodule F to fuse the first feature map v of the nth layer with the text feature e to obtain the fused feature map f of the nth layer;

[0165] Step S330: Using textual features e and n-layer fused feature map f to predict the category of the detected target to obtain the category information of the detected target in the image to be detected, including:

[0166] The classification submodule C processes the text features e and the n-layer fused feature map f to predict the category of the detected target, thereby obtaining the category information p of the detected target in the image to be detected. j ;

[0167] Step S330: Using n-layer fused feature maps f to predict the category of the detected target to obtain the location information box of the detected target in the image to be detected, including:

[0168] The localization submodule L processes the n-layer fused feature map f to predict the category of the detected target and obtain the location information box of the detected target in the image to be detected.

[0169] In some embodiments, because the language prompts for the targets of interest in the image to be detected, such as the preset prompt text mentioned above, are highly arbitrary in terms of sentence structure and vocabulary, and thus uncontrollable, please refer to... Figure 2 The defect detection model G also includes a text structuring module M. In step S200, the preset prompt text, such as `prompt`, is converted into structured text `t`, including:

[0170] The preset prompt text t is processed using the text structuring module M to obtain the structured text t.

[0171] In some embodiments, since the language prompts for the target of interest in the image to be detected, such as the preset prompt text mentioned above, are highly arbitrary in terms of sentence structure and vocabulary, and thus uncontrollable, the function of the text structuring module is to constrain and adapt the target prompts input by the user. For example, a text template can be designed to adapt to the user's input. For example, firstly, a template formula is determined, where the "template formula" can be defined by the user based on the data and task scenario, such as "an image of {target object}"; then, the target noun of the language prompt for the target of interest in the image to be detected is extracted, such as "dog", and "target object" in the template formula is replaced with "dog"; finally, a templated and structured text description t is output.

[0172] It should be noted that the text structuring module can directly adopt existing technologies in this field. Those skilled in the art can determine the specific processing procedure / form of the text structuring module according to actual needs, as long as the language prompt input, such as the preset prompt text mentioned above, is standardized and normalized.

[0173] It should be noted that during the training of the defect detection model, users do not need to manually add preset training prompt text for unknown categories of defects, but only need to add corresponding preset training prompt text for known categories of defects; when using the defect detection model or the defect detection model for inference, users add corresponding preset prompt text for the categories of detection targets of interest, including known categories and / or unknown categories.

[0174] In some embodiments, in order to further enhance the diversity of text input and improve the robustness and generalization performance of the above-mentioned defect detection model or defect detection network, the text structuring module can adopt a variety of different types of templates to adapt to different detection scenarios.

[0175] In some embodiments, if the language prompts input by the user regarding the target of interest in the image to be detected, such as the preset prompt text mentioned above, have a uniform and standardized format in terms of sentence structure and vocabulary, then the defect detection model may not include the text structuring module M. That is, the preset prompt text can be directly used as the structured text.

[0176] In some embodiments, the main function of the text submodule T is to encode structured text descriptions such as the structured text t to obtain text features.

[0177] It should be noted that the text submodule T described above can employ any network capable of encoding the structured text. For example, the text submodule T can use a pre-trained CLIP model's text encoder. CLIP (Contrastive Language-Image Pre-training) is a deep learning model representing the concept of a language-image contrastive pre-training model. The core of the CLIP model lies in mapping both images and text to the same vector space, enabling direct contrastive learning between the two modes. The key innovation of this model is the unified vector space, allowing images and text to directly compute similarity in the vector space without additional intermediate representations. CLIP uses contrastive learning for pre-training, training to more tightly bind related images and text while separating unrelated images in the feature space. This unsupervised learning method allows CLIP to learn from text and image data on the internet, performing exceptionally well in various domain tasks. Furthermore, since the CLIP model's text encoder is a well-known and commonly used network structure in this field, its specific structure will not be elaborated upon here.

[0178] It should be noted that when the text submodule T uses a pre-trained CLIP model text encoder, the weight parameters of the text submodule T during the training or inference phase are not updated. The input of the text submodule is the text description t, and the output of the text submodule is the text feature e. For example, the dimensional shape of the text feature e can be represented as [n, m], where n is the number of categories of the target of interest to the user in the image to be detected, and m is the preset length of the feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network during the training phase can be different from n in the defect detection model or defect detection network during the inference phase.

[0179] In some embodiments, the image encoding submodule V described above performs multi-scale feature extraction on the input image to be detected x to obtain an n-layer first feature map v. For example, the number of the first feature maps v can be 2 or 4 layers, etc., and those skilled in the art generally determine the number of layers of the first feature maps to be 3 layers. Each layer of the first feature map v i Each represents a scale. Here, i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. iThe smaller the resolution, the greater n ≥ 2. Each layer's first feature map represents a specific scale. Specifically, the n-layer first feature map comes from the multi-scale feature maps output by the n consecutive feature extraction layers near the input of the neural network. For example, when n = 3, the 3-layer first feature map comes from the multi-scale feature maps output by the three consecutive feature extraction layers near the input of the neural network. For example, the image coding submodule V mentioned above can directly utilize the backbone of an existing neural network model for multi-scale feature extraction. "Utilizing the backbone for multi-scale feature extraction" is a conventional technique in this field and will not be elaborated upon here.

[0180] It should be noted that the image coding submodule V described above can use the backbone network of any existing general vision model as its network structure.

[0181] In some embodiments, preferably, based on the principle of lightweight design, the image encoding submodule V can employ a pre-trained ResNet-18 backbone network. ResNet-18 is an 18-layer convolutional neural network. This is because the ResNet-18 backbone network has a compact structure, strong feature extraction capabilities, and fast inference speed. Throughout the training and inference phases of the aforementioned defect detection model or network, the weight parameters of the image encoding submodule V remain fixed and are not updated.

[0182] In some embodiments, please refer to Figure 4 The feature fusion submodule F is responsible for connecting text features e and image features. Based on text guidance, it applies text attention to the image features (such as the first feature map v at layer n mentioned above), injecting linguistic information from text features e into the image features, thus achieving interaction and fusion of text and visual information. The image feature v input to the feature fusion submodule F is a multi-scale feature, such as the first feature map v at layer n mentioned above. i The aforementioned image features v can include v1, v2, and v3, etc. The overall network of the feature fusion submodule F can have a feature pyramid structure. The feature fusion submodule F can combine the text features e with the first feature map v of each layer. i The text information in text feature e is fused to the first feature map v. i Precise guidance. The final output of the feature fusion submodule F is the n-layer fused feature map f. j The number of channels is the same, but the n-layer fused feature map f j The resolutions are different.

[0183] It should be noted that the network structure and processing flow of each text attention network Attn contained in the feature fusion submodule F are consistent.

[0184] When a defect detection model or network is used for inference, it can directly output target location information and category prediction information in an end-to-end manner, enabling accurate identification of known and / or unknown category defects.

[0185] This application designs corresponding loss functions for training the localization submodule L and classification submodule C of the entire defect detection model or defect detection network. The design of the feature fusion submodule F, classification submodule C, and loss function of the defect detection model are key to the overall structure of the defect detection model or defect detection network.

[0186] In some embodiments, please refer to Figure 6 and Figure 7 The defect detection model G described above is trained using the following process:

[0187] Step 100: Obtain the training sample image x, the preset training prompt text corresponding to the training sample image x, and the annotation data. The annotation data includes the ground truth bounding boxes of the training detection targets of interest in the training sample image; wherein, the ground truth bounding boxes correspond to the preset training prompt text; wherein, the preset training prompt text is a description of the category of the training detection targets of interest in the training sample image x.

[0188] Step 200: The text structuring module M converts the preset training prompt text into training structured text t;

[0189] Step 300: The data processing module uses the training structured text t as a guide to predict the category and / or location of the target of interest in the training sample image x, thereby obtaining the training category probability feature map p of the target of interest. cls and / or training prediction box p r Among them, the training category probability feature map p cls The predicted probability used to characterize the category of each pixel in the training sample image x belonging to the training detection target; training prediction box p r Used to characterize the location information of the training detection target;

[0190] Step 400: Train the defect detection model based on the total loss function to obtain a trained defect detection model; wherein, the total loss function is composed of the first loss function L r Second loss function L c Determine; First loss function L r Based on the training prediction box pr and training prediction box p r The corresponding real bounding box t r Determine; Second loss function L c Based on the true class label value t of the target category detected during training. cls As the predicted probabilities are determined above, the true class label value is obtained based on the preset training prompt text.

[0191] It should be noted that since the preset training prompt text is a description of the category of the training detection target of interest in the training sample image x, the "preset training prompt text" can essentially directly represent the "true category label value". That is to say, in some embodiments, the "preset training prompt text" can be directly used as the "true category label value".

[0192] In some embodiments, in step 100, the training samples used by the defect detection model G may include a training sample image x and a preset training prompt text corresponding to the training sample image. The training sample image x and the preset training prompt text are usually paired. For example, image samples with known defect categories can be collected as the training sample image x. The preset training prompt text is a verbal prompt for the training detection target in the training sample image x. Users can manually set the preset training prompt text corresponding to the training sample image. The training detection target of interest can also be determined by the user according to actual needs, as long as the true bounding box of the training detection target corresponds to the preset training prompt text.

[0193] It should be noted that in step 100, the text corresponding to each training detection target in the preset training prompt text can essentially be regarded as the true category label of that training detection target.

[0194] It should be noted that in step 100, the user does not need to manually add preset training prompt text for unknown category defects; only preset training prompt text needs to be added for known category defects. However, when using the defect detection model or performing inference using the defect detection model, the user adds corresponding preset prompt text for the category of the detection target of interest, which includes known categories and / or unknown categories.

[0195] In some embodiments, in step 200, the text structuring module M is used to achieve a templated and structured text description of the preset training prompt text, that is, to generate the training structured text t. The preset training prompt text can be in the form of nouns, phrases, or sentences. The training structured text t has a standard or uniform structural form. For example, it is uniform in its phrase or sentence structure, that is, it is a standard structural form.

[0196] In some embodiments, in step 300, the data processing module, guided by training structured text, predicts the category and / or location of the target of interest in the training sample image x, thereby obtaining a training category probability feature map and / or training prediction box for the target of interest, including:

[0197] The training text features e are obtained by text encoding the training structured text t using the text encoding submodule T;

[0198] The first training feature map v at layer n is obtained by using the image coding submodule V to perform multi-scale feature extraction on the training sample image x. i ;

[0199] The feature fusion submodule F is used to combine the first training feature map v of layer n. i The training text features e are fused to obtain an n-layer training fused feature map f. j ;

[0200] Step 330: Using the training text features e and n layers to train the fused feature map f j The category information of the training target in the training sample image is obtained by predicting the category of the training target, including:

[0201] The classification submodule C uses training text features e and n layers to train a fused feature map f. j The category information of the training target in the training sample image is obtained by predicting the category of the training target.

[0202] Step 330: Train the fused feature map f using n layers. j The location information of the training target in the training sample image is obtained by predicting the location of the training target, including:

[0203] The localization submodule L uses n layers to train and fuse the feature map f j The location information of the training detection target in the training sample image is obtained by predicting the location of the training detection target; wherein, the text encoding submodule is obtained through pre-training, and the weight parameters of the text encoding submodule T are not updated in the above training process.

[0204] In some embodiments, those skilled in the art can determine the specific representation of the above-mentioned training text features according to actual needs. For example, the dimensional shape of the above-mentioned training text feature e can be represented as [n, m], where n is the number of categories of training detection targets of interest to the user in the training sample image, and m is the preset length of the feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network during the training phase can be different from n in the defect detection model or defect detection network during the inference phase.

[0205] It should be noted that the above-mentioned training text features are essentially feature vectors.

[0206] In some embodiments, those skilled in the art can determine the number of layers of the first training feature map according to actual needs. For example, the number of the first training feature map can be 2 layers or 4 layers, etc., and those skilled in the art generally determine the number of layers of the first training feature map to be 3 layers. Each layer of the first training feature map v i Each of these represents a scale. Here, i = 1, ..., n, and the larger the value of i, the larger the corresponding first training feature map v. i The smaller the resolution, the greater n ≥ 2.

[0207] In some embodiments, in step 330, the classification submodule C can first utilize the training text features e and the fused feature maps f from each layer of training. j The categories of the training detection targets are predicted to obtain the category information of the training detection targets in each layer of the training fusion feature map. Then, the category information of the training detection targets in each layer of the training fusion feature map is post-processed to obtain the category information of the training detection targets in the training sample image. The above post-processing may include: score thresholding, etc. Since the specific process of the above post-processing is a conventional technique in this field, it will not be described in detail here.

[0208] It should be noted that the localization submodule L utilizes n layers to train the fused feature map f j The location of the training target in the training sample image is obtained by predicting its position. Then, the location information of the training target in the training sample image is obtained by performing corresponding post-processing on the location information of the training target in each layer of the fused feature map. The post-processing may include: score thresholding, non-maximum suppression, etc. The above process of "obtaining the location information of the training target in the training sample image" is similar to the previous process of "using n-layer fused feature map f". j The process of "predicting the category of the detected target to obtain the category information of the detected target in the sample image" is consistent with the process of "training prediction boxes" (i.e., "training prediction boxes" correspond one-to-one with the above categories, and if there are no training prediction boxes, the default is the background), so it will not be described in detail here.

[0209] In some embodiments, the feature fusion submodule F includes an n-layer text attention network Attn; wherein, the feature fusion submodule F is used to combine the n-layer first training feature maps v i The training text features e are fused to obtain an n-layer training fused feature map f. j ,include:

[0210] Step 321: Use an n-layer text attention network to process the first training feature map v of the n-layer network. i Each feature is fused with the training text feature e to obtain an n-layer training fused feature map;

[0211] Step 322: For the text attention network of the nth layer, take the first training feature map of the nth layer as the first training input feature map, take the training text features as the second training input feature map, input the first training input feature map and the second training input feature map into the text attention network to obtain the training output feature map, and take the obtained training output feature map as the training fusion feature map of the nth layer.

[0212] Step 323: For the text attention network at layer k, perform a first convolution operation and upsampling operation on the training fusion feature map at layer (k+1) to obtain the corresponding training upsampled feature map, wherein the resolution of the training upsampled feature map is the same as the resolution of the first training feature map at layer k; perform an element-wise addition operation on the first training feature map at layer k and the corresponding training upsampled feature map to obtain the training element-wise addition feature map; use the training element-wise addition feature map as the first training input feature map, use the training text features as the second training input feature map, input the first training input feature map and the second training input feature map into the text attention network to obtain the training output feature map, and use the obtained training output feature map as the training fusion feature map at layer k, where k = n-1, ..., 1.

[0213] The aforementioned text attention network Attn is used to guide and enhance the first training feature map v. i The training program detects the location information of the target. For example, please refer to... Figure 4 When n=3, a 3-layer text attention network Attn can be used to train the first feature map v of the 3 layers. i The training text feature e is fused with the training text feature e to obtain a 3-layer training fused feature map, as shown in f. 1、f2 and f3. Specifically, for the third-layer text attention network Attn, the first training feature map of the third layer, such as v3, is used as the first training input feature map, and the training text feature e is used as the second training input feature map. The first and second training input feature maps are then input into the third-layer text attention network Attn to obtain the training output feature map, which is then used as the third-layer training fusion feature map f3. For the second-layer text attention network Attn, the third-layer training fusion feature map f3 undergoes a first convolution operation and an upsampling operation to obtain the corresponding training upsampled feature map. The resolution of the training upsampled feature map is the same as that of the first training feature map of the second layer, such as v2. The first training feature map of the second layer, such as v2, is added element-wise with the corresponding training upsampled feature map to obtain a training element-wise added feature map. This training element-wise added feature map is used as the first training input feature map, and the training text feature e is used as the second training input feature map. The first and second training input feature maps are then input into the text attention network Attn of the second layer to obtain the training output feature map. The resulting training output feature map is used as the second-layer training fusion feature map f2. Since the process of obtaining the first-layer training fusion feature map f1 and the second-layer training fusion feature map f2 is basically the same, it will not be described in detail here. The aforementioned "first convolution operation," "upsampling operation," and "element-wise addition operation" are all conventional techniques in this field, and those skilled in the art can determine the specific process of these operations themselves; they will not be described in detail here. The purpose of the first convolution operation is to make the output channel of the training upsampled feature map obtained by the first convolution operation consistent with the second-layer first training feature map, such as v2. The purpose of the upsampling operation Up is to make the resolution of the "corresponding training upsampled feature map" consistent with that of the first training feature map of the second layer, such as v2, so as to achieve the fusion of the "corresponding training upsampled feature map" and the first training feature map of the second layer, such as v2. This fusion is achieved through element-wise addition. Element-wise addition adds the elements at corresponding positions in the feature maps. The role of the text attention network Attn is to integrate the training text features e with multi-scale features such as the first training feature map v2 of layer n. i The interaction and fusion of text information and visual features is essentially the fusion of multimodal features.

[0214] In some embodiments, the above-described method of inputting the first training input feature map and the second training input feature map into the text attention network to obtain the training output feature map includes:

[0215] The first training input feature map d and the second training input feature map e of the input text attention network are used to calculate the similarity matrix between the first text and the region using the following expression:

[0216]

[0217] Wherein, the first text-region similarity matrix is ​​used to characterize the similarity between each pixel in the first training input feature map d and each category in the second training input feature map e, Conv(d) represents the second training convolutional feature map obtained by performing a second convolution operation on the first training input feature map d, the second training input feature map e is the training text feature e, Norm(e) represents the first normalized training feature map obtained by performing a first normalization process on the training text feature e, and μ and δ are the first learnable scaling factor and the first learnable translation factor, respectively. This represents the matrix multiplication operation;

[0218] The maximum value operation is performed on the similarity matrix s1 between the first text and the region to obtain the most similar matrix between the first text and the region;

[0219] The first non-linear activation process is applied to the first text and region most similarity matrix to obtain the training text-region attention map;

[0220] The training attention enhancement feature map is obtained by performing element-wise multiplication on the training text-region attention map and the first training feature map of the input text attention network.

[0221] The training attention-enhancing feature map is subjected to a third convolution operation to obtain the corresponding training fusion feature map f. j The corresponding training fusion feature map f j The feature map is used as the training output of the text attention network.

[0222] In some embodiments, please refer to Figure 6 In step S330, the fused feature map f is trained using n layers. j The location information of the training detection targets in the training image is obtained by predicting the category of the training detection targets, including:

[0223] The fourth convolution operation is performed on each of the n-layer training fusion feature map f to output the prediction box pbox corresponding to the training fusion feature map f; where the prediction box pbox is used to represent the location information of the training detection target.

[0224] It should be noted that the "fourth convolution operation" in step S330 above is a conventional technique in this field, so its specific process will not be elaborated here. The purpose of this "fourth convolution operation" is to output the predicted bounding box pbox corresponding to the training fused feature map f. The specific process of "performing the fourth convolution operation on each of the n-layer training fused feature maps to output the predicted bounding box corresponding to the training fused feature map" is also a conventional technique in this field.

[0225] In some embodiments, the above-mentioned method of predicting the category of the training detection target using training text features e and n-layer training fusion feature map f to obtain the category information of the training detection target in the training fusion feature map f includes:

[0226] Calculate the second text-region similarity matrix corresponding to the training fusion feature map f using the training text feature e, the training fusion feature map f, and the following expression:

[0227]

[0228] The second text-region similarity matrix is ​​used to represent the training fusion feature map f of the j-th layer. j The similarity of each pixel in the training text feature e to each category is given by the following: Norm(e) represents the training feature map obtained after the second normalization process on the training text feature e; α and β are the learnable second scaling factor and learnable second translation factor, respectively. This represents the matrix multiplication operation;

[0229] The fused feature map f with the training at layer j is calculated using the second text-region similarity matrix and the following expression. j Corresponding category probability feature map:

[0230] p j = sigmoid(s2);

[0231] Wherein, sigmoid represents the second nonlinear activation process, which is fused with the training feature map f of the j-th layer. j The corresponding category probability feature map p j The training fusion feature map f used to characterize the j-th layer j The probability that each pixel belongs to the training detection target of the above categories.

[0232] In some embodiments, the image coding submodule V described above is obtained by performing low-rank fine-tuning on a pre-trained multi-scale feature extraction submodel. The low-rank fine-tuning process includes:

[0233] For each convolutional layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to that convolutional layer is constructed to form a new multi-scale feature extraction sub-model; the new multi-scale feature extraction sub-model is used as the feature extraction sub-model.

[0234] The expression for the weight parameters of each convolutional layer in the new multi-scale feature extraction sub-model is as follows:

[0235] W = W0 + ΔW = W0 + BA;

[0236] Where W0 represents the weight parameters of each convolutional layer in the pre-trained multi-scale feature extraction sub-model, ΔW represents the weight parameters of the low-rank parameter update branch corresponding to that convolutional layer, and B and A represent trainable parameters; in the above training process, the weight parameters of all convolutional layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

[0237] By decomposing the aforementioned ΔW into the multiplication of two matrices B and A, the number of training parameters for the defect detection model or network is significantly reduced, saving computational resources, reducing memory usage, and improving network detection performance. For example, assuming the shape dimension of W0 is [z×k], the shape dimensions of B and A can be [z×r] and [r×k], respectively; where r is typically set to 4. During the training of the defect detection model or network, W0 remains frozen and is not updated; ΔW is the result of multiplying B and A, where B and A are trainable parameters. When using the trained defect detection model or network for inference, W0 and ΔW can be merged without increasing inference latency.

[0238] It's important to note that Low-Rank Adaptation (LoRA) is an efficient parameter fine-tuning technique that enables rapid customization of large language models by adding a small number of trainable parameters while keeping the original model parameters unchanged. This technique decomposes the weight matrix, such as W mentioned above, into two smaller matrices, W0 and ΔW, approximating fully supervised fine-tuning with fewer parameters, thus significantly reducing the number of trainable parameters for downstream tasks. The core of low-rank fine-tuning lies in freezing the pre-trained model weights. One advantage of this approach is that when deployed to a production environment, only the updated weight matrix needs to be computed and stored, and inference can be performed as usual, without additional latency compared to other methods because no additional layers need to be added. Furthermore, the effectiveness of low-rank fine-tuning lies not only in its significant reduction in computation and storage requirements but also in its ability to achieve similar results to full fine-tuning with less computation and fine-tuning time. This makes low-rank fine-tuning a highly efficient and effective fine-tuning technique, particularly suitable for the rapid adaptation and customization of large language models in resource-constrained environments.

[0239] As can be seen, in some embodiments, low-rank adaptation can be used to construct low-rank parameter update branches corresponding to each convolutional layer in the pre-trained multi-scale feature extraction sub-model, thereby forming a new multi-scale feature extraction sub-model. When training the defect detection model, low-rank fine-tuning of the low-rank parameter update branches requires fewer training parameters and has a faster fitting speed. While retaining the feature extraction capability and generalization performance learned from large-scale data, it achieves rapid fitting of samples for specific tasks, prevents catastrophic forgetting, and significantly improves the detection performance of the defect detection model or defect detection network of this application.

[0240] In some embodiments, the expression for the total loss function in step 400 above is:

[0241] Loss=δ1L r +δ2L c ;

[0242] Among them, L r Let L represent the first loss function. c The second loss function is represented by δ1 and δ2, which are the preset weight coefficients of the first and second loss functions, respectively. Those skilled in the art can determine δ1 and δ2 according to actual needs; their values ​​are not specifically limited here. The first loss function is determined based on the trained predicted bounding boxes and the corresponding ground truth bounding boxes; the second loss function is determined based on the ground truth class label values ​​and predicted probabilities / confidence levels of the categories of the detected targets.

[0243] In some embodiments, the first loss function described above can be a Distance-IoU Loss loss function. Distance-IoU Loss focuses more on the position of the bounding box / predicted box, resulting in faster convergence and better performance. Distance-IoU Loss improves upon GIOU Loss by incorporating the distance between the center points of the bounding boxes, thus better guiding the model training process. The core idea of ​​DIOU Loss is to minimize the normalized distance between the center points of two bounding boxes. By including the center point distance in the loss function calculation, DIOU Loss can better measure the positional relationship between two bounding boxes, thereby improving the model's prediction accuracy of the target location. For example, the expression for the first loss function is:

[0244]

[0245] Where, p r The above training prediction box, t r This represents the ground truth bounding box corresponding to the training predicted bounding box. p r The center point, The above t represents r The center point, the above The above indicates and The Euclidean distance between them, where η represents the bounding distance of p. r and t r The length of the diagonal of the smallest bounding rectangle, where IoU represents the operation of finding the intersection-union ratio.

[0246] It should be noted that the above η and IoU(p r ,t r The specific process of obtaining the loss function is a conventional method in this field, and therefore will not be elaborated here. Any loss function that can regress the training prediction boxes can be used as the first loss function mentioned above. In other words, those skilled in the art can determine the first loss function themselves according to actual needs.

[0247] In some embodiments, those skilled in the art can determine the specific form and type of the second loss function according to actual needs. For example, the second loss function can be a binary classification cross-entropy loss, etc. As another example, the expression for the second loss function can be:

[0248]

[0249] Among them, the above The training fusion feature map f represents the j-th layer. j The loss is calculated by training the training class probability feature map of the target to be detected, where N represents the total number of layers in the training class probability feature map. This represents the loss of the k-th pixel in the corresponding training class probability feature map, where K represents the total number of pixels in the training class probability feature map.

[0250] It should be noted that the above The specific process of obtaining it is a conventional method in this field, so it will not be described in detail here.

[0251] The above The expression is:

[0252]

[0253] Among them, S t ∑S represents the predicted probability / confidence of the category of the target object detected in the training process at the k-th pixel above. uThis represents the sum of predicted probabilities / confidences for the categories of all non-trained detection targets in the k-th pixel. Those skilled in the art can also determine the training and non-training detection targets according to actual needs; no specific restrictions are imposed here.

[0254] It should be noted that the above S t and ∑S u The specific process of obtaining it is a conventional method in this field, so it will not be described in detail here.

[0255] In some embodiments, because the probability of a product having a defect of the dent category is low, and data collection for it is difficult, defects of the dent category are not visible to the defect detection model G or defect detection network when training the aforementioned defect detection model G or defect detection network; the aforementioned defect detection model G or defect detection network is only trained on training sample images of the crush and broken categories. Please refer to... Figure 8 When using a defect detection model G or defect detection network for inference / detection, the image to be detected, x, and the corresponding preset prompt text are input into the defect detection model G or defect detection network. The corresponding preset prompt text can include language descriptions related to the aforementioned crush, broken, and dent defects (note: dent defect samples were not used during training). The defect detection model G or defect detection network outputs the category information and / or location information, such as predicted bounding boxes, of the detection targets of interest to the user in the image x. The preset prompt text input by the user is generally quite arbitrary; different users may use different language descriptions for the same image and the same type of detection target. Therefore, the preset prompt text input by the user only needs to describe the corresponding category of detection targets of interest to the user in the image to be detected. For example, the preset prompt text could be descriptions such as "detecting scratches in the image," "all scratches," or "the detection target is scratches." Figure 8 The three dashed boxes in the figure are prediction boxes obtained by the defect detection model G, and the corresponding defect categories (such as crush, broken and dent) are also shown.

[0256] As can be seen, in some embodiments, the defect detection method of this application employs a "prompt-before-detection" strategy. That is, by adding category prompts (such as preset prompt text corresponding to the image to be detected), and then text encoding the preset prompt text or structured text to obtain text features, these text features are used to guide the accurate identification of unknown category defects of interest to the user, achieving end-to-end detection. This lowers the technical threshold for users, reduces labor costs, and improves the production efficiency of manufacturers. The aforementioned "prompt-before-detection" strategy makes the detection process flexible and convenient, and the defect detection model or network exhibits high zero-shot performance. Here, "high zero-shot performance" refers to the defect detection model or network's ability to identify new, unseen category targets such as defects with high detection accuracy and excellent performance.

[0257] As can be seen, in some embodiments, the defect detection model or defect detection network of this application is based on text-guided defect detection technology, which can be trained on training samples of known categories, to achieve accurate detection of known and unknown defects in images and improve detection performance.

[0258] As can be seen, in some embodiments, after acquiring the image to be detected and the corresponding preset prompt text, the defect detection method of this application converts the preset prompt text into structured text with a unified form / format by constraining and adapting it. Then, using the structured text as a guide, it predicts the category and / or location of the target of interest in the image to be detected x. This can enhance the adaptability to different scenarios and improve the detection performance of the defect detection model or defect detection network.

[0259] As can be seen, in some embodiments, the defect detection model or defect detection network of this application can template the preset prompt text to adapt to different task scenarios. For example, by inputting the nouns of the target in the image to be detected into a predefined text template, a structured text description can be generated, enhancing the consistency of the structured text and helping to improve the detection performance of the defect detection model or defect detection network in various tasks and scenarios.

[0260] As can be seen, in some embodiments, the defect detection model or defect detection network of this application can adopt a lightweight design to obtain a simplified and efficient network structure. At the same time, it uses low-rank fine-tuning (LoRA) technology to freeze the computationally intensive pre-trained visual feature extraction network structure and perform low-rank fine-tuning on some weight parameters. This requires fewer training samples, has a faster fitting speed, fewer learnable parameters, shorter training time, and lower memory usage, saving time and effort, reducing equipment costs for manufacturing enterprises, and achieving cost reduction and efficiency improvement. In addition, it can retain the excellent detection capabilities and strong generalization performance obtained from big data, meet the requirements of industrial production for inference time consumption, and realize real-time inference.

[0261] As can be seen, in some embodiments, the defect detection model G or defect detection network can be trained on large-scale industrial datasets to obtain a state-of-the-art baseline model, which can be directly used for detection. However, for specific tasks and specific datasets, in order to ensure detection accuracy, the defect detection model G or defect detection network can be trained on the basis of the above-mentioned baseline model using the low-rank fine-tuning method described above, based on collected samples with known defect types, to achieve accurate identification of defects of known and unknown categories.

[0262] The above is a description of the defect detection method. Some embodiments of this application also disclose a defect detection device. This defect detection device includes:

[0263] A camera is used to capture images of industrial products to be inspected and form an image of the product to be inspected.

[0264] The processor, connected to the camera, is used to process the image to be detected according to the defect detection method of any embodiment of this application, and to obtain the location information and / or category information of the target of interest in the image to be detected.

[0265] In some embodiments, the processor includes: an image acquisition and prompting module for acquiring an image to be detected and a corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detected target; and a data processing module for predicting the category and / or location of the detected target of interest in the image to be detected x, guided by structured text, to obtain the category information and / or location information of the detected target of interest.

[0266] It should be noted that the specific processes of "acquiring the image to be detected and the corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target" and "using structured text as a guide to predict the category and / or location of the target of interest in the image to be detected, thereby obtaining the category information and / or location information of the target of interest" have been discussed in detail in steps S100 and S300 of the defect detection method of this application, respectively, and therefore will not be repeated here.

[0267] In some embodiments, the processor further includes a text structuring module for performing structuring processing on the preset prompt text to obtain structured text; wherein the structured text t has a fixed format.

[0268] It should be noted that the specific process and technical effect of "processing the preset prompt text into a structured text t" have been discussed in detail in step S200 of the defect detection method of this application, so they will not be repeated here.

[0269] In some embodiments, the data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; the data processing module further includes: a localization submodule and / or a classification submodule; wherein, the text encoding submodule is used to perform text encoding on structured text to obtain text features; the image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v. i Where i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the greater the n ≥ 2; the feature fusion submodule is used to fuse the first feature map v of layer n. i The system integrates text features with text features to obtain an n-layer fused feature map; a classification submodule is used to predict the category of the detected target using text features and the n-layer fused feature map to obtain the category information of the detected target in the image to be detected; and a localization submodule is used to predict the position of the detected target using the n-layer fused feature map to obtain the position information of the detected target.

[0270] It should be noted that since the specific processing procedure and the technical effects that the data processing module can achieve have been discussed in detail in the defect detection method of this application, they will not be repeated here.

[0271] The above is a description of the defect detection method. Some embodiments of this application also disclose a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the defect detection method of any embodiment of this application.

[0272] The above is a description of the defect detection method. Some embodiments of this application also disclose a computer-readable storage medium, including a program that can be executed by a processor to implement the defect detection method as described in any embodiment of this application.

[0273] This document describes various exemplary embodiments with reference to them. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operational steps and components for performing operational steps can be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system. For example, one or more steps can be deleted, modified, or combined with other steps.

[0274] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. Furthermore, as those skilled in the art will understand, the principles herein can be reflected in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices such as hard disks and floppy disks, optical storage devices such as CD-ROMs, DVDs, Blu-ray discs, flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing on the computer or other programmable data processing apparatus can generate means for performing a specified function. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that instructions stored in the computer-readable storage medium can form an article of manufacture including means for implementing the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that instructions executing on the computer or other programmable apparatus can provide steps for implementing the specified function.

[0275] While the principles herein have been illustrated in various embodiments, numerous modifications to the structure, arrangement, proportions, elements, materials, and components, particularly suited to specific environmental and operational requirements, may be used without departing from the principles and scope of this disclosure. These modifications and other alterations or alterations will be included within the scope of this document.

[0276] The foregoing specific descriptions have been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of this disclosure. Therefore, considerations for this disclosure are to be illustrative rather than restrictive, and all such modifications are to be included within its scope. Similarly, advantages, other advantages, and solutions to problems with respect to various embodiments have been described above. However, benefits, advantages, solutions to problems, and any elements that produce these, or make them more explicit, should not be construed as critical, essential, or necessary. The term “comprising” and any other variations thereof as used herein are non-exclusive inclusion, meaning that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed or not part of the process, method, system, article, or apparatus. Furthermore, the term “coupled” and any other variations thereof as used herein refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections, and / or any other connections.

[0277] Those skilled in the art will recognize that many changes can be made to the details of the above embodiments without departing from the basic principles of the invention. Therefore, the scope of the invention should be determined only by the claims.

Claims

1. A defect detection method, characterized in that, include: Acquire the image to be detected and the corresponding preset prompt text; wherein, the preset prompt text is a description of the category of the target of interest in the image to be detected; Convert the preset prompt text into structured text; Text features are obtained by text encoding the structured text, and a fused feature map is obtained based on the text features and the image to be detected; Based on the text features and the fused feature map, calculate the second text-region similarity matrix corresponding to the fused feature map; Based on the second text and region similarity matrix, a category probability feature map corresponding to the fused feature map is obtained; wherein, the category probability feature map is used to characterize the probability that each pixel in the fused feature map belongs to the detected target of the category; The category of the detected target is predicted using the category probability feature map to obtain the category information of the detected target in the image to be detected; and / or, the location information of the detected target in the image to be detected is obtained by predicting the location of the detected target based on the fused feature map.

2. The defect detection method as described in claim 1, characterized in that, The step of converting the preset prompt text into structured text includes: The preset prompt text is processed into structured text to obtain the structured text; wherein the structured text has a fixed format.

3. The defect detection method as described in claim 1, characterized in that, A fused feature map is obtained based on the text features and the image to be detected, including: Multi-scale feature extraction is performed on the image to be detected to obtain an n-layer first feature map v. i Where i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the more likely n ≥ 2; The n-layer first feature map v i Each feature is fused with the text features to obtain an n-layer fused feature map; The step of calculating the second text-region similarity matrix corresponding to the fusion feature map based on the text features and the fusion feature map includes: using the text features and the n-layer fusion feature map to calculate the second text-region similarity matrix corresponding to the n-layer fusion feature map; The step of predicting the category of the detected target using the category probability feature map to obtain the category information of the detected target in the image to be detected includes: The category probability feature map corresponding to the n-layer fusion feature map is obtained by calculating the second text and region similarity matrix corresponding to the n-layer fusion feature map. Based on the category probability feature map corresponding to the n-layer fused feature map, the category of the detected target is predicted to obtain the category information of the detected target in the image to be detected; The step of predicting the position of the detected target in the image to be detected based on the fused feature map to obtain the position information of the detected target in the image to be detected includes: using the n-layer fused feature map to predict the position of the detected target to obtain the position information of the detected target in the image to be detected.

4. The defect detection method as described in claim 3, characterized in that, The n-layer first feature map v i The text features are fused to obtain an n-layer fused feature map, including: An n-layer text attention network is used to process the first feature map v in the nth layer. i Each feature is fused with the text features to obtain an n-layer fused feature map; Specifically, for the text attention network of the nth layer, the first feature map of the nth layer is used as the first input feature map, the text features are used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain the output feature map, and the obtained output feature map is used as the fusion feature map of the nth layer. For the text attention network at layer k, a first convolution operation and upsampling operation are performed on the fused feature map at layer (k+1) to obtain a corresponding upsampled feature map, wherein the resolution of the upsampled feature map is the same as the resolution of the first feature map at layer k; the first feature map at layer k and the corresponding upsampled feature map are added element-wise to obtain an element-wise added feature map; the element-wise added feature map is used as the first input feature map, the text feature is used as the second input feature map, and the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, which is used as the fused feature map at layer k, where k = n-1, ..., 1; wherein the text attention network is used to enhance the first feature map v. i The location information of the target is detected.

5. The defect detection method as described in claim 4, characterized in that, The step of inputting the first input feature map and the second input feature map into the text attention network to obtain the output feature map includes: The first input feature map d and the second input feature map e, which are input into the text attention network, are used to calculate the first text-region similarity matrix using the following expression: ； Among them, the This indicates that a second convolutional feature map is obtained by performing a second convolution operation on the first input feature map d. This indicates that the second input feature map e is subjected to a first normalization process to obtain a feature map after the first normalization process. and stated These are respectively a learnable first scaling factor and a learnable first translation factor, wherein... This represents the matrix multiplication operation; The first text and region similarity matrix s1 is maximized to obtain the first text and region most similarity matrix. The first nonlinear activation process is applied to the first text-region most similarity matrix to obtain a text-region attention map; An attention-enhanced feature map is obtained by performing an element-wise multiplication operation on the text-region attention map and the first input feature map d input to the text attention network; A third convolution operation is performed on the attention enhancement feature map to obtain the corresponding fused feature map, and the corresponding fused feature map is used as the output feature map of the text attention network.

6. The defect detection method as described in claim 3, characterized in that, The step of predicting the location of the detected target in the image to be detected using the n-layer fused feature map includes: The n-layer fused feature map is subjected to a fourth convolution operation to output a prediction box corresponding to the fused feature map; wherein the prediction box is used to represent the position information of the detected target.

7. The defect detection method as described in claim 3, characterized in that, The step of calculating the second text-region similarity matrix corresponding to the n-layer fused feature map using the text features and the n-layer fused feature map includes: Calculate the result using the text features, the fused feature map, and the following expression, and the result of the first... Layer fusion feature map f j The corresponding second text-region similarity matrix: ； Among them, the This indicates that the text features are subjected to a second normalization process to obtain a feature map after the second normalization process. and stated These are the learnable second scaling factor and the learnable second translation factor, respectively. This represents the matrix multiplication operation; The step of calculating the category probability feature map corresponding to the n-layer fused feature map based on the similarity matrix between the second text and the region includes: Calculate the similarity between the second text and region matrix and the following expression, and the similarity with the first... Layer fusion feature map f j Corresponding category probability feature map: ； Among them, the This represents the second nonlinear activation process, which is related to the first... Layer fusion feature map f j Corresponding category probability feature map Used to characterize the first Layer fusion feature map f j The probability that each pixel belongs to the detected target of the category.

8. The defect detection method as described in claim 3, characterized in that, The defect detection method is implemented based on a defect detection model, which includes a data processing module. The structured text is used as a guide to process the image to be detected. The category and / or location of the target of interest are predicted to obtain category information and / or location information of the target of interest, including: The data processing module uses the structured text as a guide to process the image to be detected. The category and / or location of the target of interest are predicted to obtain the category information and / or location information of the target of interest.

9. The defect detection method as described in claim 8, characterized in that, The data processing module includes a text encoding submodule, an image encoding submodule, and a feature fusion submodule. The data processing module also includes a localization submodule and / or a classification submodule. The step of obtaining text features by text encoding the structured text includes: using the text encoding submodule to perform text encoding on the structured text to obtain text features; Specifically, the process of performing multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v i This includes: using the image encoding submodule to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v. i ; Wherein, the first feature map v of the nth layer i The features are fused with the text features to obtain an n-layer fused feature map f. j This includes: using the feature fusion submodule to process the n-layer first feature map v i The features are fused with the text features to obtain an n-layer fused feature map f. j ; Wherein, the text features and the n-layer fused feature map f are utilized j Predicting the category of the detected target to obtain category information of the detected target in the image to be detected includes: The classification submodule is used to analyze the text features and the n-layer fused feature map f. j The process is performed to predict the category of the detected target and obtain the category information of the detected target in the image to be detected; Wherein, the use of the n-layer fused feature map f j Predicting the location of the target to obtain the location information of the target in the image to be detected includes: The localization submodule is used to process the n-layer fused feature map f j The process is performed to predict the location of the detected target and obtain the location information of the detected target in the image to be detected.

10. The defect detection method as described in claim 9, characterized in that, The defect detection model also includes a text structuring module; The step of converting the preset prompt text into structured text includes: The structured text is obtained by using the text structuring module to perform structuring processing on the preset prompt text.

11. The defect detection method as described in claim 10, characterized in that, The defect detection model is trained through the following training process: Acquire training sample images, as well as preset training prompt text and annotation data corresponding to the training sample images. The annotation data includes the ground truth bounding boxes of training detection targets of interest in the training sample images. The ground truth bounding boxes correspond to the preset training prompt text. The preset training prompt text is a description of the category of the training detection targets of interest in the training sample images. The text structuring module converts the preset training prompt text into training structured text; The data processing module uses the training structured text as a guide to predict the category and / or location of the target of interest in the training sample image, thereby obtaining a training category probability feature map and / or a training prediction box for the target of interest; wherein, the training category probability feature map is used to characterize the predicted probability of each pixel in the training sample image belonging to the category of the target of interest; and the training prediction box is used to characterize the location information of the target of interest. The defect detection model is trained according to the total loss function to obtain the trained defect detection model; wherein, the total loss function is determined by a first loss function and a second loss function; the first loss function is determined based on the training prediction box and the ground truth box corresponding to the training prediction box; the second loss function is determined based on the ground truth class label value of the category of the training detection target and the predicted probability, wherein the ground truth class label value is obtained based on the preset training prompt text.

12. The defect detection method as described in claim 11, characterized in that, The data processing module uses the training structured text as guidance to predict the category and / or location of the target of interest in the training sample image, thereby obtaining a training category probability feature map and / or training prediction box for the target of interest, including: The training text features are obtained by using the text encoding submodule to encode the training structured text. The image encoding submodule is used to perform multi-scale feature extraction on the training sample images to obtain an n-layer first training feature map v. i ; The feature fusion submodule is used to combine the n-layer first training feature map v i The n-layer training fusion feature map f is obtained by fusing the features of the training text with the features of the training text. j ; Wherein, the feature map f is fused using the training text features and the n-layer training. j Predicting the category of the training detection target to obtain the category information of the training detection target in the training sample image includes: The classification submodule utilizes the training text features and the n-layer training fusion feature map f j The category information of the training detection target in the training sample image is obtained by predicting the category of the training detection target; Wherein, the fusion feature map f is trained using the n layers. j Predicting the location of the training detection target to obtain the location information of the training detection target in the training sample image includes: The localization submodule utilizes the n-layer training and fusion feature map f j The location information of the training detection target in the training sample image is obtained by predicting the location of the training detection target. The text encoding submodule is pre-trained, and its weight parameters are not updated during the training process described above.

13. The defect detection method as described in claim 12, characterized in that, The feature fusion submodule includes an n-layer text attention network; Wherein, the feature fusion submodule is used to combine the n-layer first training feature map v i The n-layer training fusion feature map f is obtained by fusing the features of the training text with the features of the training text. j ,include: An n-layer text attention network is used to process the first training feature map v of the n layers. i Each feature is fused with the features of the training text to obtain an n-layer training fusion feature map; For the text attention network of the nth layer, the first training feature map of the nth layer is used as the first training input feature map, the training text features are used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain the training output feature map, and the obtained training output feature map is used as the training fusion feature map of the nth layer. For the text attention network at layer k, a first convolution operation and upsampling operation are performed on the training fusion feature map at layer k+1 to obtain the corresponding training upsampled feature map, wherein the resolution of the training upsampled feature map is the same as the resolution of the first training feature map at layer k; the first training feature map at layer k and the corresponding training upsampled feature map are added element-wise to obtain the training element-wise added feature map; the training element-wise added feature map is used as the first training input feature map, the training text features are used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain the training output feature map, and the obtained training output feature map is used as the training fusion feature map at layer k; where k=n-1,…,1.

14. The defect detection method as described in claim 12, characterized in that, The image encoding submodule is obtained by performing low-rank fine-tuning on the pre-trained multi-scale feature extraction submodel; wherein, the low-rank fine-tuning process includes: For each convolutional layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to that convolutional layer is constructed to form a new multi-scale feature extraction sub-model; this new multi-scale feature extraction sub-model is used as the feature extraction sub-model; wherein, the expression for the weight parameters of each convolutional layer in the new multi-scale feature extraction sub-model is: ； Among them, the This represents the weight parameters of each convolutional layer in the pre-trained multi-scale feature extraction sub-model. This represents the weight parameters of the low-rank parameter update branch corresponding to the convolutional layer. and stated The parameters are trainable; in the above training process, the weight parameters of all convolutional layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

15. The defect detection method as described in claim 11, characterized in that, The expression for the total loss function of the above defect detection model or defect detection network is: ； Among them, the Denotes the first loss function, the The second loss function is represented by δ1 and δ2, which are the preset weight coefficients of the first loss function and the second loss function, respectively.

16. The defect detection method as described in claim 15, characterized in that, The expression for the first loss function is: ； Among them, the The training prediction box represents the training prediction box. This represents the ground truth bounding box corresponding to the trained prediction box. Indicates the The center point, the Indicates the The center point, the Indicates the and stated The Euclidean distance between them, the Indicates the area surrounding the and stated The length of the diagonal of the smallest enclosing rectangle, the This represents the operation of finding the intersection, union, and ratio.

17. The defect detection method as described in claim 16, characterized in that, The expression for the second loss function is: ， ； Among them, the The training fusion feature map f represents the j-th layer. j The loss of the training category probability feature map of the target being detected, as described in the text, where N represents the total number of layers in the training category probability feature map. This represents the loss for the k-th pixel in the corresponding training category probability feature map, where K represents the total number of pixels in the training category probability feature map; wherein, the The expression is: The This represents the predicted probability / confidence of the category of the trained detection target in the k-th pixel. This represents the sum of the predicted probabilities / confidences of the categories of all non-trained detected targets in the k-th pixel.

18. A defect detection device, characterized in that, include: A camera is used to capture images of industrial products to be inspected and form an image of the product to be inspected. A processor, connected to the camera, is used to process the image to be detected by the defect detection method according to any one of claims 1-8, to obtain the location information and / or category information of the target of interest in the image to be detected.

19. The defect detection device as described in claim 18, characterized in that, The processor includes: The image acquisition and prompting module is used to acquire the image to be detected and the corresponding preset prompt text; wherein, the preset prompt text is a description of the category of the detected target; The data processing module is used to encode the structured text to obtain text features, and to obtain a fused feature map based on the text features and the image to be detected; to calculate a second text-region similarity matrix corresponding to the fused feature map based on the text features and the fused feature map; to calculate a category probability feature map corresponding to the fused feature map based on the second text-region similarity matrix; wherein the category probability feature map is used to characterize the probability that each pixel in the fused feature map belongs to the category of the detected target; to predict the category of the detected target using the category probability feature map to obtain the category information of the detected target in the image to be detected; and / or, to predict the position of the detected target based on the fused feature map to obtain the position information of the detected target in the image to be detected.

20. The defect detection device as described in claim 19, characterized in that, The processor also includes: The text structuring module is used to perform structuring processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

21. The defect detection device as described in claim 19 or 20, characterized in that, The data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; The data processing module further includes: a positioning submodule and / or a classification submodule; The text encoding submodule is used to encode structured text to obtain text features; The image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v. i Where i = 1, ..., n, and the larger the value of i, the larger the corresponding first feature map v. i The smaller the resolution, the more likely n ≥ 2; The feature fusion submodule is used to fuse the n-layer first feature map v i The features are fused with the text features to obtain an n-layer fused feature map f. j ; The classification submodule is used to utilize the text features and the n-layer fused feature map f j The category of the detected target is predicted to obtain the category information of the detected target in the image to be detected; The localization submodule is used to utilize the n-layer fused feature map f j The location information of the detected target is obtained by predicting the location of the detected target.

22. A computer program product, comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the defect detection method according to any one of claims 1-17.

23. A computer-readable storage medium, characterized in that, Includes a program that can be executed by a processor to implement the defect detection method as described in any one of claims 1 to 17.

Images