Defect detection method and device

Abstract

The invention provides a defect detection method and a defect detection device. The defect detection method comprises the following steps: acquiring a to-be-detected image and a corresponding preset prompt text; the preset prompt text is a description made by a category of an interested detection target in the to-be-detected image; converting the preset prompt text into a structured text; and predicting the category and / or position of the interested detection target in the to-be-detected image by taking the structured text as guidance. According to the method, the problem of detecting the defects of known categories and / or unknown categories in the to-be-detected image can be solved, and the detection performance of the defects is improved. The device comprises a text structuring module used for structuring a preset prompt text to obtain a structured text; the image acquisition and prompt module is used for acquiring a to-be-detected image and a corresponding preset prompt text; and the data processing module is used for predicting the category and / or position of the detection target by taking the structured text as guidance.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a defect detection method and a device thereof. Background Art

[0002] At present, the image defect detection method usually relies on a large amount of data collected manually, and the above large amount of data is categorized and labeled accordingly, and then a closed category data set, that is, a category data set of known defect types obtained through the above annotation, is used to train the defect detection model, so as to achieve the purpose of using the trained defect detection model to detect defects in the above image. However, in actual industrial production, due to the diversity and unknown nature of the above defect categories, the defect detection model trained using the closed category data set has poor defect recognition effect on the open category data set, that is, the category set with unknown defect categories.

[0003] The above diversity mainly refers to the wide variety of defects in the image, which covers different shapes, sizes, positions, colors, textures, etc. For example, the surface of a mobile phone protective case may produce different types of defects such as "cracks, crushing, dents", and even the intensity of the defects is different, resulting in significant diversity. The above unknown mainly refers to the fact that in industrial production, people in this field cannot collect all possible defects of industrial products. Therefore, the defect detection model obtained by training with closed category data sets cannot effectively detect unknown defects in industrial products; at the same time, since some categories of defects of industrial products are not clearly labeled, missed or incompletely labeled, it will also increase the difficulty of the above defect detection model to identify unknown defects to a certain extent. A high-precision defect detection model requires a complete category and sufficient number of training samples to ensure its performance. However, since the generation of unknown category defects in images is uncertain, defect detection models are prone to missed detection or false detection.

[0004] Therefore, it is necessary to improve the above-mentioned deficiencies in the prior art. Summary of the invention

[0005] The present invention provides a defect detection method and device thereof, which mainly solve the problem of detecting defects of known categories and / or unknown categories in an image to be detected, and improve the detection performance of the above defects.

[0006] According to the first aspect, an embodiment provides a defect detection method. The defect detection method includes:

[0007] Acquire an image to be detected and a corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target of interest in the image to be detected;

[0008] Converting the preset prompt text into structured text;

[0009] Using the structured text as a guide, the category and / or position of the detection target of interest in the to-be-detected image x is predicted to obtain category information and / or position information of the detection target of interest.

[0010] In one embodiment, converting the preset prompt text into a structured text includes: performing structured processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

[0011] In one embodiment, the method of using the structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected to obtain the category information and / or position information of the detection target of interest includes:

[0012] Performing text encoding on the structured text to obtain text features; performing multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v i ; Where i = 1, ..., n and the larger the value of i, the corresponding first feature map v i The smaller the resolution, the n≥2;

[0013] The first feature map v of the n-th layer i Perform feature fusion with the text features respectively to obtain n layers of fusion feature maps;

[0014] The category of the detection target is predicted using the text features and the n-layer fused feature map to obtain category information of the detection target in the image to be detected; and / or, the position of the detection target is predicted using the n-layer fused feature map to obtain position information of the detection target in the image to be detected.

[0015] In one embodiment, the first feature map v of the n-th layer is i The features are respectively fused with the text features to obtain n layers of fused feature maps, including:

[0016] The n-layer text attention network is used to convert the n-layer first feature map v i Performing feature fusion with the text features respectively to obtain n layers of fused feature maps; wherein, for the text attention network of the nth layer, the first feature map of the nth layer is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as the fused feature map of the nth layer;

[0017] For the k-th layer of the text attention network, a first convolution operation and an upsampling operation are performed on the k+1-th layer of fused feature map to obtain a corresponding upsampled feature map, wherein the resolution of the upsampled feature map is the same as the resolution of the first feature map of the k-th layer; the first feature map of the k-th layer and the corresponding upsampled feature map are added element by element to obtain an element-by-element addition feature map; the element-by-element addition feature map is used as a first input feature map, the text feature is used as a second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as a k-th layer of fused feature map, wherein k=n-1, ..., 1;

[0018] Wherein, the text attention network is used to enhance the first feature map v i The location information of the detected target;

[0019] In one embodiment, the step of inputting the first input feature map and the second input feature map into the text attention network to obtain an output feature map includes:

[0020] The first input feature map d and the second input feature map e input into the text attention network are used to calculate the first text and region similarity matrix through the following expression:

[0021]

[0022] Wherein, the Conv(d) represents performing a second convolution operation on the first input feature map d to obtain a second convolution feature map, the Norm(e) represents performing a first normalization operation on the text feature to obtain a first normalized feature map, the μ and the δ are respectively a learnable first scaling factor and a learnable first translation factor, and the Represents the operation of matrix multiplication;

[0023] Performing a maximum value operation on the first text and region similarity matrix s1 to obtain the first text and region most similarity matrix;

[0024] Performing a first nonlinear activation process on the first text and the region most similarity matrix to obtain a text-region attention map;

[0025] Performing an element-by-element multiplication operation on the text-region attention map and the first input feature map d input into the text attention network to obtain an attention enhancement feature map;

[0026] A third convolution operation is performed on the attention enhancement feature map to obtain a corresponding fused feature map, and the corresponding fused feature map is used as the output feature map of the text attention network.

[0027] In one embodiment, the step of using the n-layer fusion feature map to predict the position of the detection target to obtain the position information of the detection target in the image to be detected includes:

[0028] A fourth convolution operation is performed on each of the n layers of fused feature maps to output a prediction box corresponding to the fused feature map; wherein the prediction box is used to represent the position information of the detection target.

[0029] In one embodiment, the method of using the text feature and the n-layer fused feature map to predict the category of the detection target to obtain the category information of the detection target in the fused feature map includes:

[0030] The fusion feature map f of the jth layer is calculated by using the text feature, the fusion feature map and the following expression: j The corresponding second text and region similarity matrix:

[0031]

[0032] Wherein, the Norm(e) represents a feature map obtained by performing a second normalization process on the text feature, the α and the β are respectively a learnable second scaling factor and a learnable second translation factor, and the Represents the operation of matrix multiplication;

[0033] The fusion feature map f of the jth layer is calculated by using the second text and region similarity matrix and the following expression: j The corresponding category probability feature map:

[0034] p j =sigmoid(s2);

[0035] Wherein, the sigmoid represents the second nonlinear activation processing, and the fusion feature map f with the jth layer j The corresponding category probability feature map p j Used to represent the fusion feature map f of the jth layer j The probability that each pixel in belongs to the detected target of the category.

[0036] In one embodiment, the defect detection method is implemented based on a defect detection model, and the defect detection model includes a data processing module; wherein, the structured text is used as a guide to predict the category and / or position of the detection target of interest in the image to be detected x, and the category information and / or position information of the detection target of interest is obtained, including: the data processing module uses the structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected x, and obtains the category information and / or position information of the detection target of interest.

[0037] In one embodiment, the data processing module includes a text encoding submodule, an image encoding submodule, and a feature fusion submodule, and the data processing module also includes a positioning submodule and / or a classification submodule;

[0038] The step of performing text encoding on the structured text to obtain text features comprises: performing text encoding on the structured text using the text encoding submodule to obtain text features;

[0039] Wherein, the multi-scale feature extraction is performed on the image to be detected to obtain the n-layer first feature map v i ,include:

[0040] The image encoding submodule is used to extract multi-scale features of the image to be detected to obtain the first feature map v of the nth layer. i ; wherein the first feature map v of the n-th layer i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j , including: using the feature fusion submodule to transform the n-layer first feature map v i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j ; Wherein, the use of the text features and the n-layer fusion feature map f j Predicting the category of the detection target to obtain category information of the detection target in the image to be detected includes:

[0041] The classification submodule is used to classify the text features and the n-layer fusion feature map f j Processing is performed to predict the category of the detection target and obtain the category information of the detection target in the image to be detected; wherein the use of the n-layer fusion feature map f j Predicting the position of the detection target to obtain the position information of the detection target in the image to be detected includes:

[0042] The positioning submodule is used to perform the n-layer fusion feature map f jProcessing is performed to predict the position of the detection target and obtain the position information of the detection target in the image to be detected.

[0043] In one embodiment, the defect detection model further includes a text structuring module; the step of converting the preset prompt text into a structured text includes:

[0044] The text structuring module is used to perform structural processing on the preset prompt text to obtain the structured text.

[0045] In one embodiment, the defect detection model is trained through the following training process:

[0046] Acquire a training sample image, and preset training prompt text and annotation data corresponding to the training sample image, wherein the annotation data includes a real target frame of a training detection target of interest in the training sample image; wherein the real target frame corresponds to the preset training prompt text; wherein the preset training prompt text is a description of a category of the training detection target of interest in the training sample image;

[0047] The text structuring module converts the preset training prompt text into a training structured text;

[0048] The data processing module uses the training structured text as a guide to predict the category and / or position of the detection target of interest in the training sample image, and obtains a training category probability feature map and / or a training prediction frame of the training detection target; wherein the training category probability feature map is used to characterize the predicted probability that each pixel in the training sample image belongs to the category of the training detection target; and the training prediction frame is used to characterize the position information of the training detection target;

[0049] The defect detection model is trained according to the total loss function to obtain the trained defect detection model; wherein the total loss function is determined by a first loss function and a second loss function; the first loss function is determined according to the training prediction box and the real box corresponding to the training prediction box; the second loss function is determined according to the real category label value of the category of the training detection target and the predicted probability, and the real category label value is obtained based on the preset training prompt text.

[0050] In one embodiment, the data processing module uses the training structured text as a guide to predict the category and / or position of the detection target of interest in the training sample image, and obtains a training category probability feature map and / or a training prediction box of the training detection target, including:

[0051] Using the text encoding submodule to perform text encoding on the training structured text to obtain training text features;

[0052] The image encoding submodule is used to perform multi-scale feature extraction on the training sample image to obtain an n-layer first training feature map v i ;

[0053] The feature fusion submodule is used to transform the first training feature map v of the n-layer i Perform feature fusion with the training text features respectively to obtain n layers of training fusion feature graphs;

[0054] The method of using the training text features and the n-layer training fusion feature map to predict the category of the training detection target to obtain the category information of the training detection target in the training sample image includes:

[0055] The classification submodule predicts the category of the training detection target by using the training text features and the n-layer training fusion feature map to obtain the category information of the training detection target in the training sample image;

[0056] Wherein, the step of using the n-layer training fusion feature map to predict the position of the training detection target to obtain the position information of the training detection target in the training sample image includes:

[0057] The positioning submodule predicts the position of the training detection target using the n-layer training fusion feature map to obtain the position information of the training detection target in the training sample image;

[0058] The text encoding submodule is obtained through pre-training, and in the above training process, the weight parameters of the text encoding submodule are not updated.

[0059] In one embodiment, the feature fusion submodule includes an n-layer text attention network; wherein the n-layer first training feature map v is converted into a text attention network by using the feature fusion submodule. i The features are respectively fused with the training text features to obtain n layers of training fusion feature graphs, including:

[0060] The n-layer text attention network is used to train the first training feature map v i The features are fused with the training text features to obtain n layers of training fusion feature graphs; wherein,

[0061] For the text attention network of the nth layer, the first training feature map of the nth layer is used as the first training input feature map, the training text feature is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain a training output feature map, and the obtained training output feature map is used as the training fusion feature map of the nth layer;

[0062] For the k-th layer text attention network, a first convolution operation and an upsampling operation are performed on the k+1-th layer training fusion feature map to obtain a corresponding training upsampling feature map, wherein the resolution of the training upsampling feature map is the same as the resolution of the first training feature map of the k-th layer; an element-by-element addition operation is performed on the first training feature map of the k-th layer and the corresponding training upsampling feature map to obtain a training element-by-element addition feature map; the training element-by-element addition feature map is used as the first training input feature map, the training text feature is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain a training output feature map, and the obtained training output feature map is used as the k-th layer training fusion feature map, wherein k=n-1,…,1.

[0063] In one embodiment, the image encoding submodule is obtained by performing low-rank fine-tuning on a pre-trained multi-scale feature extraction submodel; wherein the low-rank fine-tuning process includes:

[0064] For each convolution layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to the convolution layer is constructed to form a new multi-scale feature extraction sub-model; the new multi-scale feature extraction sub-model is used as the feature extraction sub-model; wherein the expression of the weight parameter of each convolution layer in the new multi-scale feature extraction sub-model is:

[0065] W = W0 + ΔW = W0 + BA;

[0066] Wherein, W0 represents the weight parameter of each convolution layer in the pre-trained multi-scale feature extraction sub-model, ΔW represents the weight parameter of the low-rank parameter update branch corresponding to the convolution layer, and B and A represent trainable parameters; wherein, in the above training process, the weight parameters of all convolution layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

[0067] In one embodiment, the total loss function of the above-mentioned defect detection model or defect detection network is expressed as:

[0068] Loss = δ1L r +δ2L c ;

[0069] Among them, the L r represents the first loss function, the L c represents the second loss function; δ1 and δ2 are respectively the preset weight coefficient of the first loss function and the preset weight coefficient of the second loss function.

[0070] In one embodiment, the expression of the first loss function is:

[0071]

[0072] Among them, the p r represents the training prediction box, the t r represents the real box corresponding to the training prediction box, Indicates that p r The center point of Indicates the t r The center point of Indicates the and stated The Euclidean distance between the two, η represents the distance between the two surrounding p r and the r The length of the diagonal of the minimum enclosing rectangle, and the IoU represents the operation of calculating the intersection over union.

[0073] In one embodiment, the expression of the second loss function is:

[0074]

[0075] Among them, the Represents the training fusion feature map f of the jth layer j The loss of the training category probability feature map of the training detection target in the training detection target, wherein N represents the total number of layers of the training category probability feature map, and represents the loss of the k-th pixel in the corresponding training category probability feature map, and K represents the total number of pixels in the training category probability feature map;

[0076] Among them, the The expression is:

[0077]

[0078] Among them, the S t represents the predicted probability / confidence of the category of the training detection target in the k-th pixel, ∑S u Represents the sum of the predicted probabilities / confidences of the categories of all non-training detection targets in the k-th pixel.

[0079] According to the second aspect, an embodiment provides a defect detection device. The defect detection device includes:

[0080] A camera is used to capture images of industrial products to be inspected and form images to be inspected;

[0081] A processor is connected to the camera and is used to process the image to be detected according to the defect detection method described in any embodiment of the present application to obtain the position information and / or category information of the detection target of interest in the image to be detected.

[0082] In one embodiment, the processor includes: an image acquisition and prompt module, used to acquire the image to be detected and the corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target; a data processing module, used to use the structured text as a guide to predict the category and / or position of the detection target of interest in the image x to be detected, and obtain the category information and / or position information of the detection target of interest.

[0083] In one embodiment, the processor further includes: a text structuring module for performing structural processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

[0084] In one embodiment, the data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; the data processing module also includes: a positioning submodule and / or a classification submodule; wherein the text encoding submodule is used to perform text encoding on the structured text to obtain text features; the image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v i ; Where i = 1, ..., n and the larger the value of i, the corresponding first feature map v i The smaller the resolution, the n ≥ 2; the feature fusion submodule is used to combine the n-layer first feature map v i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j ; The classification submodule is used to utilize the text features and the n-layer fusion feature map f j The category of the detection target is predicted to obtain the category information of the detection target in the image to be detected; the positioning submodule is used to use the n-layer fusion feature map f j The position of the detection target is predicted to obtain the position information of the detection target.

[0085] According to a third aspect, an embodiment provides a computer program product, including a computer program and / or instructions. When the computer program and / or instructions are executed by a processor, the defect detection method described in any embodiment of the present application is implemented.

[0086] According to a fourth aspect, an embodiment provides a computer-readable storage medium, characterized in that it includes a program, and the program can be executed by a processor to implement the defect detection method as described in any embodiment of the present application.

[0087] The beneficial effects of this application are:

[0088] The defect detection method of the present application includes: obtaining an image to be detected and a corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target of interest in the image to be detected; converting the preset prompt text into a structured text; using the structured text as a guide, predicting the category and / or position of the detection target of interest in the image to be detected, and obtaining the category information and / or position information of the detection target of interest; that is, the present application obtains the preset prompt text corresponding to the image to be detected, and then performs text encoding on the preset prompt text or structured text to obtain text features, and uses the text features to guide and indicate the target in the image to be detected (such as the category and / or position of the detection target of interest mentioned above), and aligning / matching the image to be detected and the text features, thereby achieving accurate detection of defects of unknown categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 A schematic diagram of a defect detection method according to an embodiment of the present invention;

[0090] Figure 2 A schematic diagram of a processing flow of a defect detection model or a defect detection network according to an embodiment;

[0091] Figure 3 A schematic diagram of a flow chart of obtaining category information and / or location information of a detection target of interest according to an embodiment;

[0092] Figure 4 A schematic diagram of a data processing process of a feature fusion submodule in an embodiment;

[0093] Figure 5 A schematic diagram of a process of inputting a first input feature map and a second input feature map into a text attention network to obtain an output feature map according to an embodiment;

[0094] Figure 6 A schematic diagram of a defect detection model training process according to an embodiment;

[0095] Figure 7 A flowchart for training a defect detection model for an embodiment;

[0096] Figure 8 A schematic diagram of a defect detection model according to an embodiment predicting an image to be detected. DETAILED DESCRIPTION

[0097] The present invention is further described in detail below by specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments adopt associated similar element numbers. In the following embodiments, many detailed descriptions are for making the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different situations, or can be replaced by other elements, materials, methods. In some cases, some operations related to the present application are not shown or described in the specification, this is to avoid the core part of the present application being overwhelmed by too much description, and for those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations according to the description in the specification and the general technical knowledge in the art.

[0098] In addition, the features, operations or characteristics described in the specification can be combined in any appropriate manner to form various implementations. At the same time, the steps or actions in the method description can also be interchanged or adjusted in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a required sequence, unless otherwise specified that a certain sequence must be followed.

[0099] The serial numbers assigned to the components herein, such as "first", "second", etc., are only used to distinguish the objects described and do not have any order or technical meaning. The "connection" and "connection" mentioned in this application include direct and indirect connections unless otherwise specified.

[0100] In the field of machine vision detection, the commonly used detection scheme for unknown category defects in images usually adopts the preset vector query method. Among them, the preset vector query method first inputs the image features, the corresponding text features and the preset query vector into the multimodal decoder to obtain the predicted query vector; then determines the detection box and the target text according to the predicted query vector and the above-mentioned image and text features; finally, the target text corresponding to the target prediction vector is used as the predicted category to realize the detection of unknown category defects. However, the inventor found through research: First, the preset vector query method lacks template adaptation of the description text of the detection target, and the input content is highly arbitrary, resulting in weak scene adaptability and poor generalization performance; second, the acquisition of the preset query vector requires a preset threshold, and the setting of the threshold requires manual parameter adjustment, which is time-consuming and labor-intensive, and usually requires high professional knowledge personnel, thereby raising the threshold for users and increasing the production burden of enterprises; third, the network structure of the multimodal decoder that decodes the image, text features and preset query vector is usually more complex, with a large number of network parameters and poor real-time reasoning. The combined effect of the above three aspects leads to a decline in its detection performance.

[0101] In view of the problems existing in the above-mentioned prior art, the present application proposes a defect detection method. One purpose of the defect detection method is to train a defect detection model or a defect detection network, etc., when only data of known categories are involved in the training, solve the problem of detecting defects of known categories and / or unknown categories in the image, and improve the detection performance of the above-mentioned defects. Among them, the defect detection network can be a network that can achieve basically the same functions as the defect detection model.

[0102] The technical concept of this defect detection method is: based on the text information added by the user, such as the preset prompt text mentioned below, the defect detection model or defect detection network is trained on target samples of known categories in an end-to-end manner, thereby realizing accurate detection of defects of known categories and / or unknown categories in the image.

[0103] The principle of this defect detection method, which is based on the guidance of user-added text information, to achieve accurate detection of defects, especially defects of unknown categories, is that through a text encoder network pre-trained with big data, such as the text encoding submodule described below, and a visual network, such as the image encoding submodule described below, the matching relationship between the target area and the text information (such as the preset prompt text or structured text described below) in the training data (such as the training sample image described below) can be learned, that is, the above text information guides the target information in the instructions and / or enhanced image features, and realizes the alignment of image features and text features, thereby realizing open set target detection, that is, accurate detection of defects of unknown categories. In essence, a series of calculation processes (this part will be described in detail below) are used to detect the similarity or matching of text features and visual features such as fused feature maps, so as to ultimately achieve open set detection, that is, accurate detection of unknown defects.

[0104] The technical solution of the present application will be described in detail below in conjunction with embodiments.

[0105] Please refer to Figure 1 , the present application provides a defect detection method. The defect detection method comprises:

[0106] Step S100: obtaining an image to be detected and a preset prompt text;

[0107] Step S200: converting the preset prompt text into structured text;

[0108] Step S300: predicting the detection target of interest in the image to be detected.

[0109] Specifically, in step S100, the preset prompt text is a description of the category of the detection target of interest in the image to be detected.

[0110] Specifically, in step S300, the category and / or position of the detection target of interest in the image to be detected x is predicted with the structured text as a guide, so as to obtain the category information and / or position information of the detection target of interest.

[0111] It should be noted that, in step S100, the categories of some or all of the detection targets in the above-mentioned image to be detected may be unknown. In other words, the category information contained in the above-mentioned detection targets may be known and / or unknown.

[0112] In some embodiments, in the above step S100, those skilled in the art can select the above image to be detected according to actual needs. For example, the above image to be detected can be an image obtained by photographing an industrial product or other object. The user can determine the "detection target of interest in the image to be detected" according to actual needs.

[0113] In some embodiments, in step S200, please refer to Figure 2 , convert the preset prompt text prompt into structured text t, including:

[0114] The preset prompt text prompt is structured to obtain a structured text t; wherein the structured text t has a fixed format.

[0115] It should be noted that the purpose of the above-mentioned "structuring the preset prompt text prompt to obtain the structured text t" is to perform constraint adaptation on various preset prompt texts through templates, and make the structured text t conducive to the defect detection model or defect detection network to identify the effective features of defects in the image to be detected, thereby enhancing the adaptability of the defect detection model or defect detection network to different scenarios, so as to ultimately improve the detection performance of the present defect detection method. The above-mentioned "constraint adaptation" refers to: converting the user's language input such as the above-mentioned preset prompt text into a template with a fixed format, and then normalizing and standardizing the user's language input, so as to facilitate the text encoder in the defect detection model to accurately extract the corresponding effective features in the "structured text", so as to facilitate the defect detection model or defect detection network to better understand the user's description such as the intention to be expressed by the above-mentioned preset prompt text. For example, in some embodiments, the target object that the user is interested in in the image to be detected is "dog", and the language prompts for it are inconsistent among different users, and there is a great deal of randomness. For example, descriptions such as "dog in the image to be detected", "detected target dog", "dog located in the upper left position of the image to be detected", etc. are diverse and random. By performing the above-mentioned "constraint adaptation" through templating, the noun of the detection target is extracted as "dog", and using a preset template formula, such as "the detected object is {xxx}", the preset prompt text can be templated into "the detected object is a dog", that is, the corresponding structured text.

[0116] For some examples, please refer to Figure 2 and Figure 3 In step S300, the category and / or position of the detection target of interest in the detection image x is predicted using the structured text as a guide, and the category information and / or position information of the detection target of interest is obtained, including:

[0117] Step S310: performing text encoding on the structured text t to obtain text features e; performing multi-scale feature extraction on the image to be detected x to obtain an n-layer first feature map v;

[0118] Step S320: The first feature map v of the nth layer i The features are fused with the text features e to obtain n-layer fusion feature graph f;

[0119] Step S330: Use the text feature e and the n-layer fusion feature map f to predict the category of the detection target and obtain the category information of the detection target in the image to be detected. pj ; and / or, using the n-layer fusion feature map f to predict the category of the detection target and obtain the location information box of the detection target in the image to be detected.

[0120] In some embodiments, in step S310, those skilled in the art can determine the specific expression form of the above text features according to actual needs. For example, the dimensional shape of the above text feature e can be expressed as [n, m], where n is the number of categories of detection targets that the user is interested in in the image to be detected, and m is the length of the preset feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network in the training stage can be different from n in the defect detection model or defect detection network in the inference stage.

[0121] It should be noted that the above text features are essentially feature vectors, and those skilled in the art can determine the specific form of the above text features according to actual needs.

[0122] In some embodiments, in step S310, those skilled in the art may determine the number of layers of the first feature map according to actual needs. For example, the number of the first feature map may be 2 layers or 4 layers, and those skilled in the art generally determine the number of layers of the first feature map to be 3 layers. Each layer of the first feature map v i Each represents a scale. Among them, i = 1, ..., n and the larger the value of i, the larger the corresponding first feature map v i The smaller the resolution, n≥2.

[0123] In some embodiments, in step S320, the n-layer first feature graph v is respectively fused with the text features to obtain the n-layer fused feature graph f, including:

[0124] An n-layer text attention network is used to fuse the first feature map v of the n-layer with the text features to obtain an n-layer fused feature map f, where:

[0125] For the text attention network of the nth layer, the first feature map of the nth layer is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain the output feature map, and the obtained output feature map is used as the fusion feature map f of the nth layer;

[0126] For the text attention network of the kth layer, a first convolution operation and an upsampling operation are performed on the k+1th layer fusion feature map to obtain a corresponding upsampled feature map, wherein the resolution of the upsampled feature map is the same as that of the first feature map of the kth layer; an element-by-element addition operation is performed on the first feature map of the kth layer and the corresponding upsampled feature map to obtain an element-by-element addition feature map; the element-by-element addition feature map is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as the kth layer fusion feature map f, wherein k=n-1,…,1.

[0127] It should be noted that the text attention network is used to guide and enhance the position information of the detected target in the first feature map v.

[0128] For some examples, please refer to Figure 4 , at this time n = 3, a three-layer text attention network Attn is used to fuse the three first feature maps (such as v1, v2 and v3) with the text feature e to obtain a three-layer fusion feature map f (Note: f includes f 1、 f2 and f3). For the third-layer text attention network Attn, the first feature map v3 of the third layer is used as the first input feature map, the text feature e is used as the second input feature map, the first input feature map and the second input feature map are input into the third-layer text attention network Attn to obtain the output feature map, and the obtained output feature map is used as the third-layer fusion feature map f3; for the second-layer text attention network Attn, the third-layer fusion feature map f3 is subjected to the first convolution operation Conv and the upsampling operation Up to obtain the corresponding upsampled feature map; wherein the resolution of the upsampled feature map is the same as the resolution of the first feature map v2 of the second layer; the first feature map v2 of the second layer is subjected to the element-by-element addition operation Add with the corresponding upsampled feature map to obtain the element-by-element addition feature map; the element-by-element addition feature map is used as the first input feature map, the text feature e is used as the second input feature map, the first input feature map and the second input feature map are input into the second-layer text attention network Attn to obtain the output feature map, and the obtained output feature map is used as the second-layer fusion feature map f2. Since the process of obtaining the first-layer fusion feature map f1 and the second-layer fusion feature map f2 is basically the same, it will not be repeated here.

[0129] It should be noted that the "first convolution operation", "upsampling operation" and "element-by-element addition operation" in the above step S320 are all conventional technical means in this field. For example, the "first convolution operation" can use a convolution layer with a convolution kernel of 1×1. Those skilled in the art can determine the specific process of the above operations by themselves, which will not be repeated here. The purpose of the above-mentioned first convolution operation is to make the output channel of the feature map obtained by the first convolution operation consistent with the first feature map of the second layer, such as v2. The purpose of the above-mentioned upsampling operation Up is to make the resolution of the "corresponding upsampled feature map" consistent with the resolution of the first feature map v2 of the second layer, so that the "corresponding upsampled feature map" can be fused with the first feature map v2 of the second layer. The above-mentioned fusion is achieved by the element-by-element addition operation Add. The element-by-element addition operation Add is to add the elements in corresponding positions of the feature map.

[0130] It should be noted that the role of the text attention network Attn is to realize the interaction and fusion of text features e and multi-scale features (such as v1, v2 and v3), which is essentially to complete the multimodal feature fusion of text information and visual features.

[0131] For some examples, please refer to Figure 5 , the above-mentioned inputting the first input feature map and the second input feature map into the text attention network to obtain the output feature map includes:

[0132] Step 10: The first input feature map d and the second input feature map e of the input text attention network Attn are used to calculate the first text and region similarity matrix s1 through the following expression:

[0133]

[0134] Among them, the first text and region similarity matrix is ​​used to characterize the similarity between each pixel in the first input feature map d and each category in the second input feature map e, Conv(d) means that the second convolution operation is performed on the first input feature map d to obtain the second convolution feature map, the second input feature map e is the text feature e, Norm(e) means that the first normalization processing is performed on the text feature e to obtain the feature map after the first normalization processing, μ and δ are the learnable first scaling factor and the learnable first translation factor, respectively. Represents the operation of matrix multiplication;

[0135] Step 20: performing a maximum value operation on the first text and region similarity matrix s1 to obtain the first text and region most similarity matrix;

[0136] Step 30: performing a first nonlinear activation process on the first text and the region most similarity matrix to obtain a text-region attention map;

[0137] Step 40: performing an element-by-element multiplication operation on the text-region attention map and the first input feature map d of the input text attention network to obtain an attention enhancement feature map;

[0138] Step 50: Perform a third convolution operation on the attention enhancement feature map to obtain the corresponding fusion feature map f j , the corresponding fusion feature map f j As the output feature map of the text attention network.

[0139] It should be noted that in the above step 10, μ and δ are not preset parameters, but learnable parameters. In essence, μ and δ are respectively a learnable first scaling factor and a learnable first translation factor. In other words, μ and δ can be adaptively adjusted / updated along with the training process of the above defect detection model or the above defect detection network, which allows the above defect detection model or the above defect detection network to better adapt to the training tasks and data.

[0140] It should be noted that the process of calculating the "first text and region similarity matrix s1" in step 10 is an intermediate calculation process for obtaining the text-region attention map. The purpose of performing the second convolution operation on the first input feature map d to obtain the second convolution feature map is to make the number of channels of the second convolution feature map equal to the length of the second input feature map e. In essence, the matrix multiplication operation in step 10 is What is calculated is the multi-category similarity of the text features of each category and the pixel-by-pixel text-region in the fused features (such as Conv(d) above).

[0141] It should be noted that the purpose of the above step 20 is to obtain the first text and region most similarity matrix by taking the maximum value operation on the first text and region similarity matrix s1 in the channel dimension. The first text and region most similarity matrix is ​​used to represent the maximum value of the similarity between each pixel in the first input feature map d and each category in the text feature e.

[0142] It should be noted that in the above step 30, the first text and the region most similarity matrix are subjected to the first nonlinear activation process to obtain the text-region attention map. The function of the text-region attention map is to combine the information from the language prompt, that is, the preset prompt text such as prompt, with the image features (such as the first feature map v n layer i) to obtain the degree of importance of the foreground area, thereby indicating the target importance of the image feature in space. For example, the preset prompt text is "dog", and the text-region attention map indicates the importance of the pixel position of the foreground, that is, the existence of the detection target of interest "dog" in the space of image features. The text-region attention map is essentially a pixel-by-pixel probability map. The value of each element in the text-region attention map is used to indicate the probability that the pixel corresponding to the element in the first input feature map d belongs to a category.

[0143] It should be noted that in the above step 40, the "attention enhanced feature map" represents the multimodal fusion information of text features and visual features. Its function is to align text features with image features such as the above-mentioned multi-scale features, and guide and enhance the spatial importance of image features through text features, which is beneficial for the defect detection model or defect detection network to enhance the perception of the location information of the area where the above-mentioned detection target is located.

[0144] It should be noted that the purpose of the above step 50: performing the third convolution operation on the attention enhancement feature map is to make the number of its channels equal to the length of the text feature, and then obtain the corresponding fusion feature map f j .

[0145] It should be noted that the "second convolution operation", "first normalization processing", "maximum value operation", "first nonlinear activation processing", "element-by-element multiplication operation" and "third convolution operation" in the above steps 10 to 50 are all conventional technical means in the art. For example, the "second convolution operation" can use 1×1 convolution, where 1×1 convolution is a special convolution operation with a convolution kernel size of 1×1; the "first nonlinear activation processing" can use Sigmoid activation function for nonlinear activation processing, etc. Personnel in the art can determine the specific process of the above operations by themselves, and will not be repeated here.

[0146] It should be noted that "applying text attention to image features in a text-guided manner, injecting language information into image features, and realizing the interaction and fusion of text and visual information" means: fusing text features with the above-mentioned multi-scale features to obtain the desired results. In other words, the entire process of fusing text features with the above-mentioned multi-scale features is used to "realize the interaction and fusion of text and visual information."

[0147] In some embodiments, in step S330, using the n-layer fusion feature map f j The position of the detection target is predicted to obtain the position information of the detection target in the image to be detected, including: j The fourth convolution operation is performed respectively and the output and fusion feature map f jThe corresponding prediction box; wherein the prediction box is used to represent the location information of the detected target.

[0148] It should be noted that the "fourth convolution operation" in the above step S330 is a conventional technical means in the art, so its specific process will not be repeated here. The purpose of the "fourth convolution operation" is to output and fuse the feature map f j The specific process of "performing the fourth convolution operation on the n-layer fused feature map and outputting the prediction box corresponding to the fused feature map" also belongs to the conventional technical means in this field. For example, the dimension shape of any layer of the prediction box can be [b, 4, h, w], where b represents the batch number, 4 represents the position information of the prediction box, and h and w are the height and width of the fused feature map of the current detection layer, respectively.

[0149] In some embodiments, in step S330, the text features and the n-layer fusion feature map f are used. j Predict the category of the detected target and obtain the fusion feature map f j The category information of the detected target includes:

[0150] Step S331: Using text feature e and fusion feature map f j And the following expression is used to calculate and fuse the feature map f j The corresponding second text and region similarity matrix:

[0151]

[0152] Among them, the second text and region similarity matrix is ​​used to represent the fusion feature map f of the jth layer j The similarity between each pixel in and each category in the text feature e is respectively expressed as follows. Norm(e) represents the feature map obtained by performing the second normalization processing on the text feature e. The above α and β are respectively the learnable second scaling factor and the learnable second translation factor. Represents the operation of matrix multiplication;

[0153] Step S332: Calculate the fusion feature map f of the jth layer using the second text and region similarity matrix and the following expression: j The corresponding category probability feature map:

[0154] p j =sigmoid(s2);

[0155] Among them, sigmoid represents the second nonlinear activation processing, and the fusion feature map f of the jth layer j The corresponding category probability feature map p j The fused feature map f used to represent the jth layer jThe probability that each pixel belongs to the above categories of detection targets.

[0156] It should be noted that the "fourth convolution operation", "second normalization processing", and "second nonlinear activation processing" in the above steps S331 and S332 are all conventional technical means in the art. For example, the "second nonlinear activation processing" can use a Sigmoid activation function to perform nonlinear activation processing. The specific process of the above operations can be determined by those skilled in the art, and will not be described in detail here.

[0157] It should be noted that α and β in the above step S331 are not preset parameters, but learnable parameters. In essence, the above α and β are respectively a learnable second scaling factor and a learnable second translation factor, which can be adaptively adjusted / updated along with the training process of the above defect detection model or the above defect detection network, and their role is to allow the above defect detection model or the above defect detection network to better adapt to the training tasks and data.

[0158] In some embodiments, the above defect detection method may also be implemented based on a defect detection network. The defect detection network may be a network that can implement substantially the same functions as the defect detection model.

[0159] For some examples, please refer to Figure 2 The above defect detection method is implemented based on the defect detection model G, and the defect detection model G includes a data processing module; wherein step S300: using the structured text t as a guide, predicting the category and / or position of the detection target of interest in the detection image x, and obtaining the category information and / or position information of the detection target of interest, includes:

[0160] The data processing module uses the structured text t as a guide to predict the category and / or position of the detection target of interest in the detection image x, and obtains the category information p of the detection target of interest j and / or location information box.

[0161] For some examples, please refer to Figure 2 The data processing module includes a text encoding submodule T, an image encoding submodule V, and a feature fusion submodule F. The data processing module also includes a positioning submodule L and / or a classification submodule C; wherein, in step S310, text encoding is performed on the structured text t to obtain text features e, including:

[0162] The text encoding submodule T is used to encode the structured text t to obtain the text feature e;

[0163] Wherein, step S310: performing multi-scale feature extraction on the image to be detected x to obtain an n-layer first feature map v, comprising: performing multi-scale feature extraction on the image to be detected x using an image encoding submodule V to obtain an n-layer first feature map v;

[0164] Wherein, step S320: performing feature fusion on the n-layer first feature graph v and the text feature e respectively to obtain the n-layer fused feature graph f, comprising: using the feature fusion submodule F to perform feature fusion on the n-layer first feature graph v and the text feature e respectively to obtain the n-layer fused feature graph f;

[0165] Wherein, step S330: using the text feature e and the n-layer fusion feature map f to predict the category of the detection target to obtain the category information of the detection target in the image to be detected, includes:

[0166] The classification submodule C is used to process the text feature e and the n-layer fusion feature map f to predict the category of the detection target and obtain the category information p of the detection target in the image to be detected. j ;

[0167] Among them, step S330: using the n-layer fusion feature map f to predict the category of the detection target to obtain the location information box of the detection target in the image to be detected, includes:

[0168] The positioning submodule L is used to process the n-layer fusion feature map f to predict the category of the detection target and obtain the location information box of the detection target in the image to be detected.

[0169] In some embodiments, since the language prompts of the detection targets of interest in the detection images input by the user, such as the preset prompt texts, are highly arbitrary in sentence structure and vocabulary and are uncontrollable, please refer to Figure 2 , the defect detection model G also includes a text structuring module M. In step S200, the preset prompt text such as prompt is converted into a structured text t, including:

[0170] The text structuring module M is used to perform structural processing on the preset prompt text prompt to obtain the structured text t.

[0171] In some embodiments, since the language prompts of the detection targets of interest in the detection image input by the user, such as the above-mentioned preset prompt text, are highly arbitrary in sentence structure and vocabulary and are uncontrollable, the function of the above-mentioned text structuring module is to constrain and adapt the target prompts input by the user. For example, a text template can be designed to adapt to the above-mentioned input content of the user. For example, first determine the template formula, where the "template formula" can be defined by the user according to the data and task scenario, for example, "an image of {target object}"; then, extract the target noun of the user's language prompts for the detection target in the detection image, such as "dog", and replace the "target object" in the template formula with "dog"; finally, output the templated and structured text description t.

[0172] It should be noted that the text structuring module can directly adopt the existing technology in the field. Those skilled in the art can determine the specific processing process / form of the text structuring module according to actual needs, as long as the language prompt input such as the preset prompt text is normalized and standardized.

[0173] It should be noted that during the training process of the defect detection model, the user does not need to manually add preset training prompt texts for unknown category defects, but only needs to add corresponding preset training prompt texts for known category defects; when using the defect detection model or the defect detection model for reasoning, the user adds corresponding preset prompt texts to the category of the detection target of interest, and the category of interest includes known categories and / or unknown categories.

[0174] In some embodiments, in order to further enhance the diversity of text input and improve the robustness and generalization performance of the above-mentioned defect detection model or the above-mentioned defect detection network, the text structuring module can adopt a variety of different types of templates to adapt to different detection scenarios.

[0175] In some embodiments, if the language prompts for the detection targets of interest in the image to be detected input by the user are in a unified format and standard in terms of sentence structure and vocabulary, such as the preset prompt text, the defect detection model may not include the text structuring module M. In other words, the preset prompt text is directly used as the structured text.

[0176] In some embodiments, the main function of the text submodule T is to encode the structured text description such as the structured text t to obtain text features.

[0177] It should be noted that the above-mentioned text submodule T can adopt any network that can encode the above-mentioned structured text. For example, the above-mentioned text submodule T can adopt a text encoder (TextEncoder) of a pre-trained CLIP model. Among them, the CLIP (Contrastive Language-Image Pre-training) model is a deep learning model, which represents the concept of a language image contrast pre-training model. The core of the CLIP model is to map both images and texts into the same vector space, so that contrast learning can be performed directly between the two modes. The key innovation of this model lies in the unified vector space, which enables images and texts to calculate similarities directly in the vector space without the need for additional intermediate representations. CLIP uses contrastive learning to perform pre-training, and through training, related images and texts are more closely combined, while unrelated images are separated in the feature space. This unsupervised learning method enables CLIP to learn from text and image data on the Internet, and perform well in tasks in various fields. In addition, since the text encoder Text Encoder of the CLIP model belongs to a network structure that is well known and commonly used by people in the field, the specific structure of the text encoder of the CLIP model will not be repeated here.

[0178] It should be noted that when the above-mentioned text submodule T adopts the text encoder of the pre-trained CLIP model, the weight parameters of the above-mentioned text submodule T in the training stage or the inference stage are not updated. The input of the above-mentioned text submodule is the text description t, and the output of the above-mentioned text submodule is the text feature e. For example, the dimensional shape of the above-mentioned text feature e can be expressed as [n, m], where n here is the number of categories of detection targets that the user is interested in in the image to be detected, and m is the length of the preset feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network in the training stage may be different from n in the defect detection model or defect detection network in the inference stage.

[0179] In some embodiments, the image encoding submodule V is used to extract multi-scale features from the input image to be detected x to obtain n layers of first feature maps v. For example, the number of the first feature maps v can be 2 or 4 layers, and those skilled in the art generally determine the number of layers of the first feature maps to be 3 layers. Each layer of the first feature map v i Each represents a scale. Among them, i = 1, ..., n and the larger the value of i, the larger the corresponding first feature map v iThe smaller the resolution, n≥2. Each layer of the first feature map represents a scale. Specifically, the n-layer first feature map specifically comes from the multi-scale feature map output by the continuous n-layer feature extraction layer of the neural network close to the input end of the neural network. For example, when n=3, the 3-layer first feature map specifically comes from the multi-scale feature map output by the continuous three-layer feature extraction layer of the neural network close to the input end of the neural network. For example, the above-mentioned image encoding submodule V can directly use the backbone network (Backbone) of the existing neural network model to perform multi-scale feature extraction, and "using the backbone network (Backbone) to perform multi-scale feature extraction" belongs to the conventional technical means in this field, so it will not be repeated here.

[0180] It should be noted that the above-mentioned image encoding submodule V can adopt the backbone network of any existing general visual model as its network structure.

[0181] In some embodiments, preferably, based on the lightweight design principle, the above-mentioned image encoding submodule V can adopt a pre-trained ResNet-18 backbone network (Backbone). Among them, ResNet-18 is a convolutional neural network with a depth of 18 layers. This is because the backbone network (Backbone) of ResNet-18 has a compact structure, strong feature extraction capability, and fast reasoning speed. During the entire training phase and reasoning phase of the above-mentioned defect detection model or defect detection network, the weight parameters of the above-mentioned image encoding submodule V remain fixed and are not updated.

[0182] For some examples, please refer to Figure 4 The function of the feature fusion submodule F is to connect the text feature e and the image feature, apply text attention to the image feature (such as the first feature map v of the n-th layer mentioned above) in a text-guided manner, inject the language information in the text feature e into the image feature, and realize the interaction and fusion of text information and visual information. Among them, the image feature v input to the feature fusion submodule F is a multi-scale feature such as the first feature map v of the n-th layer mentioned above i The above-mentioned image features v may include v1, v2 and v3. The overall network of the feature fusion submodule F may be a feature pyramid structure. The feature fusion submodule F can combine the text features e with the first feature map v of each layer. i Fusion is performed to realize the text information in the text feature e on the first feature map v i The n-layer fusion feature map f finally output by the feature fusion submodule F j The number of channels is the same, but the n-layer fusion feature map f j The resolution is different.

[0183] It should be noted that the network structure and processing flow of each text attention network Attn contained in the feature fusion submodule F are consistent.

[0184] When the defect detection model or defect detection network is used for reasoning, the defect detection model or defect detection network can directly output target positioning information and category prediction information in an end-to-end manner to achieve accurate identification of known category defects and / or unknown category defects.

[0185] This application designs corresponding loss functions for training the positioning submodule L and classification submodule C of the entire defect detection model or defect detection network. For the overall structure of the defect detection model or defect detection network, the design of the feature fusion submodule F, the classification submodule C and the loss function of the defect detection model is key.

[0186] For some examples, please refer to Figure 6 and Figure 7 , the above defect detection model G is trained through the following training process:

[0187] Step 100: obtaining a training sample image x, and a preset training prompt text prompt and annotation data corresponding to the training sample image x, wherein the annotation data includes a real target frame of a training detection target of interest in the training sample image; wherein the real target frame corresponds to the preset training prompt text prompt; wherein the preset training prompt text is a description of the category of the training detection target of interest in the training sample image x;

[0188] Step 200: The text structuring module M converts the preset training prompt text prompt into a training structured text t;

[0189] Step 300: The data processing module uses the training structured text t as a guide to predict the category and / or position of the detection target of interest in the training sample image x, and obtains the training category probability feature map p of the training detection target cls and / or train prediction box p r ; Among them, the training category probability feature map p cls It is used to characterize the predicted probability that each pixel in the training sample image x belongs to the category of the training detection target; the training prediction box p r Used to represent the location information of the training detection target;

[0190] Step 400: Train the defect detection model according to the total loss function to obtain a trained defect detection model; wherein the total loss function is composed of the first loss function L r and the second loss function L c Determine; the first loss function L r According to the training prediction box pr and the training prediction box p r The corresponding ground-truth box t r Determine; the second loss function L c The true category label value t of the category of the training detection target cls Determined with the above predicted probabilities, the true category label value is obtained based on the preset training prompt text.

[0191] It should be noted that, since the preset training prompt text is a description of the category of the training detection target of interest in the training sample image x, the "preset training prompt text" can essentially directly represent the "true category label value". That is to say, in some embodiments, the "preset training prompt text" can be directly used as the "true category label value".

[0192] In some embodiments, in step 100, the training samples used by the above-mentioned defect detection model G may include a training sample image x and a preset training prompt text prompt corresponding to the training sample image. The training sample image x and the preset training prompt text prompt are usually in a paired form. For example, an image sample with a known defect category can be collected as the above-mentioned training sample image x. The preset training prompt text prompt is a language prompt for the training detection target in the above-mentioned training sample image x. The user can manually set the preset training prompt text corresponding to the training sample image. The training detection target of interest can also be determined by the user according to actual needs, as long as the real target frame of the training detection target corresponds to the preset training prompt text.

[0193] It should be noted that, in step 100, essentially, the text corresponding to each training detection target in the preset training prompt text can be regarded as the true category label of the training detection target.

[0194] It should be noted that in step 100, the user does not need to manually add preset training prompt texts for unknown category defects, but only needs to add corresponding preset training prompt texts for known category defects. When using the defect detection model or the defect detection model for reasoning, the user adds corresponding preset prompt texts for the category of the detection target of interest, and the category of interest includes known categories and / or unknown categories.

[0195] In some embodiments, in step 200, the text structuring module M is to achieve a templated and structured text description of the preset training prompt text prompt, that is, to generate a training structured text t. The preset training prompt text prompt can be in the form of a noun, a phrase, or a sentence. The training structured text t has a standard or unified structural form. For example, the structural form is unified in the form of a phrase or a sentence, that is, a standard structural form.

[0196] In some embodiments, in step 300, the data processing module uses the training structured text as a guide to predict the category and / or position of the detection target of interest in the training sample image x, and obtains a training category probability feature map and / or a training prediction box of the training detection target, including:

[0197] Using the text encoding submodule T to perform text encoding on the training structured text t to obtain the training text feature e;

[0198] The image encoding submodule V is used to extract multi-scale features from the training sample image x to obtain the n-layer first training feature map v i ;

[0199] The feature fusion submodule F is used to transform the first training feature map v of the n layer i The n-layer training fusion feature graph f is obtained by fusion with the training text feature e respectively. j ;

[0200] Wherein, step 330: using the training text feature e and the n-layer training fusion feature map f j The category of the training detection target is predicted to obtain the category information of the training detection target in the training sample image, including:

[0201] The classification submodule C uses the training text feature e and the n-layer training fusion feature map f j Predicting the category of the training detection target to obtain the category information of the training detection target in the training sample image;

[0202] Wherein, step 330: using n layers of training fusion feature map f j The position of the training detection target is predicted to obtain the position information of the training detection target in the training sample image, including:

[0203] The localization submodule L uses n layers of training to fuse the feature map f j The position of the training detection target is predicted to obtain the position information of the training detection target in the training sample image; wherein, the text encoding submodule is obtained through pre-training, and in the above training process, the weight parameters of the text encoding submodule T are not updated.

[0204] In some embodiments, those skilled in the art can determine the specific expression form of the above training text features according to actual needs. For example, the dimensional shape of the above training text feature e can be expressed as [n, m], where n here is the number of categories of training detection targets of interest to the user in the training sample image, and m is the length of the preset feature vector, for example, m can usually be set to 512. It should be noted that n in the defect detection model or defect detection network in the training stage can be different from n in the defect detection model or defect detection network in the inference stage.

[0205] It should be noted that the above training text features are also feature vectors in nature.

[0206] In some embodiments, the number of layers of the first training feature map can be determined by the person skilled in the art according to actual needs. For example, the number of the first training feature map can be 2 layers or 4 layers, etc. The person skilled in the art generally determines the number of layers of the first training feature map to be 3 layers. i Each represents a scale. Where i = 1, ..., n and the larger the value of i, the larger the corresponding first training feature map v i The smaller the resolution, n≥2.

[0207] In some embodiments, in step 330, the classification submodule C may first use the training text feature e and the training fusion feature map f of each layer. j The category of the training detection target is predicted to obtain the category information of the training detection target in each layer of the training fusion feature map. Then, the category information of the training detection target in each layer of the training fusion feature map is post-processed accordingly to obtain the category information of the training detection target in the training sample image. The above post-processing may include: screening of score thresholds, etc. Since the specific process of the above post-processing belongs to the conventional technical means in this field, it will not be repeated here.

[0208] It should be noted that the positioning submodule L uses the n-layer training fusion feature map f j The position of the training detection target is predicted to obtain the position information of the training detection target in the training sample image, and then the position information of the training detection target in each layer of the training fusion feature map is post-processed accordingly to obtain the position information of the training detection target in the training sample image. The above post-processing may include: screening of score thresholds, non-maximum suppression, etc. The above processing process of "obtaining the position information of the training detection target in the training sample image" is similar to the above processing process of "using n layers of fusion feature maps f j The processing process of "predicting the category of the detection target and obtaining the category information of the detection target in the sample image" remains consistent (that is, the "training prediction box" corresponds to the above categories one by one, and if there is no training prediction box, it defaults to the background), so it will not be repeated here.

[0209] In some embodiments, the feature fusion submodule F includes an n-layer text attention network Attn; wherein, the feature fusion submodule F is used to transform the n-layer first training feature map v i The n-layer training fusion feature graph f is obtained by fusion with the training text feature e respectively. j ,include:

[0210] Step 321: Use the n-layer text attention network to convert the n-layer first training feature map v i Perform feature fusion with the training text feature e respectively to obtain n layers of training fusion feature graphs;

[0211] Step 322: for the text attention network of the nth layer, using the first training feature map of the nth layer as the first training input feature map, using the training text feature as the second training input feature map, inputting the first training input feature map and the second training input feature map into the text attention network to obtain a training output feature map, and using the obtained training output feature map as the training fusion feature map of the nth layer;

[0212] Step 323: For the text attention network of the kth layer, perform a first convolution operation and an upsampling operation on the training fusion feature map of the k+1th layer to obtain a corresponding training upsampling feature map, wherein the resolution of the training upsampling feature map is the same as the resolution of the first training feature map of the kth layer; perform an element-by-element addition operation on the first training feature map of the kth layer and the corresponding training upsampling feature map to obtain a training element-by-element addition feature map; use the training element-by-element addition feature map as the first training input feature map, use the training text features as the second training input feature map, input the first training input feature map and the second training input feature map into the text attention network to obtain a training output feature map, and use the obtained training output feature map as the training fusion feature map of the kth layer, wherein k=n-1,…,1.

[0213] The above text attention network Attn is used to guide and enhance the first training feature map v i For example, please refer to Figure 4 When n=3, a 3-layer text attention network Attn can be used to convert the first training feature map v i The three layers of training fusion feature graphs are obtained by fusion with the training text feature e respectively, as shown in f 1、f2 and f3. For the third-layer text attention network Attn, the first training feature map of the third layer, such as v3, is used as the first training input feature map, the training text feature e is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the third-layer text attention network Attn to obtain the training output feature map, and the obtained training output feature map is used as the third-layer training fusion feature map f3; for the second-layer text attention network Attn, the first convolution operation and upsampling operation are performed on the third-layer training fusion feature map f3 to obtain the corresponding training upsampled feature map. Among them, the resolution of the training upsampled feature map is the same as the resolution of the first training feature map of the second layer, such as v2; the first training feature map of the second layer, such as v2, is added element by element to the corresponding training upsampled feature map to obtain the training element by element addition feature map; the training element by element addition feature map is used as the first training input feature map, the training text feature e is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network Attn of the second layer to obtain the training output feature map, and the obtained training output feature map is used as the second layer training fusion feature map f2. Since the acquisition process of the first layer training fusion feature map f1 and the second layer training fusion feature map f2 is basically the same, it will not be repeated here. The above-mentioned "first convolution operation", "upsampling operation" and "element by element addition operation" are all conventional technical means in this field, and the specific process of the above-mentioned operation can be determined by the person skilled in the art, and it will not be repeated here. The purpose of the above-mentioned first convolution operation is to make the output channel of the training upsampled feature map obtained by the first convolution operation consistent with the first training feature map of the second layer, such as v2. The purpose of the above upsampling operation Up is to make the resolution of the "corresponding training upsampled feature map" consistent with the first training feature map of the second layer, such as v2, so as to realize the fusion of the "corresponding training upsampled feature map" and the first training feature map of the second layer, such as v2. The above fusion is realized by element-by-element addition operation. The element-by-element addition operation is to add the elements at the corresponding positions of the feature map. The role of the text attention network Attn is to realize the training text feature e and the multi-scale feature such as the first training feature map v of the n layer. i The interaction and fusion of text information and visual features is essentially the multimodal feature fusion of text information and visual features.

[0214] In some embodiments, the step of inputting the first training input feature map and the second training input feature map into the text attention network to obtain the training output feature map includes:

[0215] The first training input feature map d and the second training input feature map e of the input text attention network are used to calculate the first text and region similarity matrix through the following expression:

[0216]

[0217] Among them, the first text and region similarity matrix is ​​used to characterize the similarity between each pixel in the first training input feature map d and each category in the second training input feature map e, Conv(d) represents the second training convolution feature map obtained by performing the second convolution operation on the first training input feature map d, the second training input feature map e is the training text feature e, Norm(e) represents the first normalization processing on the training text feature e to obtain the first normalized training feature map, μ and δ are the first learnable scaling factor and the first learnable translation factor, respectively. Represents the operation of matrix multiplication;

[0218] Performing a maximum value operation on the first text and region similarity matrix s1 to obtain the first text and region most similarity matrix;

[0219] Performing a first nonlinear activation process on the first text and the region most similarity matrix to obtain a training text-region attention map;

[0220] Performing an element-by-element multiplication operation on the training text-region attention map and the first training feature map of the input text attention network to obtain a training attention enhanced feature map;

[0221] Perform the third convolution operation on the training attention enhancement feature map to obtain the corresponding training fusion feature map f j , the corresponding training fusion feature map f j As the training output feature map of the text attention network.

[0222] For some examples, please refer to Figure 6 In step S330, the n-layer training fusion feature map f j Predicting the category of the training detection target to obtain the position information of the training detection target in the training image to be detected includes:

[0223] The fourth convolution operation is performed on the n-layer training fusion feature map f respectively to output the prediction box pbox corresponding to the training fusion feature map f; wherein the prediction box pbox is used to represent the position information of the training detection target.

[0224] It should be noted that the "fourth convolution operation" in the above step S330 is a conventional technical means in this field, so its specific process will not be repeated here. Among them, the purpose of the "fourth convolution operation" is to output the prediction box pbox corresponding to the training fusion feature map f. The specific process of "performing the fourth convolution operation on the n layers of training fusion feature maps respectively and outputting the prediction box corresponding to the training fusion feature map" is also a conventional technical means in this field.

[0225] In some embodiments, the above-mentioned use of the training text feature e and the n-layer training fusion feature map f to predict the category of the training detection target to obtain the category information of the training detection target in the training fusion feature map f respectively includes:

[0226] The second text and region similarity matrix corresponding to the training fusion feature map f is calculated using the training text feature e, the training fusion feature map f and the following expression:

[0227]

[0228] Among them, the second text and region similarity matrix is ​​used to represent the training fusion feature map f of the jth layer j The similarity between each pixel in and each category in the training text feature e is respectively expressed as follows. Norm(e) represents the training feature map obtained by performing the second normalization on the training text feature e. α and β are the learnable second scaling factor and the learnable second translation factor, respectively. Represents the operation of matrix multiplication;

[0229] The second text and region similarity matrix and the following expression are used to calculate the training fusion feature map f of the jth layer: j The corresponding category probability feature map:

[0230] p j =sigmoid(s2);

[0231] Among them, sigmoid represents the second nonlinear activation processing, which is fused with the training feature map f of the jth layer. j The corresponding category probability feature map p j The training fusion feature map f used to represent the jth layer j The probability that each pixel in belongs to the training detection target of the above category.

[0232] In some embodiments, the image encoding submodule V is obtained by performing low-rank fine-tuning on the pre-trained multi-scale feature extraction submodel. The low-rank fine-tuning process includes:

[0233] For each convolution layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to the convolution layer is constructed to form a new multi-scale feature extraction sub-model; the new multi-scale feature extraction sub-model is used as the feature extraction sub-model;

[0234] Among them, the expression of the weight parameter of each convolution layer in the new multi-scale feature extraction sub-model is:

[0235] W = W0 + ΔW = W0 + BA;

[0236] Among them, W0 represents the weight parameter of each convolution layer in the pre-trained multi-scale feature extraction sub-model, ΔW represents the weight parameter of the low-rank parameter update branch corresponding to the convolution layer, and B and A represent trainable parameters; Among them, in the above training process, the weight parameters of all convolution layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

[0237] By decomposing the above ΔW into two matrices B and A for multiplication, the amount of training parameters of the defect detection model or defect detection network can be greatly reduced, computing resources can be saved, memory usage can be reduced, and the detection performance of the network can be improved. For example, assuming that the shape dimension of the above W0 is [z×k], the shape dimensions of the above B and A can be [z×r] and [r×k], respectively; wherein r=4 is usually set. During the training process of the defect detection model or defect detection network, the above W0 remains frozen without being updated, and ΔW is the result of the multiplication of B and A, and both B and A are trainable parameters. When using the trained defect detection model or defect detection network for reasoning, the above W0 and the above ΔW can be merged, thereby not increasing any reasoning delay.

[0238] It should be noted that Low-Rank Adaptation (LoRA) is an efficient parameter fine-tuning technique that enables rapid customization of large language models by adding a small number of trainable parameters to the model while keeping the original model parameters unchanged. This technique approximates fully supervised fine-tuning with fewer parameters by decomposing the weight matrix such as W mentioned above into two smaller matrices such as W0 and ΔW mentioned above, thereby greatly reducing the number of trainable parameters for downstream tasks. The core of low-rank fine-tuning is that it freezes the pre-trained model weights. One advantage of this method is that when deployed to a production environment, only the updated weight matrix needs to be calculated and stored, and reasoning is performed as usual, with no additional delay compared to other methods because no more layers need to be attached. In addition, the effectiveness of low-rank fine-tuning is not only reflected in its ability to significantly reduce computing and storage requirements, but also in its ability to achieve similar effects to full fine-tuning with less computing and fine-tuning time. This makes low-rank fine-tuning an efficient and effective fine-tuning technique, especially suitable for rapid adaptation and customization of large language models in resource-constrained environments.

[0239] It can be seen that in some embodiments, low-rank adaptation can be used to build low-rank parameter update branches corresponding to each convolution layer in the pre-trained multi-scale feature extraction sub-model to form a new multi-scale feature extraction sub-model. When training the defect detection model, the low-rank parameter update branch is low-rank fine-tuned, requiring fewer training parameters and fast fitting speed. While retaining the feature extraction ability and generalization performance learned from large-scale data, it can achieve rapid fitting of samples for specific tasks, prevent catastrophic forgetting, and greatly improve the detection performance of the defect detection model or defect detection network of this application.

[0240] In some embodiments, the total loss function in step 400 is expressed as:

[0241] Loss = δ1L r +δ2L c ;

[0242] Among them, L r represents the first loss function, L c Represents the second loss function; δ1 and δ2 are respectively the preset weight coefficients of the first loss function and the preset weight coefficients of the second loss function. Personnel in this field can determine δ1 and δ2 according to actual needs, and their values ​​are not specifically limited here. The first loss function is determined according to the training prediction box and the real box corresponding to the training prediction box; the second loss function is determined according to the real category label value and the predicted probability / confidence of the category of the training detection target.

[0243] In some embodiments, the first loss function may adopt a loss function called Distance-IoU Loss. The Distance-IoU Loss may pay more attention to the position of the bounding box / prediction box, converge faster, and have better performance. The Distance-IoU Loss improves the DIOU Loss loss function by introducing the distance between the center points of the bounding boxes, thereby better guiding the training process of the model. The core idea of ​​DIOU Loss is to minimize the standardized distance between the center points of the two bounding boxes. By incorporating the center point distance into the calculation of the loss function, DIOU Loss can better measure the positional relationship between the two bounding boxes, thereby improving the model's prediction accuracy for the target position. For example, the expression of the first loss function is:

[0244]

[0245] Among them, p r represents the above training prediction box, t r represents the real box corresponding to the training prediction box, Indicates p r The center point of Indicates the above t r The center point of the above Indicates the above and The Euclidean distance between them, the above η represents the enclosing p r and t r The length of the diagonal of the minimum enclosing rectangle, IoU represents the operation of finding the intersection over union.

[0246] It should be noted that the above η and IoU(p r ,t r ) is a conventional method in this field, so it will not be described here. Any loss function that can regress the training prediction box, etc. can be used as the first loss function. In other words, those skilled in the art can determine the first loss function according to actual needs.

[0247] In some embodiments, those skilled in the art may determine the specific form and type of the second loss function according to actual needs. For example, the second loss function may adopt a binary cross entropy loss, etc. For another example, the expression of the second loss function may be:

[0248]

[0249] Among them, the above Represents the j-th layer training fusion feature map f j The loss of the training category probability feature map of the training detection target in the training, where N represents the total number of layers of the training category probability feature map, and represents the loss of the kth pixel in the corresponding training category probability feature map, and the above K represents the total number of pixels in the training category probability feature map.

[0250] It should be noted that the above The specific process of obtaining is a conventional method in the art, so it will not be described in detail here.

[0251] Above The expression is:

[0252]

[0253] Among them, S t Represents the predicted probability / confidence of the category of the training detection target in the k-th pixel point above, ∑S uRepresents the sum of the predicted probabilities / confidences of the categories of all non-training detection targets in the k-th pixel. Personnel in this field can also determine the above-mentioned training detection targets and non-training detection targets according to actual needs, and no specific restrictions are imposed on the above-mentioned training detection targets and non-training detection targets.

[0254] It should be noted that the above S t and ∑S u The specific process of obtaining is a conventional method in the art, so it will not be described in detail here.

[0255] In some embodiments, since the probability of a defect of the category of dent occurring in a certain product is low and its data collection is difficult, when training the above-mentioned defect detection model G or defect detection network, the defect of the category of dent is invisible to the above-mentioned defect detection model G or defect detection network; the above-mentioned defect detection model G or defect detection network is only trained on training sample images of the category of crush and the category of broken. Please refer to Figure 8 , when using the defect detection model G or the defect detection network for reasoning / detection, the image x to be detected and the corresponding preset prompt text are input into the defect detection model G or the defect detection network. The corresponding preset prompt text may include language descriptions related to the above-mentioned crush, broken and dent (Note: defect samples with the category of dent have not been used in the training process). The defect detection model G or the defect detection network outputs the category information and / or location information such as the prediction box of the detection target that the user is interested in in the image to be detected x. Among them, the preset prompt text input by the user is generally more arbitrary, and different users may have different prompt language descriptions for the same image and the same type of detection target. Therefore, the preset prompt text input by the user only needs to be a description of the detection target of the corresponding category that the user is interested in in the image to be detected. For example, the preset prompt text can be descriptions such as "scratches in the detection image", "all scratches", "the detection target is scratches", etc. Among them, Figure 8 The three dotted boxes in FIG. 1 are all prediction boxes obtained by the defect detection model G, and the defect categories corresponding to the prediction boxes (such as crush, broken and dent).

[0256] It can be seen that in some embodiments, the defect detection method of the present application uses a detection strategy of "prompt first, then detect", that is, by adding category prompts (such as preset prompt text corresponding to the image to be detected), and then encoding the preset prompt text or structured text to obtain text features, the text features are used to guide the precise identification of unknown category defects that the user is interested in, and end-to-end detection is achieved, which reduces the user's technical threshold, reduces labor costs, and improves the manufacturer's production efficiency. The above-mentioned "prompt first, then detect" detection strategy can make the detection process flexible and convenient, and the zero-sample performance of the defect detection model or defect detection network is high. Among them, the above-mentioned "high zero-sample performance" means that the defect detection model or defect detection network can identify new and unseen category targets such as defects, and has high detection accuracy and excellent performance.

[0257] It can be seen that in some embodiments, the defect detection model or defect detection network of the present application is based on text-guided defect detection technology, and can be trained on training samples of known categories to achieve accurate detection of known and unknown defects in images and improve detection performance.

[0258] It can be seen that in some embodiments, after obtaining the image to be detected and the corresponding preset prompt text, the defect detection method of the present application converts the preset prompt text into a structured text with a unified form / format by constraining the preset prompt text, and then uses the structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected x, which can enhance the adaptability to different scenarios and improve the detection performance of the defect detection model or defect detection network.

[0259] It can be seen that in some embodiments, the defect detection model or defect detection network of the present application can template the preset prompt text to adapt to different task scenarios. For example, by inputting the noun of the target in the image to be detected into a predefined text template, a structured text description can be generated, which enhances the consistency of the structured text and helps improve the detection performance of the defect detection model or defect detection network in various tasks and scenarios.

[0260] It can be seen that in some embodiments, the defect detection model or defect detection network of the present application can adopt a lightweight design to obtain a streamlined and efficient network structure. At the same time, the low-rank fine-tuning (LoRA) technology is used to freeze the network structure of the pre-trained visual feature extraction with large computational complexity, and low-rank fine-tuning is performed on some weight parameters. The number of training samples required is small, the fitting speed is fast, the number of learnable parameters is small, the training time is short, the video memory occupancy is low, time and labor are saved, the equipment cost of the production enterprise is reduced, and cost reduction and efficiency improvement are achieved. In addition, the excellent detection capability and strong generalization performance obtained in big data can be retained to meet the requirements of industrial production for time-consuming reasoning and realize real-time reasoning.

[0261] It can be seen that in some embodiments, the defect detection model G or defect detection network can be trained on a large-scale industrial data set to obtain a baseline model with advanced performance, which can be directly used for detection. In order to ensure the accuracy of detection for specific tasks and specific data sets, the defect detection model G or defect detection network can be trained on the basis of the above baseline model using the above low-rank fine-tuning method using the collected samples with known defect types, so as to achieve accurate identification of defects of known and unknown categories.

[0262] The above is some description about the defect detection method. In some embodiments of the present application, a defect detection device is also disclosed. The defect detection device includes:

[0263] A camera is used to capture images of industrial products to be inspected and form images to be inspected;

[0264] The processor is connected to the camera and is used to process the image to be detected according to the defect detection method of any embodiment of the present application to obtain the position information and / or category information of the detection target of interest in the image to be detected.

[0265] In some embodiments, the processor includes: an image acquisition and prompt module, used to acquire the image to be detected and the corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target; a data processing module, used to use structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected x, and obtain the category information and / or position information of the detection target of interest.

[0266] It should be noted that the specific process of "obtaining the image to be inspected and the corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target" and the specific process of "using structured text as a guide to predict the category and / or position of the detection target of interest in the image to be inspected, and obtain the category information and / or position information of the detection target of interest" have been discussed in detail in step S100 and step S300 of the defect detection method of the present application, respectively, and will not be repeated here.

[0267] In some embodiments, the processor further includes: a text structuring module, configured to perform structural processing on a preset prompt text to obtain a structured text; wherein the structured text t has a fixed format.

[0268] It should be noted that since the specific process of "structuring the preset prompt text to obtain the structured text t" and the technical effects that can be achieved have been discussed in detail in step S200 of the defect detection method of the present application, they will not be repeated here.

[0269] In some embodiments, the data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; the data processing module also includes: a positioning submodule and / or a classification submodule; wherein the text encoding submodule is used to perform text encoding on the structured text to obtain text features; the image encoding submodule is used to perform multi-scale feature extraction on the image to be detected to obtain the n-layer first feature map v i ; Where i = 1, ..., n and the larger the value of i, the corresponding first feature map v i The smaller the resolution, n ≥ 2; the feature fusion submodule is used to combine the first feature map v of the n layer i The features are fused with the text features to obtain n-layer fused feature maps; the classification submodule is used to predict the category of the detection target using the text features and the n-layer fused feature maps to obtain the category information of the detection target in the image to be detected; the positioning submodule is used to predict the position of the detection target using the n-layer fused feature maps to obtain the position information of the detection target.

[0270] It should be noted that since the specific processing process of the data processing module here and the technical effects that can be achieved have been discussed in detail in the defect detection method of this application, they will not be repeated here.

[0271] The above is some description about the defect detection method. In some embodiments of the present application, a computer program product is also disclosed, including a computer program and / or instructions, which implement the defect detection method of any embodiment of the present application when the computer program and / or instructions are executed by a processor.

[0272] The above is some description about the defect detection method. In some embodiments of the present application, a computer-readable storage medium is also disclosed, including a program, which can be executed by a processor to implement the defect detection method as in any embodiment of the present application.

[0273] This document is described with reference to various exemplary embodiments. However, those skilled in the art will recognize that changes and modifications may be made to the exemplary embodiments without departing from the scope of this document. For example, various operating steps and components for performing the operating steps may be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system. For example, one or more steps may be deleted, modified, or combined with other steps.

[0274] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. In addition, as understood by those skilled in the art, the principles of this article can be reflected in a computer program product on a computer-readable storage medium, which is pre-installed with a computer-readable program code. Any tangible, non-temporary computer-readable storage medium can be used, including magnetic storage devices hard disks, floppy disks, etc., optical storage devices CD to ROM, DVD, Blu Ray disks, etc., flash memory and / or the like. These computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer or other programmable data processing device to form a machine, so that these instructions executed on a computer or other programmable data processing device can generate a device that implements a specified function. These computer program instructions can also be stored in a computer-readable memory, which can instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory can form a manufactured product, including an implementation device that implements a specified function. Computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operating steps are performed on a computer or other programmable device to generate a computer-implemented process, so that the instructions executed on a computer or other programmable device can provide steps for implementing a specified function.

[0275] Although the principles of this invention have been shown in various embodiments, many modifications of structures, arrangements, proportions, elements, materials and components particularly suitable for specific environments and operational requirements can be used without departing from the principles and scope of this invention. The above modifications and other changes or amendments will be included in the scope of this invention.

[0276] The foregoing specific description has been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of the present disclosure. Therefore, the consideration of the present disclosure will be illustrative rather than restrictive, and all these modifications will be included in its scope. Similarly, the advantages, other advantages and solutions to the problems of various embodiments have been described above. However, the benefits, advantages, solutions to the problems and any elements that can produce these, or make them more clear, should not be interpreted as critical, necessary or necessary. The term "include" and any other variants used in this article are all non-exclusive inclusions, so that the process, method, article or device including the list of elements not only includes these elements, but also includes other elements that are not explicitly listed or do not belong to the process, method, system, article or device. In addition, the term "coupled" and any other variants used in this article refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections and / or any other connections.

[0277] Those skilled in the art will appreciate that many changes may be made to the details of the above-described embodiments without departing from the basic principles of the invention. Therefore, the scope of the present invention should be determined solely by the claims.

Claims

1. A defect detection method, characterized in that: include: Acquire an image to be detected and a corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target of interest in the image to be detected; Converting the preset prompt text into structured text; Using the structured text as a guide, the category and / or position of the detection target of interest in the image to be detected is predicted to obtain category information and / or position information of the detection target of interest.

2. The defect detection method according to claim 1, characterized in that: The converting the preset prompt text into structured text comprises: The preset prompt text is structured to obtain the structured text; wherein the structured text has a fixed format.

3. The defect detection method according to claim 1, characterized in that: The method of using the structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected, and obtaining the category information and / or position information of the detection target of interest, includes: Performing text encoding on the structured text to obtain text features; performing multi-scale feature extraction on the image to be detected to obtain an n-layer first feature map v i ; Where i = 1, ..., n and the larger the value of i, the corresponding first feature map v i The smaller the resolution, the n≥2; The first feature map v of the n-th layer i Perform feature fusion with the text features respectively to obtain n layers of fusion feature maps; The category of the detection target is predicted using the text features and the n-layer fused feature map to obtain category information of the detection target in the image to be detected; and / or, the position of the detection target is predicted using the n-layer fused feature map to obtain position information of the detection target in the image to be detected.

4. The defect detection method according to claim 3, characterized in that: The first feature map v of the n-th layer i The features are respectively fused with the text features to obtain n layers of fused feature maps, including: The n-layer text attention network is used to convert the n-layer first feature map v i Perform feature fusion with the text features respectively to obtain n layers of fusion feature maps; Wherein, for the text attention network of the nth layer, the first feature map of the nth layer is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as the fusion feature map of the nth layer; For the k-th layer text attention network, a first convolution operation and an upsampling operation are performed on the k+1-th layer fusion feature map to obtain a corresponding upsampled feature map, wherein the resolution of the upsampled feature map is the same as the resolution of the k-th layer first feature map; the k-th layer first feature map and the corresponding upsampled feature map are added element by element to obtain an element-by-element addition feature map; the element-by-element addition feature map is used as the first input feature map, the text feature is used as the second input feature map, the first input feature map and the second input feature map are input into the text attention network to obtain an output feature map, and the obtained output feature map is used as the k-th layer fusion feature map, wherein k=n-1,…,1; wherein the text attention network is used to enhance the first feature map v i The location information of the detected target.

5. The defect detection method according to claim 4, characterized in that: The step of inputting the first input feature map and the second input feature map into the text attention network to obtain an output feature map includes: The first input feature map d and the second input feature map e input into the text attention network are used to calculate the first text and region similarity matrix through the following expression: Wherein, the Conv(d) represents the second convolution operation performed on the first input feature map d to obtain the second convolution feature map, the Norm(e) represents the first normalization process performed on the second input feature map e to obtain the first normalized feature map, the μ and the δ are respectively the first learnable scaling factor and the first learnable translation factor, the Represents the operation of matrix multiplication; Performing a maximum value operation on the first text and region similarity matrix s1 to obtain the first text and region most similarity matrix; Performing a first nonlinear activation process on the first text and the region most similarity matrix to obtain a text-region attention map; Performing an element-by-element multiplication operation on the text-region attention map and the first input feature map d input into the text attention network to obtain an attention enhancement feature map; A third convolution operation is performed on the attention enhancement feature map to obtain a corresponding fused feature map, and the corresponding fused feature map is used as the output feature map of the text attention network.

6. The defect detection method according to claim 3, characterized in that: The method of using the n-layer fusion feature map to predict the position of the detection target to obtain the position information of the detection target in the image to be detected includes: A fourth convolution operation is performed on each of the n layers of fused feature maps to output a prediction box corresponding to the fused feature map; wherein the prediction box is used to represent the position information of the detection target.

7. The defect detection method according to claim 3, characterized in that: The method of using the text feature and the n-layer fused feature map to predict the category of the detection target to obtain the category information of the detection target in the fused feature map respectively includes: The fusion feature map f of the jth layer is calculated by using the text feature, the fusion feature map and the following expression: j The corresponding second text and region similarity matrix: Wherein, the Norm(e) represents a feature map obtained by performing a second normalization process on the text feature, the α and the β are respectively a learnable second scaling factor and a learnable second translation factor, and the Represents the operation of matrix multiplication; The fusion feature map f of the jth layer is calculated by using the second text and region similarity matrix and the following expression: j The corresponding category probability feature map: p j =sigmoid(s2); Wherein, the sigmoid represents the second nonlinear activation processing, and the fusion feature map f with the jth layer j The corresponding category probability feature map pj is used to represent the fusion feature map f of the jth layer j The probability that each pixel in belongs to the detected target of the category.

8. The defect detection method according to claim 3, characterized in that: The defect detection method is implemented based on a defect detection model, and the defect detection model includes a data processing module; The method of using the structured text as a guide to predict the category and / or position of the detection target of interest in the image to be detected x, and obtaining the category information and / or position information of the detection target of interest, includes: The data processing module uses the structured text as a guide to predict the category and / or position of the detection target of interest in the image x to be detected, and obtains the category information and / or position information of the detection target of interest.

9. The defect detection method according to claim 8, characterized in that: The data processing module includes a text encoding submodule, an image encoding submodule, and a feature fusion submodule, and the data processing module also includes a positioning submodule and / or a classification submodule; The step of performing text encoding on the structured text to obtain text features comprises: performing text encoding on the structured text using the text encoding submodule to obtain text features; Wherein, the multi-scale feature extraction is performed on the image to be detected to obtain the n-layer first feature map v i , including: using the image encoding submodule to extract multi-scale features of the image to be detected to obtain an n-layer first feature map v i ; Wherein, the first feature map v of the n-th layer is i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j , including: using the feature fusion submodule to transform the n-layer first feature map v i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j ; Wherein, the use of the text features and the n-layer fusion feature map f j Predicting the category of the detection target to obtain category information of the detection target in the image to be detected includes: The classification submodule is used to classify the text features and the n-layer fusion feature map f j Processing to predict the category of the detection target and obtain category information of the detection target in the image to be detected; Wherein, the use of the n-layer fusion feature map f j Predicting the position of the detection target to obtain the position information of the detection target in the image to be detected includes: The positioning submodule is used to perform the n-layer fusion feature map f j Processing is performed to predict the position of the detection target and obtain the position information of the detection target in the image to be detected.

10. The defect detection method according to claim 9, characterized in that: The defect detection model also includes a text structuring module; The converting the preset prompt text into structured text comprises: The text structuring module is used to perform structural processing on the preset prompt text to obtain the structured text.

11. The defect detection method according to claim 10, characterized in that: The defect detection model is trained through the following training process: Acquire a training sample image, and preset training prompt text and annotation data corresponding to the training sample image, wherein the annotation data includes a real target frame of a training detection target of interest in the training sample image; wherein the real target frame corresponds to the preset training prompt text; wherein the preset training prompt text is a description of a category of the training detection target of interest in the training sample image; The text structuring module converts the preset training prompt text into a training structured text; The data processing module uses the training structured text as a guide to predict the category and / or position of the detection target of interest in the training sample image, and obtains a training category probability feature map and / or a training prediction frame of the training detection target; wherein the training category probability feature map is used to characterize the predicted probability that each pixel in the training sample image belongs to the category of the training detection target; and the training prediction frame is used to characterize the position information of the training detection target; The defect detection model is trained according to the total loss function to obtain the trained defect detection model; wherein the total loss function is determined by a first loss function and a second loss function; the first loss function is determined according to the training prediction box and the real box corresponding to the training prediction box; the second loss function is determined according to the real category label value of the category of the training detection target and the predicted probability, and the real category label value is obtained based on the preset training prompt text.

12. The defect detection method according to claim 11, characterized in that: The data processing module uses the training structured text as a guide to predict the category and / or position of the detection target of interest in the training sample image, and obtains a training category probability feature map and / or a training prediction box of the training detection target, including: Using the text encoding submodule to perform text encoding on the training structured text to obtain training text features; The image encoding submodule is used to perform multi-scale feature extraction on the training sample image to obtain an n-layer first training feature map v i ; The feature fusion submodule is used to transform the first training feature map v of the n-layer i The n-layer training fusion feature graph f is obtained by performing feature fusion with the training text features respectively. j ; Wherein, the use of the training text features and the n-layer training fusion feature map f j Predicting the category of the training detection target to obtain category information of the training detection target in the training sample image includes: The classification submodule uses the training text features and the n-layer training fusion feature map f j Predicting the category of the training detection target to obtain category information of the training detection target in the training sample image; Wherein, the use of the n-layer training fusion feature map f j Predicting the position of the training detection target to obtain the position information of the training detection target in the training sample image includes: The positioning submodule uses the n-layer training fusion feature map f j Predicting the position of the training detection target to obtain the position information of the training detection target in the training sample image; The text encoding submodule is obtained through pre-training, and in the above training process, the weight parameters of the text encoding submodule are not updated.

13. The defect detection method according to claim 12, characterized in that: The feature fusion submodule includes an n-layer text attention network; Wherein, the feature fusion submodule is used to transform the n-layer first training feature map v i The n-layer training fusion feature graph f is obtained by performing feature fusion with the training text features respectively. j ,include: The n-layer text attention network is used to train the first training feature map v i Perform feature fusion with the training text features respectively to obtain n layers of training fusion feature graphs; For the text attention network of the nth layer, the first training feature map of the nth layer is used as the first training input feature map, the training text feature is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain a training output feature map, and the obtained training output feature map is used as the training fusion feature map of the nth layer; For the k-th layer text attention network, a first convolution operation and an upsampling operation are performed on the k+1-th layer training fusion feature map to obtain a corresponding training upsampling feature map, wherein the resolution of the training upsampling feature map is the same as the resolution of the first training feature map of the k-th layer; an element-by-element addition operation is performed on the first training feature map of the k-th layer and the corresponding training upsampling feature map to obtain a training element-by-element addition feature map; the training element-by-element addition feature map is used as the first training input feature map, the training text feature is used as the second training input feature map, the first training input feature map and the second training input feature map are input into the text attention network to obtain a training output feature map, and the obtained training output feature map is used as the k-th layer training fusion feature map; wherein k=n-1,…,1.

14. The defect detection method according to claim 12, characterized in that: The image encoding submodule is obtained by performing low-rank fine-tuning on the pre-trained multi-scale feature extraction submodel; wherein the low-rank fine-tuning process includes: For each convolution layer in the pre-trained multi-scale feature extraction sub-model, a low-rank parameter update branch corresponding to the convolution layer is constructed to form a new multi-scale feature extraction sub-model; the new multi-scale feature extraction sub-model is used as the feature extraction sub-model; wherein the expression of the weight parameter of each convolution layer in the new multi-scale feature extraction sub-model is: W = W0 + ΔW = W0 + BA; Wherein, W0 represents the weight parameter of each convolution layer in the pre-trained multi-scale feature extraction sub-model, ΔW represents the weight parameter of the low-rank parameter update branch corresponding to the convolution layer, and B and A represent trainable parameters; wherein, in the above training process, the weight parameters of all convolution layers in the pre-trained multi-scale feature extraction sub-model are not updated, while the weight parameters of the low-rank parameter update branch are updated.

15. The defect detection method according to claim 11, characterized in that: The total loss function of the above defect detection model or defect detection network is expressed as: Loss=δ1L r +δ2L c ; Among them, the L r represents the first loss function, the L c represents the second loss function; δ1 and δ2 are respectively the preset weight coefficient of the first loss function and the preset weight coefficient of the second loss function.

16. The defect detection method according to claim 15, characterized in that: The expression of the first loss function is: Among them, the p r represents the training prediction box, the t r represents the real box corresponding to the training prediction box, Indicates that the p r The center point of Indicates the t r The center point of Indicates the and stated The Euclidean distance between the two, η represents the distance between the two surrounding p r and the r The length of the diagonal of the minimum enclosing rectangle, and the IoU represents the operation of calculating the intersection over union.

17. The defect detection method according to claim 16, characterized in that: The expression of the second loss function is: Among them, the Represents the training fusion feature map f of the jth layer j The loss of the training category probability feature map of the training detection target in the training detection target, wherein N represents the total number of layers of the training category probability feature map, and represents the loss of the kth pixel in the corresponding training category probability feature map, and K represents the total number of pixels in the training category probability feature map; wherein, The expression is: The S t represents the predicted probability / confidence of the category of the training detection target in the k-th pixel, ∑S u Represents the sum of the predicted probabilities / confidences of the categories of all non-training detection targets in the k-th pixel.

18. A defect detection device, characterized in that: include: A camera is used to capture images of the industrial products to be inspected and form images to be inspected; A processor is connected to the camera and is used to process the image to be detected according to the defect detection method described in any one of claims 1 to 8 to obtain the position information and / or category information of the detection target of interest in the image to be detected.

19. The defect detection device according to claim 18, characterized in that: The processor comprises: An image acquisition and prompt module, used to acquire the image to be detected and the corresponding preset prompt text; wherein the preset prompt text is a description of the category of the detection target; The data processing module is used to predict the category and / or position of the detection target of interest in the image x to be detected by taking the structured text as a guide, so as to obtain the category information and / or position information of the detection target of interest.

20. The defect detection device according to claim 19, characterized in that: The processor further comprises: The text structuring module is used to perform structural processing on the preset prompt text to obtain the structured text; wherein the structured text has a fixed format.

21. The defect detection device according to claim 19 or 20, characterized in that: The data processing module includes: a text encoding submodule, an image encoding submodule, and a feature fusion submodule; The data processing module further includes: a positioning submodule and / or a classification submodule; Wherein, the text encoding submodule is used to perform text encoding on the structured text to obtain text features; The image encoding submodule is used to extract multi-scale features of the image to be detected to obtain an n-layer first feature map v i ; Where i = 1, ..., n and the larger the value of i, the corresponding first feature map v i The smaller the resolution, the n≥2; The feature fusion submodule is used to combine the first feature map v of the n-th layer i The n-layer fusion feature graph f is obtained by performing feature fusion with the text features respectively. j ; The classification submodule is used to utilize the text features and the n-layer fusion feature map f j Predicting the category of the detection target to obtain category information of the detection target in the image to be detected; The positioning submodule is used to utilize the n-layer fusion feature map f j The position of the detection target is predicted to obtain the position information of the detection target.

22. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the defect detection method according to any one of claims 1 to 17 is implemented.

23. A computer-readable storage medium, characterized in that: The method comprises a program which can be executed by a processor to implement the defect detection method according to any one of claims 1 to 17.

Images