Industrial defect detection method based on vision-language prompt

Abstract

The invention relates to the technical field of industrial scene defect detection, and particularly provides an industrial defect detection method based on vision-language prompt. The method comprises the steps of building a feature extraction network of a target detection model; building a vision-language prompt module according to the feature extraction network; building a vision-language detection module, and connecting the feature extraction network with a vision-language prompt module to construct an industrial defect detection model; according to the method, the constructed industrial defect detection model is trained, the trained industrial defect detection model is packaged and deployed, and the defect detection result is obtained through the trained industrial defect detection model, so that the industrial defects are accurately distinguished, the identification and positioning of the industrial defect target are accelerated, and the detection efficiency is improved. Therefore, the accuracy of industrial defect image detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial scene defect detection, and in particular to an industrial defect detection method based on visual-language prompts, which is used for industrial defect detection. Background Art

[0002] Product defect detection tasks in industrial scenarios play an important role in protecting the personal and property safety of users. Although deep learning methods have achieved great success in defect detection tasks in industrial scenarios, expert models in industrial scenarios usually use unimodal or multimodal data from a single field and strictly follow category-aware detection methods, which limits the model's ability to process multi-scene data and generalize to open datasets. Recently, the Segment Everything model has achieved great success in the field of natural scene images. The Segment Everything model not only has powerful interactive capabilities, but also shows strong zero-shot capabilities in remote sensing, medicine and other fields. The uniqueness of the Segment Everything model lies in the design of human-computer interactive prompts, allowing segmentation based on user-provided point, line, and box prompts.

[0003] However, since images in industrial defect scenes are significantly different from natural scenes in terms of imaging conditions, environment, scale, and distribution, there are two major challenges in applying the pre-training-cueing paradigm of the Segment Everything model to industrial defect scenes. The existing Segment Everything model cueing paradigm relies on user-provided cues (points, boxes, coarse masks) to segment objects in the target image when dealing with complex scenes and a large number of images. The user's familiarity with the target object significantly affects the effect of providing specific cues. In addition, existing methods use coarse visual cues, such as colored boxes or circles, which often lead to suboptimal performance due to the inclusion of irrelevant and noisy pixels. To address the above issues, some methods combine semantic association models to obtain pseudo-masks of the target. CPT and ReCLIP use visual cueing techniques to establish relationships between part instances. CoCoOp further turns image-generated cues into conditional inputs and dynamically combines them with language cues. These methods ignore false positives in pseudo-masks and show high sensitivity to hyperparameters. Therefore, it heavily relies on the quality of pseudo-masks and has poor generalization ability. Summary of the invention

[0004] In view of this, the present invention provides an industrial defect detection method based on visual-language cues to accurately distinguish industrial defects, accelerate the recognition and positioning of industrial defect targets, and thus improve the accuracy of industrial defect image detection.

[0005] In a first aspect, the present invention provides an industrial defect detection method based on visual-linguistic cues, the method comprising:

[0006] Step S1, building a feature extraction network for a target detection model;

[0007] Step S2, building a visual-language prompt module based on the feature extraction network in step S1;

[0008] Step S3, building a visual-language detection module, and connecting the feature extraction network in step S1 and the visual-language prompt module in step S2 to build an industrial defect detection model;

[0009] Step S4: train the industrial defect detection model constructed in step S3, package and deploy the trained industrial defect detection model, and obtain defect detection results through the trained industrial defect detection model.

[0010] Optionally, the feature extraction network is composed of a segmentation model, a convolution expert model, a language encoder and a mask domain adaptation module; the segmentation model branch, the convolution expert model branch and the language encoder branch extract features in parallel;

[0011] The segmentation model branch includes an encoder structure, wherein the encoder structure is divided into a first attention module, a second attention module, a third attention module and a fourth attention module;

[0012] The convolution expert model branch includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module.

[0013] Optionally, the structure of the mask field adaptation module includes:

[0014] (1) Align the feature maps in the first attention module, the second attention module, the third attention module, and the fourth attention module of the split model branch with the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch. The formula is as follows:

[0015] L = γ × D (L 1 (F i ), L 2 (S j ));

[0016] Among them, L 1 , L 2 represents the mapping function, F i , i∈{1, 2, 3, 4} represents the feature maps in the first attention module, the second attention module, the third attention module, and the fourth attention module that split all large model branches; S j, j∈{1, 2, 3, 4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch; D represents the alignment loss function, which is used to penalize different F i and S j The distance between them, γ is the weight to balance the two losses; the formula of D is:

[0017]

[0018] (2) Divide the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all large model branches into N groups according to the feature block size of 32 to obtain the neighborhood multi-scale features where N∈2 k , k∈N + , i∈{1, 2, 3, 4};

[0019] (3) Mask some feature blocks uniformly according to the random sampling method. The two-dimensional index of each feature block is defined as Randomly sample M indices to get a binary mask, and i Multiply to get the multi-scale features of the mask The formula is:

[0020]

[0021]

[0022] in, is the range of M indexes extracted; is a binary mask, and P is the operation of segmenting the image into image blocks;

[0023] (4) Obtaining multi-scale features of the mask Finally, a mask decoder is built to reconstruct the masked features, and the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all the branches of the segmentation model are obtained. Its mask decoder formula is:

[0024]

[0025] (5) Introduce a multi-scale domain optimization function to additionally optimize the domain adaptation layer. When the optimization function is infinitely small, the optimization of the domain adaptation layer will stop. The optimization function F target The formula is:

[0026]

[0027] Among them, n = 4 represents the number of different multi-scale features, Sj , j∈{1, 2, 3, 4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch.

[0028] Optionally, the visual-language prompt module in step S2 is composed of a refined visual prompt module and a visual-language interaction module.

[0029] Optionally, the structure of the refined visual prompt module includes:

[0030] (6) The feature maps of the first attention module, the second attention module, the third attention module, and the fourth attention module of all the large model branches after segmentation are reconstructed. Each pixel is scored by the energy evaluation function to obtain the uncertainty score M of each pixel position. k , the formula is:

[0031]

[0032] Where k is a normalization constant, N i is the set of neighboring pixels of the ith pixel, is the similarity measure between the i-th pixel and the j-th pixel;

[0033] (7) Image block segmentation with different neighborhood sizes M k ; For the position of a single pixel in the feature space, the set of neighboring pixels It is expressed as:

[0034]

[0035] Where (h′, w′) is the relative position of the pixel;

[0036] (8) Given a set of segmented neighboring pixels, a sparse selection mechanism is used to select high-frequency pixels. Sparse sample selection is completed by selecting pixels whose internal pixel values ​​of the top-k patches are greater than the mean. The formula is:

[0037]

[0038] in, express Any pixel in represents the pixel-level mean in the neighborhood space, Represents the selected sparse samples;

[0039] (9) Introduce an IoU-based optimization mechanism to iteratively optimize the activation mechanism of the feature; the goal of the optimization mechanism is to make the selected high-frequency pixels fall within the area of ​​the real frame; the optimization mechanism uses an additional detection head to Perform regression prediction and optimize the effect of feature activation through CIoU.

[0040] Optionally, the structure of the visual language interaction module includes:

[0041] (11) The feature maps in the first attention module, the second attention module, the third attention module and the fourth attention module of all the large model branches after segmentation are reconstructed. and After multiplication, the feature map S in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch j Add together to get the refined visual feature F L , the formula is:

[0042]

[0043] (12) By embedding the text feature T and image feature F generated by the text encoder L ∈R CxHxW (L∈{1, 2, 3, 4}), use the maximum Sigmoid attention to query the semantic features of text-image matching; aggregate text features into image features, the formula is:

[0044]

[0045] Among them, F image-text represents the refined visual-linguistic features, σ represents the activation function Sigmoid, Represents the transposed matrix of text embeddings containing multiple keywords.

[0046] Optionally, the structure of the vision-language detection module in step S3 includes:

[0047] (15) Output predictions of N object positions through regression, use position regression to locate the position of industrial defect objects, use task-aligned truth label assignment to select the correct object position prediction, and use each selected object position prediction as a positive sample to match the text as a classification label;

[0048] (16) On the text feature embedding T, the region-text contrast loss is calculated by the similarity between the object region and the text T and the cross entropy between the object and the text; the IOU loss and the distributed focal loss are used to optimize the prediction of the object position; when the input image contains a true box, and are calculated, otherwise they are not calculated.

[0049] In a second aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the industrial defect detection method based on visual-language cues in the first aspect or any possible implementation of the first aspect.

[0050] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to execute the industrial defect detection method based on visual-language cues in the first aspect or any possible implementation of the first aspect.

[0051] In the technical solution provided by the present invention, the method includes building a feature extraction network of a target detection model; building a visual-language prompt module based on the feature extraction network; building a visual-language detection module, and connecting the feature extraction network and the visual-language prompt module to build an industrial defect detection model; training the constructed industrial defect detection model, encapsulating and deploying the trained industrial defect detection model, and obtaining defect detection results through the trained industrial defect detection model. The method accurately distinguishes industrial defects, accelerates the recognition and positioning of industrial defect targets, and thus improves the accuracy of industrial defect image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A flow chart of an industrial defect detection method provided by an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of an industrial defect detection model provided by an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] It should be clear that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings.

[0059] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0060] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0061] Figure 1 A flowchart of an industrial defect detection method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0062] Step S1: Build a feature extraction network for the target detection model.

[0063] In the embodiment of the present invention, the input image and language are as follows: Figure 2 As shown in FIG. 1 , the feature extraction network is composed of a segmentation model, a convolutional expert model, a language encoder, and a mask domain adaptation module; the segmentation model branch, the convolutional expert model branch, and the language encoder branch extract features in parallel;

[0064] The segmentation model branch includes an encoder structure, wherein the encoder structure is divided into a first attention module, a second attention module, a third attention module and a fourth attention module;

[0065] The convolution expert model branch includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module.

[0066] In the embodiment of the present invention, the structure of the mask field adaptation module includes:

[0067] (1) Align the feature maps in the first attention module, the second attention module, the third attention module, and the fourth attention module of the split model branch with the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch. The formula is as follows:

[0068] L = γ × D (L 1 (F i ), L 2 (S j ));

[0069] Among them, L 1 , L 2 represents the mapping function, F i , i∈{1, 2, 3, 4} represents the feature maps in the first attention module, the second attention module, the third attention module, and the fourth attention module that split all large model branches; S j , j∈{1, 2, 3, 4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch; D represents the alignment loss function, which is used to penalize different F i and S j The distance between them, γ is the weight to balance the two losses; the formula of D is:

[0070]

[0071] (2) Divide the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all large model branches into N groups according to the feature block size of 32 to obtain the neighborhood multi-scale features where N∈2 k , k∈N + , i∈{1, 2, 3, 4};

[0072] (3) Mask some feature blocks uniformly according to the random sampling method. The two-dimensional index of each feature block is defined as Randomly sample M indices to get a binary mask, and iMultiply to get the multi-scale features of the mask The formula is:

[0073]

[0074]

[0075] in, is the range of M indexes extracted; is a binary mask, and P is the operation of segmenting the image into image blocks;

[0076] (4) Obtaining multi-scale features of the mask Finally, a mask decoder is built to reconstruct the masked features, and the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all the branches of the segmentation model are obtained. Its mask decoder formula is:

[0077]

[0078] (5) Introduce a multi-scale domain optimization function to additionally optimize the domain adaptation layer. When the optimization function is infinitely small, the optimization of the domain adaptation layer will stop. The optimization function F target The formula is:

[0079]

[0080] Among them, n = 4 represents the number of different multi-scale features, S i , j∈{1, 2, 3, 4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch.

[0081] Step S2: Building a visual-language prompt module based on the feature extraction network in step S1.

[0082] In the embodiment of the present invention, the visual-language prompt module in step S2 is composed of a detailed visual prompt module and a visual-language interaction module.

[0083] In the embodiment of the present invention, the structure of the refined visual prompt module includes:

[0084] (6) The feature maps of the first attention module, the second attention module, the third attention module, and the fourth attention module of all the large model branches after segmentation are reconstructed. Each pixel is scored by the energy evaluation function to obtain the uncertainty score M of each pixel position. k , the formula is:

[0085]

[0086] Where k is a normalization constant, N i is the set of neighboring pixels of the ith pixel, is the similarity measure between the i-th pixel and the j-th pixel;

[0087] (7) Image block segmentation with different neighborhood sizes M k ; For the position of a single pixel in the feature space, the set of neighboring pixels It is expressed as:

[0088]

[0089] Where (h′, w′) is the relative position of the pixel;

[0090] (8) Given a set of segmented neighboring pixels, a sparse selection mechanism is used to select high-frequency pixels. Sparse sample selection is completed by selecting pixels whose internal pixel values ​​of the top-k patches are greater than the mean. The formula is:

[0091]

[0092] in, express Any pixel in represents the pixel-level mean in the neighborhood space, Represents the selected sparse samples;

[0093] (9) Introduce an IoU-based optimization mechanism to iteratively optimize the activation mechanism of the feature; the goal of the optimization mechanism is to make the selected high-frequency pixels fall within the area of ​​the real frame; the optimization mechanism uses an additional detection head to Perform regression prediction and optimize the effect of feature activation through CIoU.

[0094] In the embodiment of the present invention, the structure of the visual language interaction module includes:

[0095] (11) The feature maps in the first attention module, the second attention module, the third attention module and the fourth attention module of all the large model branches after segmentation are reconstructed. and After multiplication, the feature map S in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch j Add together to get the refined visual feature F L , the formula is:

[0096]

[0097] (12) By embedding the text feature T and image feature F generated by the text encoder L ∈R CxHxW (L∈{1, 2, 3, 4}), use the maximum Sigmoid attention to query the semantic features of text-image matching; aggregate text features into image features, the formula is:

[0098]

[0099] Among them, F image-text represents the refined visual-linguistic features, σ represents the activation function Sigmoid, Represents the transposed matrix of text embeddings containing multiple keywords.

[0100] Step S3: build a visual-language detection module, and connect the feature extraction network in step S1 and the visual-language prompt module in step S2 to build an industrial defect detection model.

[0101] In the embodiment of the present invention, the structure of the vision-language detection module in step S3 includes:

[0102] (15) Output predictions of N object positions through regression, use position regression to locate the position of industrial defect objects, use task-aligned truth label assignment to select the correct object position prediction, and use each selected object position prediction as a positive sample to match the text as a classification label;

[0103] (16) On the text feature embedding T, the region-text contrast loss is calculated by the similarity between the object region and the text T and the cross entropy between the object and the text; the IOU loss and the distributed focal loss are used to optimize the prediction of the object position; when the input image contains a true box, and are calculated, otherwise they are not calculated.

[0104] Step S4: train the industrial defect detection model constructed in step S3, package and deploy the trained industrial defect detection model, and obtain defect detection results through the trained industrial defect detection model.

[0105] In the technical solution provided by the present invention, the method includes building a feature extraction network of a target detection model; building a visual-language prompt module based on the feature extraction network; building a visual-language detection module, and connecting the feature extraction network and the visual-language prompt module to build an industrial defect detection model; training the constructed industrial defect detection model, encapsulating and deploying the trained industrial defect detection model, and obtaining defect detection results through the trained industrial defect detection model. The method accurately distinguishes industrial defects, accelerates the recognition and positioning of industrial defect targets, and thus improves the accuracy of industrial defect image detection.

[0106] Each step of the embodiment of the present invention may be performed by an electronic device, which includes but is not limited to a mobile phone, a tablet computer, a portable PC, a desktop computer, etc.

[0107] An embodiment of the present invention provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the electronic device where the computer-readable storage medium is located is controlled to execute the above-mentioned embodiment of the industrial defect detection method based on visual-language prompts.

[0108] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 21 includes: a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, the industrial defect detection method based on visual-language cues in the embodiment is implemented. To avoid repetition, they are not described one by one here.

[0109] The electronic device 21 includes, but is not limited to, a processor 211 and a memory 212. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 21 and does not constitute a limitation of the electronic device 21. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0110] The processor 211 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0111] The memory 212 may be an internal storage unit of the electronic device 21, such as a hard disk or memory of the electronic device 21. The memory 212 may also be an external storage device of the electronic device 21, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 21. Further, the memory 212 may also include both an internal storage unit of the electronic device 21 and an external storage device. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 may also be used to temporarily store data that has been output or is to be output.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for industrial defect detection based on visual-linguistic cues, characterized in that: The method comprises: Step S1, building a feature extraction network for a target detection model; Step S2, building a visual-language prompt module based on the feature extraction network in step S1; Step S3, building a visual-language detection module, and connecting the feature extraction network in step S1 and the visual-language prompt module in step S2 to build an industrial defect detection model; Step S4: train the industrial defect detection model constructed in step S3, package and deploy the trained industrial defect detection model, and obtain defect detection results through the trained industrial defect detection model.

2. The method according to claim 1, characterized in that The feature extraction network is composed of a segmentation model, a convolution expert model, a language encoder and a mask domain adaptation module; the segmentation model branch, the convolution expert model branch and the language encoder branch extract features in parallel; The segmentation model branch includes an encoder structure, wherein the encoder structure is divided into a first attention module, a second attention module, a third attention module and a fourth attention module; The convolution expert model branch includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module.

3. The method according to claim 2, characterized in that The structure of the mask domain adaptation module includes: (1) Align the feature maps in the first attention module, the second attention module, the third attention module, and the fourth attention module of the split model branch with the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch. The formula is as follows: L=γ×D(L1(F i ),L2(S j )); Among them, L1 and L2 represent mapping functions, F i , i∈{1,2,3,4} represents the feature maps in the first attention module, the second attention module, the third attention module and the fourth attention module that split all the large model branches; S j , j∈{1,2,3,4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module, and the fourth convolution module of the convolution expert model branch; D represents the alignment loss function, which is used to penalize different F i and S j The distance between them, γ is the weight to balance the two losses; the formula of D is: (2) Divide the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all large model branches into N groups according to the feature block size of 32 to obtain the neighborhood multi-scale features where N∈2 k ,k∈N + ,i∈{1,2,3,4}; (3) Mask some feature blocks uniformly according to the random sampling method. The two-dimensional index of each feature block is defined as Randomly sample M indices to get a binary mask, and i Multiply to get the multi-scale features of the mask The formula is: in, is the range of M indexes extracted; is a binary mask, and P is the operation of segmenting the image into image blocks; (4) Obtaining multi-scale features of the mask Finally, a mask decoder is built to reconstruct the masked features, and the features in the first attention module, the second attention module, the third attention module, and the fourth attention module of all the branches of the segmentation model are obtained. Its mask decoder formula is: (5) Introduce a multi-scale domain optimization function to additionally optimize the domain adaptation layer. When the optimization function is infinitely small, the optimization of the domain adaptation layer will stop. The optimization function F target The formula is: Among them, n = 4 represents the number of different multi-scale features, S j ,j∈{1,2,3,4} represents the feature maps in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch.

4. The method according to claim 1, characterized in that: The visual-language prompt module in step S2 is composed of a refined visual prompt module and a visual-language interaction module.

5. The method according to claim 4, characterized in that The structure of the refined visual prompt module includes: (6) The feature maps of the first attention module, the second attention module, the third attention module, and the fourth attention module of all the large model branches after segmentation are reconstructed. Each pixel is scored by the energy evaluation function to obtain the uncertainty score M of each pixel position. k , the formula is: Where k is a normalization constant, N i is the set of neighboring pixels of the ith pixel, is the similarity measure between the i-th pixel and the j-th pixel; (7) Image block segmentation with different neighborhood sizes M k ; For the position of a single pixel in the feature space, the set of neighboring pixels It is expressed as: Among them, (h′, w′) is the relative position of the pixel; (8) Given a set of segmented neighboring pixels, a sparse selection mechanism is used to select high-frequency pixels. Sparse sample selection is completed by selecting pixels whose internal pixel values ​​of the top-k patches are greater than the mean. The formula is: in, express Any pixel in represents the pixel-level mean in the neighborhood space, represents the selected sparse samples; (9) Introduce an IoU-based optimization mechanism to iteratively optimize the activation mechanism of the feature; the goal of the optimization mechanism is to make the selected high-frequency pixels fall within the area of ​​the real frame; the optimization mechanism uses an additional detection head to Perform regression prediction and optimize the effect of feature activation through CIoU.

6. The method according to claim 4, characterized in that The structure of the visual language interaction module includes: (11) The feature maps in the first attention module, the second attention module, the third attention module and the fourth attention module of all the large model branches after segmentation are reconstructed. and After multiplication, the feature map S in the first convolution module, the second convolution module, the third convolution module and the fourth convolution module of the convolution expert model branch j Add together to get the refined visual feature F L , the formula is: (12) By embedding the text feature T and image feature F generated by the text encoder L ∈R CxHxW (L∈{1,2,3,4}), use the maximum Sigmoid attention to query the semantic features of text-image matching; aggregate text features into image features, the formula is: Among them, F image-text represents the refined visual-linguistic features, σ represents the activation function Sigmoid, Represents the transposed matrix of text embeddings containing multiple keywords.

7. The method according to claim 1, characterized in that The structure of the vision-language detection module in step S3 includes: (15) Output predictions of N object positions through regression, use position regression to locate the position of industrial defect objects, use task-aligned truth label assignment to select the correct object position prediction, and use each selected object position prediction as a positive sample to match the text as a classification label; (16) On the text feature embedding T, the region-text contrast loss is calculated by the similarity between the object region and the text T and the cross entropy between the object and the text; the IOU loss and the distributed focal loss are used to optimize the prediction of the object position; when the input image contains a true box, and are calculated, otherwise they are not calculated.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is run, the computer-readable storage medium is controlled to execute the industrial defect detection method based on visual-language prompts as described in any one of claims 1 to 7.

9. An electronic device, characterized in that: include: one or more processors; Memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the device, enable the device to perform the industrial defect detection method based on visual-language cues as described in any one of claims 1 to 7.

Images