Defect detection method and system based on language prompt and collaborative teaching

Abstract

The invention relates to the field of image processing, and provides a defect detection method and system based on language prompt and collaborative teaching, which avoids the problem of insufficient sample data in actual production through data enhancement and expansion of normal sample image data and defect sample image data, and further improves the detection accuracy by introducing a learnable language prompt text. The detection capability of different defects of different types of products is enhanced, more and more accurate local detail information is obtained through a local feature attention enhancement mechanism to enhance the overall detection accuracy, and more comprehensive defect detection results are obtained through prediction of two independent defect detection results. Meanwhile, the two defect detection modules mutually prompt and update, so that the overall detection is more in line with the actual application condition, the universality of the detection method is improved, and the accuracy and universality of the defect detection method are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a defect detection method and system based on language prompting and collaborative teaching. Background Art

[0002] With the rapid development of industrial production, the requirements for industrial manufacturing are gradually increasing. In particular, there are widespread problems of industrial product defects in industrial production, which may cause a variety of safety accidents. Therefore, detecting whether there are defects in industrial products in industrial production is an important part of industrial production.

[0003] In the existing technology, professional quality inspectors are often required to manually detect defects and make subjective judgments on the surface of industrial products. This is not only costly, but also has limited detection efficiency. Therefore, computer vision technology has become a key detection method. Surface defect detection of industrial products is performed through deep learning image anomaly detection methods. However, existing image detection methods often require a large amount of labeled sample data. However, due to the large number of product defect types in industrial production, it is difficult to obtain a large amount of labeled sample data, and the detection accuracy is insufficient. At the same time, the detection situation in actual production applications is complex and changeable, resulting in the prediction results not being consistent with the actual production situation.

[0004] Therefore, how to design a defect detection method to adapt to the changes in actual production applications and improve the accuracy of defect detection has become an urgent problem to be solved. Summary of the invention

[0005] Based on this, the present invention provides a defect detection method and system based on language prompts and collaborative teaching, which expands normal sample image data and defect sample image data through data enhancement, thereby avoiding the problem of insufficient sample data in actual production, and then introduces learnable language prompt text to enhance the detection ability of different defects of different types of products, and obtains more and more accurate local detail information through the local feature attention enhancement mechanism to enhance the overall detection accuracy, and obtains more comprehensive defect detection results through two independent defect detection result predictions. At the same time, the two defect detection modules prompt and update each other, so that the overall detection is more in line with actual application conditions, thereby improving the versatility of the detection method. The present invention improves the accuracy and versatility of the defect detection method.

[0006] The present invention proposes a defect detection method based on language prompts and collaborative teaching, comprising: Collecting the image data of the target to be tested in real time and performing data enhancement processing, wherein the data enhancement processing is used to expand and obtain the normal sample image data and defect sample image data of the target to be tested; Acquire a language prompt text according to the category of the target image data to be tested, and then acquire a language prompt text enhanced feature according to the language prompt, wherein the language prompt text is in a learnable state and is used to guide feature positioning and feature extraction; Performing attention enhancement processing on the target image data to be tested according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature, wherein the attention enhancement processing includes performing multimodal feature alignment according to the language prompt text enhancement feature; According to the local feature attention enhancement feature, the language prompt defect detection result and the basic image defect detection result are respectively obtained to calculate the final anomaly score and perform defect detection update.

[0007] In summary, according to the above-mentioned defect detection method based on language prompts and collaborative teaching, through data enhancement, the normal sample image data and the defective sample image data are expanded, so as to avoid the problem of insufficient sample data in actual production, and then by introducing learnable language prompt text, the detection ability of different defects of different types of products is enhanced, and more and more accurate local detail information is obtained through the local feature attention enhancement mechanism to enhance the overall detection accuracy, and through two independent defect detection result predictions, more comprehensive defect detection results are obtained. At the same time, the two defect detection modules prompt and update each other, so that the overall detection is more in line with the actual application situation, thereby improving the versatility of the detection method. The present invention improves the accuracy and versatility of the defect detection method. Specifically, the image data of the target to be tested is collected in real time and data enhancement processing is performed. The data enhancement processing is used to expand and obtain normal sample image data and defect sample image data of the target to be tested, thereby avoiding the problem of deviation in detection accuracy caused by insufficient sample data in actual use. A language prompt text is obtained according to the category of the image data of the target to be tested, and then a language prompt text enhancement feature is obtained according to the language prompt. The language prompt text is in a learnable state and is used to guide feature positioning and feature extraction. The image data of the target to be tested is subjected to attention enhancement processing according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature. The attention enhancement processing includes multimodal feature alignment according to the language prompt text enhancement feature, further enhancing the attention to local detail information, avoiding the influence of complex detection conditions with a wide variety of products and defects in the actual production process, and further improving the accuracy of the overall detection. According to the local feature attention enhancement feature, the language prompt defect detection result and the basic image defect detection result are respectively obtained to calculate the final abnormality score, and the defect detection update is performed, thereby further improving the versatility of the detection method to be more in line with the actual application scenario. The present invention improves the accuracy and versatility of the defect detection method.

[0008] Furthermore, the step of real-time acquisition of the target image data to be measured and performing data enhancement processing specifically includes: The data enhancement module collects the image data of the target to be tested in real time and then performs data enhancement; Acquire a plurality of normal sample image data according to random affine transformation and sharpening, and then simulate defects according to Perlin noise to fit the defects to the normal sample image data to acquire a plurality of defect sample image data; The specific algorithm of data enhancement is as follows: , in, represents defect sample image data, Represents a binary image obtained by randomly generating Perlin noise through a random threshold. express The inverse matrix of represents the input image, represents a random texture image, represents the random transparency parameter, Represents element-wise multiplication.

[0009] Furthermore, the step of obtaining a language prompt text according to the category of the target image data to be tested, and then obtaining a language prompt text enhancement feature according to the language prompt, specifically includes: The language prompt learning module obtains the language prompt text according to the category of the target image data to be tested. The language prompt text includes normal language prompt text and abnormal language prompt text. The specific algorithm for obtaining the language prompt text is as follows: , , in, Indicates normal language prompt text, Indicates abnormal language prompt text. Learnable word embeddings representing normal language prompt text, Learnable word embeddings to represent abnormal language hint text, Indicates the category of the target image data to be tested, Defective boot, Indicates the defect name; According to the language prompt text, the corresponding language prompt text enhancement feature is generated by feeding forward, and the specific algorithm of the feedforward step is as follows: in, Represents the input hint text of the hint layer. Represents the prompt layer, Indicates the prompt layer number, represents the maximum number of learnable hint layers, Indicates learnable hint text.

[0010] Furthermore, the step of performing attention enhancement processing on the target image data to be measured according to the local feature attention enhancement algorithm to obtain the local feature attention enhancement feature specifically includes: The image encoding module performs attention enhancement processing on the target image data to be measured according to the local feature attention enhancement algorithm, and the image encoding module includes multiple local feature attention enhancement mechanism layers; Extract local features. The specific algorithm for extracting local features is as follows: in, represents the original output features, represents the local feature attention enhancement feature, represents the image category feature, represents the last local feature, denote the query matrix, key matrix and value matrix respectively, express Attention projection, represents the local feature attention projection, represents the attention function.

[0011] Furthermore, the step of respectively obtaining the language prompt defect detection result and the basic image defect detection result according to the local feature attention enhancement feature specifically includes: The image encoding module selects different local feature attention enhancement mechanism layers according to a preset sequence number to obtain multiple intermediate image block-level features, performs multimodal feature alignment on the intermediate image block-level features and the language prompt text, and obtains multimodal alignment features; The language prompt defect detection result is obtained according to the language prompt text enhancement feature and the multimodal alignment feature. The specific algorithm for obtaining the language prompt defect detection result is as follows: , in, Indicates the language prompt defect detection results, represents the upsampling operation, Represents the cosine similarity calculation operation, represents the multimodal alignment feature, and They represent normal language prompt text enhancement features and abnormal language prompt text enhancement features respectively. represents the number of layers of the local feature attention enhancement mechanism, represents the total number of selected local feature attention enhancement mechanism layers, represents the exponential function; The defect detection module includes a first defect detection submodule and a second defect detection submodule, wherein the first defect detection submodule and the second defect detection submodule respectively obtain basic image defect detection results according to multimodal alignment features; The specific algorithm for obtaining the basic image defect detection result is as follows: , , in, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and They represent the first defect detection submodule and the second defect detection submodule respectively.

[0012] Furthermore, the step of calculating the final anomaly score specifically includes: The final abnormality score is calculated based on the language prompt defect detection result and the basic image defect detection result. The specific algorithm for calculating the final abnormality score is as follows: , in, represents the final anomaly score, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, Indicates language prompt defect detection results; Then the loss calculation is performed, and the specific algorithm of the loss calculation is as follows: , , , in, represents focal loss, represents dice loss, represents the total loss, Represents the total number of pixels, Represents the pixel ordinal number, represents the predicted probability, represents the adjustable parameter of weight, represents the output of the decoder, represents the true label, and Represents the loss balance coefficient.

[0013] Furthermore, the step of performing defect detection update specifically includes: Performing threshold filtering according to the language prompt defect detection result to obtain a normal image area, so as to update a first defect detection submodule and a second defect detection submodule of the defect detection module according to the normal image area; The specific algorithm of the update is as follows: , , , in, represents the pixel region coordinates, Represents pixel region labeling, Indicates the language prompt defect detection results, represents the threshold learned from the expanded defect sample image data, and represent the first defect detection submodule and the second defect detection submodule respectively, represents the learning rate, represents the calculation of gradient, represents the total loss; Performing collaborative teaching on a first defect detection submodule and a second defect detection submodule of the defect detection module; Performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the first defect detection submodule to update the second defect detection submodule, and performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the second defect detection submodule to update the first defect detection submodule, and the high-confidence image area used for updating in the first defect detection submodule and the second defect detection submodule is not updated; The specific algorithm of the collaborative teaching is as follows: , in, and Respectively represent the confidence of the basic image defect detection results of the first defect detection submodule and the second defect detection submodule, represents the defect classification threshold, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and Represents the first defect detection submodule and the second defect detection submodule after collaborative teaching is updated.

[0014] The present invention proposes a defect detection system based on language prompts and collaborative teaching, comprising: A data enhancement module is used to collect the image data of the target to be tested in real time and perform data enhancement processing, wherein the data enhancement processing is used to expand and obtain the normal sample image data and defect sample image data of the target to be tested; A language prompt learning module is used to obtain a language prompt text according to the category of the target image data to be tested, and then obtain a language prompt text enhancement feature according to the language prompt, wherein the language prompt text is in a learnable state and is used to guide feature positioning and feature extraction; An image encoding module, used for performing attention enhancement processing on the target image data to be tested according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature, wherein the attention enhancement processing includes performing multimodal feature alignment according to the language prompt text enhancement feature; The defect detection module is used to obtain the language prompt defect detection result and the basic image defect detection result respectively according to the local feature attention enhancement feature to calculate the final anomaly score and perform defect detection update.

[0015] The present invention also provides a storage medium, wherein the storage medium stores one or more programs, and when the programs are executed by a processor, the defect detection method based on language prompts and collaborative teaching as described above is implemented.

[0016] The present invention also provides a computer device, the computer device comprising a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the defect detection method based on language prompts and collaborative teaching as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a defect detection method based on language prompts and collaborative teaching proposed in the first embodiment of the present invention; Figure 2 A flow chart of a defect detection method based on language prompts and collaborative teaching proposed in a second embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a defect detection system based on language prompts and collaborative teaching proposed in the third embodiment of the present invention.

[0018] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0019] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0020] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0022] See also Figure 1 , which is a flow chart of a defect detection method based on language prompts and collaborative teaching proposed in the first embodiment of the present invention, the defect detection method based on language prompts and collaborative teaching comprises steps S01 to S04, wherein: Step S01: real-time acquisition of target image data and data enhancement processing; It should be noted that the data enhancement processing in this embodiment is used to expand the normal sample image data and defect sample image data of the target to be tested, and the data enhancement module performs data enhancement after collecting the image data of the target to be tested in real time; Acquire a plurality of normal sample image data according to random affine transformation and sharpening, and then simulate defects according to Perlin noise to fit the defects to the normal sample image data to acquire a plurality of defect sample image data; The specific algorithm of data enhancement is as follows: , in, represents defect sample image data, Represents a binary image obtained by randomly generating Perlin noise through a random threshold. express The inverse matrix of represents the input image, represents a random texture image, represents the random transparency parameter, Represents element-wise multiplication.

[0023] Step S02: obtaining a language prompt text according to the category of the target image data to be tested, and then obtaining a language prompt text enhancement feature according to the language prompt; It should be noted that in this embodiment, the language prompt text is in a learnable state. The language prompt text is used to guide feature positioning and feature extraction. The language prompt learning module obtains the language prompt text according to the category of the target image data to be tested. The language prompt text includes normal language prompt text and abnormal language prompt text. The specific algorithm for obtaining the language prompt text is as follows: , , in, Indicates normal language prompt text, Indicates abnormal language prompt text. Learnable word embeddings representing normal language prompt text, Learnable word embeddings to represent abnormal language hint text, Indicates the category of the target image data to be tested, Defective boot, Indicates the defect name; According to the language prompt text, the corresponding language prompt text enhancement feature is generated by feeding forward, and the specific algorithm of the feedforward step is as follows: in, Represents the input hint text of the hint layer. Represents the prompt layer, Indicates the prompt layer number, represents the maximum number of learnable hint layers, Indicates learnable hint text.

[0024] Step S03: performing attention enhancement processing on the target image data to be measured according to the local feature attention enhancement algorithm to obtain the local feature attention enhancement feature; It should be noted that the attention enhancement processing in this embodiment includes performing multimodal feature alignment according to the language prompt text enhancement feature, and the image encoding module performs attention enhancement processing on the target image data to be tested according to the local feature attention enhancement algorithm, and the image encoding module includes multiple local feature attention enhancement mechanism layers; Extract local features. The specific algorithm for extracting local features is as follows: in, represents the original output features, represents the local feature attention enhancement feature, represents the image category feature, represents the last local feature, denote the query matrix, key matrix and value matrix respectively, express Attention projection, represents the local feature attention projection, represents the attention function.

[0025] Step S04: acquiring language prompt defect detection results and basic image defect detection results respectively according to the local feature attention enhancement feature to calculate the final anomaly score and perform defect detection update; It should be noted that in this embodiment, the image encoding module selects different local feature attention enhancement mechanism layers according to the preset sequence number to obtain multiple intermediate image block-level features, and performs multimodal feature alignment on the intermediate image block-level features and the language prompt text to obtain multimodal alignment features; The language prompt defect detection result is obtained according to the language prompt text enhancement feature and the multimodal alignment feature. The specific algorithm for obtaining the language prompt defect detection result is as follows: , in, Indicates the language prompt defect detection results, represents the upsampling operation, Represents the cosine similarity calculation operation, represents the multimodal alignment feature, and They represent normal language prompt text enhancement features and abnormal language prompt text enhancement features respectively. represents the number of layers of the local feature attention enhancement mechanism, represents the total number of selected local feature attention enhancement mechanism layers, represents the exponential function; The defect detection module includes a first defect detection submodule and a second defect detection submodule, wherein the first defect detection submodule and the second defect detection submodule respectively obtain basic image defect detection results according to multimodal alignment features; The specific algorithm for obtaining the basic image defect detection result is as follows: , , in, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and represent the first defect detection submodule and the second defect detection submodule respectively; The final abnormality score is calculated based on the language prompt defect detection result and the basic image defect detection result. The specific algorithm for calculating the final abnormality score is as follows: , in, represents the final anomaly score, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, Indicates language prompt defect detection results; Then the loss calculation is performed, and the specific algorithm of the loss calculation is as follows: , , , in, represents focal loss, represents dice loss, represents the total loss, Represents the total number of pixels, Represents the pixel ordinal number, represents the predicted probability, represents the adjustable parameter of weight, represents the output of the decoder, represents the true label, and represents the loss balance coefficient; Performing threshold filtering according to the language prompt defect detection result to obtain a normal image area, so as to update a first defect detection submodule and a second defect detection submodule of the defect detection module according to the normal image area; The specific algorithm of the update is as follows: , , , in, represents the pixel region coordinates, Represents pixel region labeling, Indicates the language prompt defect detection results, represents the threshold learned from the expanded defect sample image data, and represent the first defect detection submodule and the second defect detection submodule respectively, represents the learning rate, represents the calculation of gradient, represents the total loss; Performing collaborative teaching on a first defect detection submodule and a second defect detection submodule of the defect detection module; Performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the first defect detection submodule to update the second defect detection submodule, and performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the second defect detection submodule to update the first defect detection submodule, and the high-confidence image area used for updating in the first defect detection submodule and the second defect detection submodule is not updated; The specific algorithm of the collaborative teaching is as follows: , in, and Respectively represent the confidence of the basic image defect detection results of the first defect detection submodule and the second defect detection submodule, represents the defect classification threshold, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and Represents the first defect detection submodule and the second defect detection submodule after collaborative teaching is updated.

[0026] In summary, according to the above-mentioned defect detection method based on language prompts and collaborative teaching, through data enhancement, the normal sample image data and the defective sample image data are expanded, so as to avoid the problem of insufficient sample data in actual production, and then by introducing learnable language prompt text, the detection ability of different defects of different types of products is enhanced, and more and more accurate local detail information is obtained through the local feature attention enhancement mechanism to enhance the overall detection accuracy, and through two independent defect detection result predictions, more comprehensive defect detection results are obtained. At the same time, the two defect detection modules prompt and update each other, so that the overall detection is more in line with the actual application situation, thereby improving the versatility of the detection method. The present invention improves the accuracy and versatility of the defect detection method. Specifically, the image data of the target to be tested is collected in real time and data enhancement processing is performed. The data enhancement processing is used to expand and obtain normal sample image data and defect sample image data of the target to be tested, thereby avoiding the problem of deviation in detection accuracy caused by insufficient sample data in actual use. A language prompt text is obtained according to the category of the image data of the target to be tested, and then a language prompt text enhancement feature is obtained according to the language prompt. The language prompt text is in a learnable state and is used to guide feature positioning and feature extraction. The image data of the target to be tested is subjected to attention enhancement processing according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature. The attention enhancement processing includes multimodal feature alignment according to the language prompt text enhancement feature, further enhancing the attention to local detail information, avoiding the influence of complex detection conditions with a wide variety of products and defects in the actual production process, and further improving the accuracy of the overall detection. According to the local feature attention enhancement feature, the language prompt defect detection result and the basic image defect detection result are respectively obtained to calculate the final abnormality score, and the defect detection update is performed, thereby further improving the versatility of the detection method to be more in line with the actual application scenario. The present invention improves the accuracy and versatility of the defect detection method.

[0027] See also Figure 2 , which is a flow chart of a polyp image segmentation method based on a detail restoration network proposed in a second embodiment of the present invention, the polyp image segmentation method based on a detail restoration network comprises steps S11 to S16, wherein: Step S11: the data enhancement module collects the target image data to be tested in real time and then performs data enhancement, obtains a plurality of normal sample image data according to random affine transformation and sharpening, and then simulates defects according to Perlin noise to fit them into the normal sample image data, thereby obtaining a plurality of defect sample image data; Step S12: the language prompt learning module obtains the language prompt text according to the category of the target image data to be tested, feeds forward according to the language prompt text and generates corresponding language prompt text enhancement features; Step S13: the image encoding module performs attention enhancement processing on the target image data to be measured according to the local feature attention enhancement algorithm to extract local features; Step S14: the image encoding module selects different local feature attention enhancement mechanism layers according to the preset sequence number to obtain multiple intermediate image block-level features, performs multimodal feature alignment on the intermediate image block-level features and the language prompt text to obtain multimodal alignment features, obtains language prompt defect detection results according to the language prompt text enhancement features and the multimodal alignment features, and obtains basic image defect detection results according to the multimodal alignment features; It should be noted that the image encoding module in this embodiment includes 24 local feature attention enhancement mechanism layers, and the preset sequence numbers are [6, 12, 18, 24] to correspond to the selection of four intermediate image block-level features of the [6, 12, 18, 24]th layer.

[0028] Step S15: Calculate the final abnormality score based on the language prompt defect detection result and the basic image defect detection result, and then calculate the loss; Step S16: performing threshold filtering according to the language prompt defect detection result to obtain the normal area of ​​the image, and updating the first defect detection submodule and the second defect detection submodule of the defect detection module according to the normal area of ​​the image, and performing collaborative teaching on the defect detection module; It should be noted that the defect detection method based on language prompts and collaborative teaching of the present invention is compared with the existing defect detection model in a comparative test, and the comparison is performed under four evaluation indicators: average accuracy, area overlap rate, area under the pixel-level curve, and area under the image-level curve. The data set of the comparative test adopts the MVTec AD defect detection data set. The specific comparison results are shown in Table 1 below: Table 1 According to the above table, compared with the existing defect detection model, the defect detection method of the present invention is in a comprehensive lead in the four evaluation indicators of average accuracy, area overlap rate, area under the pixel-level curve, and area under the image-level curve, and has made significant progress.

[0029] In summary, according to the above-mentioned defect detection method based on language prompts and collaborative teaching, through data enhancement, the normal sample image data and the defective sample image data are expanded, so as to avoid the problem of insufficient sample data in actual production, and then by introducing learnable language prompt text, the detection ability of different defects of different types of products is enhanced, and more and more accurate local detail information is obtained through the local feature attention enhancement mechanism to enhance the overall detection accuracy, and through two independent defect detection result predictions, more comprehensive defect detection results are obtained. At the same time, the two defect detection modules prompt and update each other, so that the overall detection is more in line with the actual application situation, thereby improving the versatility of the detection method. The present invention improves the accuracy and versatility of the defect detection method. Specifically, the image data of the target to be tested is collected in real time and data enhancement processing is performed. The data enhancement processing is used to expand and obtain normal sample image data and defect sample image data of the target to be tested, thereby avoiding the problem of deviation in detection accuracy caused by insufficient sample data in actual use. A language prompt text is obtained according to the category of the image data of the target to be tested, and then a language prompt text enhancement feature is obtained according to the language prompt. The language prompt text is in a learnable state and is used to guide feature positioning and feature extraction. The image data of the target to be tested is subjected to attention enhancement processing according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature. The attention enhancement processing includes multimodal feature alignment according to the language prompt text enhancement feature, further enhancing the attention to local detail information, avoiding the influence of complex detection conditions with a wide variety of products and defects in the actual production process, and further improving the accuracy of the overall detection. According to the local feature attention enhancement feature, the language prompt defect detection result and the basic image defect detection result are respectively obtained to calculate the final abnormality score, and the defect detection update is performed, thereby further improving the versatility of the detection method to be more in line with the actual application scenario. The present invention improves the accuracy and versatility of the defect detection method.

[0030] See also Figure 3 , which is a schematic diagram of the structure of a defect detection system based on language prompts and collaborative teaching proposed in the third embodiment of the present invention, the system includes: The data enhancement module 10 is used to collect the image data of the target to be tested in real time and perform data enhancement processing, wherein the data enhancement processing is used to expand and obtain the normal sample image data and defect sample image data of the target to be tested; A language prompt learning module 20 is used to obtain a language prompt text according to the category of the target image data to be tested, and then obtain a language prompt text enhancement feature according to the language prompt, wherein the language prompt text is in a learnable state and is used to guide feature positioning and feature extraction; An image encoding module 30 is used to perform attention enhancement processing on the target image data to be tested according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature, wherein the attention enhancement processing includes performing multimodal feature alignment according to the language prompt text enhancement feature; The defect detection module 40 is used to obtain the language prompt defect detection result and the basic image defect detection result respectively according to the local feature attention enhancement feature to calculate the final abnormality score and perform defect detection update.

[0031] The present invention also proposes a computer storage medium on which one or more programs are stored. When the programs are executed by a processor, the above-mentioned defect detection method based on language prompts and collaborative teaching is implemented.

[0032] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned defect detection method based on language prompts and collaborative teaching.

[0033] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0034] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0035] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0036] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0037] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A defect detection method based on language prompts and collaborative teaching, characterized in that: include: Collecting the image data of the target to be tested in real time and performing data enhancement processing, wherein the data enhancement processing is used to expand and obtain the normal sample image data and defect sample image data of the target to be tested; Acquire a language prompt text according to the category of the target image data to be tested, and then acquire a language prompt text enhanced feature according to the language prompt, wherein the language prompt text is in a learnable state and is used to guide feature positioning and feature extraction; Performing attention enhancement processing on the target image data to be tested according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature, wherein the attention enhancement processing includes performing multimodal feature alignment according to the language prompt text enhancement feature; According to the local feature attention enhancement feature, the language prompt defect detection result and the basic image defect detection result are respectively obtained to calculate the final anomaly score and perform defect detection update.

2. The defect detection method based on language prompts and collaborative teaching according to claim 1 is characterized in that: The step of real-time acquisition of the target image data to be measured and performing data enhancement processing specifically includes: The data enhancement module collects the image data of the target to be tested in real time and then performs data enhancement; Acquire a plurality of normal sample image data according to random affine transformation and sharpening, and then simulate defects according to Perlin noise to fit the defects to the normal sample image data to acquire a plurality of defect sample image data; The specific algorithm of data enhancement is as follows: ， in, represents defect sample image data, Represents a binary image obtained by randomly generating Perlin noise through a random threshold. express The inverse matrix of represents the input image, represents a random texture image, represents the random transparency parameter, Represents element-wise multiplication.

3. The defect detection method based on language prompts and collaborative teaching according to claim 1 is characterized in that: The step of obtaining a language prompt text according to the category of the target image data to be tested, and then obtaining a language prompt text enhancement feature according to the language prompt specifically includes: The language prompt learning module obtains the language prompt text according to the category of the target image data to be tested. The language prompt text includes normal language prompt text and abnormal language prompt text. The specific algorithm for obtaining the language prompt text is as follows: ， ， in, Indicates normal language prompt text, Indicates abnormal language prompt text. Learnable word embeddings representing normal language prompt text, Learnable word embeddings to represent abnormal language hint text, Indicates the category of the target image data to be tested, Defective boot, Indicates the defect name; According to the language prompt text, the corresponding language prompt text enhancement feature is generated by feeding forward, and the specific algorithm of the feedforward step is as follows: in, Represents the input hint text of the hint layer. Represents the prompt layer, Indicates the prompt layer number, represents the maximum number of learnable hint layers, Indicates learnable hint text.

4. The defect detection method based on language prompts and collaborative teaching according to claim 1 is characterized in that: The step of performing attention enhancement processing on the target image data to be tested according to the local feature attention enhancement algorithm to obtain the local feature attention enhancement feature specifically includes: The image encoding module performs attention enhancement processing on the target image data to be measured according to the local feature attention enhancement algorithm, and the image encoding module includes multiple local feature attention enhancement mechanism layers; Extract local features. The specific algorithm for extracting local features is as follows: in, represents the original output features, represents the local feature attention enhancement feature, represents the image category feature, represents the last local feature, denote the query matrix, key matrix and value matrix respectively, express Attention projection, represents the local feature attention projection, represents the attention function.

5. The defect detection method based on language prompts and collaborative teaching according to claim 1, characterized in that: The step of respectively acquiring the language prompt defect detection result and the basic image defect detection result according to the local feature attention enhancement feature specifically includes: The image encoding module selects different local feature attention enhancement mechanism layers according to a preset sequence number to obtain multiple intermediate image block-level features, performs multimodal feature alignment on the intermediate image block-level features and the language prompt text, and obtains multimodal alignment features; The language prompt defect detection result is obtained according to the language prompt text enhancement feature and the multimodal alignment feature. The specific algorithm for obtaining the language prompt defect detection result is as follows: ， in, Indicates the language prompt defect detection results, represents the upsampling operation, Represents the cosine similarity calculation operation, represents the multimodal alignment feature, and They represent normal language prompt text enhancement features and abnormal language prompt text enhancement features respectively. represents the number of layers of the local feature attention enhancement mechanism, represents the total number of selected local feature attention enhancement mechanism layers, represents the exponential function; The defect detection module includes a first defect detection submodule and a second defect detection submodule, wherein the first defect detection submodule and the second defect detection submodule respectively obtain basic image defect detection results according to multimodal alignment features; The specific algorithm for obtaining the basic image defect detection result is as follows: ， ， in, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and They represent the first defect detection submodule and the second defect detection submodule respectively.

6. The defect detection method based on language prompts and collaborative teaching according to claim 1 is characterized in that: The step of calculating the final anomaly score specifically includes: The final abnormality score is calculated based on the language prompt defect detection result and the basic image defect detection result. The specific algorithm for calculating the final abnormality score is as follows: ， in, represents the final anomaly score, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, Indicates language prompt defect detection results; Then the loss calculation is performed, and the specific algorithm of the loss calculation is as follows: ， ， ， in, represents focal loss, represents dice loss, represents the total loss, Represents the total number of pixels, Represents the pixel ordinal number, represents the predicted probability, represents the adjustable parameter of weight, represents the output of the decoder, represents the true label, and Represents the loss balance coefficient.

7. The defect detection method based on language prompts and collaborative teaching according to claim 1 is characterized in that: The step of performing defect detection and updating specifically includes: Performing threshold filtering according to the language prompt defect detection result to obtain a normal image area, so as to update a first defect detection submodule and a second defect detection submodule of the defect detection module according to the normal image area; The specific algorithm of the update is as follows: ， ， ， in, represents the pixel region coordinates, Represents pixel region labeling, Indicates the language prompt defect detection results, represents the threshold learned from the expanded defect sample image data, and represent the first defect detection submodule and the second defect detection submodule respectively, represents the learning rate, represents the calculation of gradient, represents the total loss; Performing collaborative teaching on a first defect detection submodule and a second defect detection submodule of the defect detection module; Performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the first defect detection submodule to update the second defect detection submodule, and performing loss calculation according to the high-confidence image area in the basic image defect detection result obtained by the second defect detection submodule to update the first defect detection submodule, and the high-confidence image area used for updating in the first defect detection submodule and the second defect detection submodule is not updated; The specific algorithm of the collaborative teaching is as follows: ， in, and Respectively represent the confidence of the basic image defect detection results of the first defect detection submodule and the second defect detection submodule, represents the defect classification threshold, and represent the basic image defect detection results obtained by the first defect detection submodule and the second defect detection submodule respectively, and Represents the first defect detection submodule and the second defect detection submodule after collaborative teaching is updated.

8. A defect detection system based on language prompts and collaborative teaching, characterized in that: include: A data enhancement module is used to collect the image data of the target to be tested in real time and perform data enhancement processing, wherein the data enhancement processing is used to expand and obtain the normal sample image data and defect sample image data of the target to be tested; A language prompt learning module is used to obtain a language prompt text according to the category of the target image data to be tested, and then obtain a language prompt text enhancement feature according to the language prompt, wherein the language prompt text is in a learnable state and is used to guide feature positioning and feature extraction; An image encoding module, used for performing attention enhancement processing on the target image data to be tested according to a local feature attention enhancement algorithm to obtain a local feature attention enhancement feature, wherein the attention enhancement processing includes performing multimodal feature alignment according to the language prompt text enhancement feature; The defect detection module is used to obtain the language prompt defect detection result and the basic image defect detection result respectively according to the local feature attention enhancement feature to calculate the final anomaly score and perform defect detection update.

9. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by a processor, implement the defect detection method based on language prompts and collaborative teaching as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the defect detection method based on language prompts and collaborative teaching described in any one of claims 1 to 7 is implemented.

Images