Industrial defect detection method based on vision-language model and related equipment

Abstract

The invention discloses an industrial defect detection method and related equipment based on a vision-language model, and the method comprises the steps: training an industrial defect detection model constructed based on a pre-trained vision-language model according to an image data set, obtaining an optimal model, and extracting the optimal model weight of the optimal model; a feature mapping module of the industrial defect detection model is used for realizing alignment of visual features and a text semantic space, and a text prompt generation module is used for dynamically generating a text prompt; loading the optimal model weight into the to-be-detected industrial defect detection model to obtain a target industrial defect detection model; and inputting the to-be-detected image into the target industrial defect detection model for defect detection to obtain a target defect detection result. According to the method, the model does not need to be retrained for each new category of image, the samples which do not appear can be detected, the detection requirements of different categories of products are met, the model detection efficiency and accuracy are improved, and the method can be widely applied to the technical field of artificial intelligence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an industrial defect detection method based on a vision-language model and related equipment. BACKGROUND

[0002] Industrial defect detection plays a crucial role in modern manufacturing, and its core goal is to determine whether a product has abnormalities and to accurately locate the defect area when an abnormality is detected. At present, unsupervised or normal sample-based anomaly detection methods have become the mainstream direction of research. Such methods not only significantly reduce the dependence on abnormal samples, but also improve the adaptability and scalability of the model in actual industrial environments.

[0003] However, the current mainstream methods mostly adopt a "one-class-one-model" strategy, that is, a model is trained for each product category to obtain better detection performance. This approach has obvious limitations: in actual applications, industrial production lines often cover multiple product categories. If an independent model needs to be trained for each product, not only is the training cost high, but also tedious hyperparameter tuning needs to be performed for each category, which seriously hinders the large-scale deployment and application of the model. In addition, the model of this strategy must first collect a sufficient number of normal samples of new products for training before it can be put into use. This step not only increases the deployment period of the detection system, but also puts higher requirements on sample acquisition.

[0004] In summary, the technical problems in the related art need to be improved. SUMMARY

[0005] The embodiments of the present application aim to at least solve one of the technical problems in the related art. To this end, the main purpose of the embodiments of the present application is to propose an industrial defect detection method based on a vision-language model and related equipment, which aims to avoid retraining the model for each new category of image and can perform defect detection on samples that have not appeared before, to adapt to more different categories of product detection needs and improve the detection efficiency and accuracy of the model.

[0006] To achieve the above-mentioned purpose, one aspect of the embodiments of the present application proposes an industrial defect detection method based on a vision-language model, which comprises the following steps:

[0007] An initial industrial defect detection image data set is obtained;

[0008] The initial industrial defect detection image data set is preprocessed to obtain a target industrial defect detection image data set;

[0009] According to the target industrial defect detection image dataset, an initial industrial defect detection model based on a pre-trained visual-linguistic model is trained to obtain an optimal industrial defect detection model, and optimal model weights in the optimal industrial defect detection model are extracted; the initial industrial defect detection model comprises the pre-trained visual-linguistic model, a feature mapping module, and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and a text semantic space, and the text prompt generation module is used to dynamically generate a text prompt;

[0010] The optimal model weights are loaded into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model;

[0011] The image-preprocessed to-be-detected image is input into the target industrial defect detection model for defect detection to obtain a target defect detection result.

[0012] In some embodiments, the image preprocessing of the initial industrial defect detection image dataset to obtain a target industrial defect detection image dataset comprises:

[0013] The initial industrial defect detection image dataset is subjected to random cropping processing to obtain a first training image dataset;

[0014] The first training image dataset is subjected to bicubic interpolation scaling processing to obtain a second training image dataset;

[0015] The second training image dataset is subjected to image channel unification processing to obtain a third training image dataset;

[0016] The third training image dataset is subjected to tensor conversion processing to obtain a fourth training image dataset;

[0017] The fourth training image dataset is subjected to data standardization processing to obtain the target industrial defect detection image dataset.

[0018] In some embodiments, the training of the initial industrial defect detection model based on the pre-trained visual-linguistic model according to the target industrial defect detection image dataset to obtain the optimal industrial defect detection model and the extraction of the optimal model weights in the optimal industrial defect detection model comprise:

[0019] inputting the target industrial defect detection image dataset into a visual encoder in the initial industrial defect detection model, extracting target visual feature data corresponding to the target industrial defect detection image dataset through a feature mapping module in the visual encoder; wherein the target visual feature data comprises first visual feature data, second visual feature data, third visual feature data, and fourth visual feature data;

[0020] inputting the fourth visual feature data into the text prompt generation module, and generating a target text prompt according to the target visual feature data through the text prompt generation module;

[0021] inputting the target text prompt into a text encoder in the initial industrial defect detection model, and generating target text feature data according to the target text prompt through the text encoder;

[0022] performing global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score;

[0023] performing local score calculation on the target visual feature data and the target text feature data to obtain a local anomaly score;

[0024] performing model loss calculation on the global anomaly score and the local anomaly score to obtain a total model training loss;

[0025] performing back propagation and optimization on the initial industrial defect detection model according to the total model training loss until model training is completed;

[0026] performing quality evaluation on the trained initial industrial defect detection model, if the model weight of the trained initial industrial defect detection model reaches a best performance threshold, taking the trained initial industrial defect detection model as the best industrial defect detection model, and extracting the best model weight in the best industrial defect detection model.

[0027] In some embodiments, the global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score comprises:

[0028] performing cosine similarity calculation on the first visual feature data and the target text feature data to obtain a first cosine similarity;

[0029] performing cosine similarity calculation on the second visual feature data and the target text feature data to obtain a second cosine similarity;

[0030] performing cosine similarity calculation on the third visual feature data and the target text feature data to obtain a third cosine similarity;

[0031] performing cosine similarity calculation on the fourth visual feature data and the target text feature data to obtain a fourth cosine similarity;

[0032] performing summation processing on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity to obtain a target cosine similarity;

[0033] performing average calculation processing on the target cosine similarity to obtain the global anomaly score.

[0034] In some embodiments, the model loss calculation on the global anomaly score and the local anomaly score to obtain a model training total loss comprises:

[0035] performing loss calculation on the global anomaly score and a real global label to obtain a global loss;

[0036] performing up-sampling processing on the local anomaly score, and performing loss calculation on the up-sampling processed local anomaly score and a real local label to obtain a local loss;

[0037] determining the model training total loss according to the global loss and the local loss.

[0038] In some embodiments, the method further comprises:

[0039] If the model weight of the trained initial industrial defect detection model does not reach the best performance threshold, returning to perform the step of inputting the target industrial defect detection image dataset into the visual encoder of the initial industrial defect detection model, extracting target visual feature data corresponding to the target industrial defect detection image dataset through the feature mapping module in the visual encoder, until the model weight of the trained initial industrial defect detection model reaches the best performance threshold, to obtain the best industrial defect detection model, and extract the best model weight in the best industrial defect detection model.

[0040] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes an industrial defect detection device based on a visual-linguistic model, which comprises the following modules:

[0041] An image dataset acquisition module is configured to acquire an initial industrial defect detection image dataset;

[0042] An image preprocessing module is configured to perform image preprocessing on the initial industrial defect detection image dataset to obtain a target industrial defect detection image dataset;

[0043] The model training module is configured to train an initial industrial defect detection model based on a pre-trained visual-linguistic model according to the target industrial defect detection image dataset, obtain an optimal industrial defect detection model, and extract optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model comprises the pre-trained visual-linguistic model, a feature mapping module, and a text prompt generation module, the feature mapping module is configured to align visual features and a text semantic space, and the text prompt generation module is configured to dynamically generate a text prompt;

[0044] The optimal model weight loading module is configured to load the optimal model weights into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model.

[0045] The defect detection application module is configured to input a to-be-detected image after image preprocessing into the target industrial defect detection model for defect detection to obtain a target defect detection result.

[0046] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0047] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the above method.

[0048] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the above method.

[0049] The embodiments of the present application at least have the following beneficial effects: The present application provides an industrial defect detection method and related equipment based on a vision-language model, which obtains an initial industrial defect detection image data set, performs image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set, trains an initial industrial defect detection model based on a pre-trained vision-language model according to the target industrial defect detection image data set to obtain an optimal industrial defect detection model, and extracts optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model includes a pre-trained vision-language model, a feature mapping module, and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and a text semantic space, and the text prompt generation module is used to dynamically generate a text prompt; the optimal model weights are loaded into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model; a to-be-detected image after image preprocessing is input into the target industrial defect detection model for defect detection to obtain a target defect detection result. The embodiments of the present application train an initial industrial defect detection model based on a pre-trained vision-language model to obtain an optimal industrial defect detection model, extract optimal model weights of the optimal industrial defect detection model, load the optimal model weights into a to-be-detected industrial defect detection model, and then use the industrial defect detection model containing the optimal model weights to detect defects of an industrial image, without the need to retrain a model for each new category image, so that the model can process samples that have not appeared before, that is, the model can adapt to more product detection requirements of different categories, greatly improving the generalization ability of the model. In addition, the industrial defect detection model of the embodiments of the present application includes a feature mapping module and a text prompt generation module, the feature mapping module can realize deep alignment of visual features and a text semantic space, thereby enhancing the expression of the model in the form of image and text, improving the pixel-level abnormal perception ability, and through the text prompt generation module, exclusive text prompts are generated for each image, which can significantly enhance the generalization ability of the model, overcome challenges such as unknown categories, difficult-to-describe states, and privacy desensitization in an industrial scene, thereby breaking away from the restrictions of fixed text prompts, and improving the detection efficiency and accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a flowchart of an industrial defect detection method based on a vision-language model provided by the embodiments of the present application;

[0051] Figure 2 is a network structure schematic diagram of an industrial defect detection model provided by the embodiments of the present application;

[0052] Figure 3 is a structure schematic diagram of a mapping block provided by the embodiments of the present application;

[0053] Figure 4 is a structural schematic diagram of an output mapping block provided by an embodiment of the present application;

[0054] Figure 5 is a structural schematic diagram of a category mapping module provided by an embodiment of the present application;

[0055] Figure 6 is a structural schematic diagram of a state mapping module provided by an embodiment of the present application;

[0056] Figure 7 is a model training flowchart of an industrial defect detection model provided by an embodiment of the present application;

[0057] Figure 8 is a model testing flowchart of an industrial defect detection model provided by an embodiment of the present application;

[0058] Figure 9 is a structural schematic diagram of an industrial defect detection device based on a visual-linguistic model provided by an embodiment of the present application;

[0059] Figure 10 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementation described in the following exemplary embodiments does not represent all the implementations consistent with the embodiments of the present application, but is only an example of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0061] It can be understood that the terms "first", "second", and the like used in the present application can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0062] As used herein, the terms “at least one”, “multiple”, “each”, “any of’ or the like, at least one includes one, two, or more, multiple includes two or more, each refers to each of a corresponding plurality, and any of refers to any one of a plurality.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0064] Currently, industrial defect detection plays a crucial role in modern manufacturing, and its core goal is to determine whether a product has abnormalities and further accurately locate the defect area when an abnormality is detected. This task not only requires global classification judgment of the image, but also requires pixel-level fine prediction ability, so it is a highly challenging but valuable research direction in the field of computer vision. With the continuous improvement of manufacturing line automation, the production efficiency of products has been significantly improved, and accordingly, higher requirements have been put forward for the automatic detection of product quality.

[0065] Exemplarily, although the traditional manual visual detection method is intuitive, it has many limitations: (1) manual detection is difficult to continue for a long time, is prone to fatigue, and has low efficiency; (2) the detection accuracy is obviously affected by the working time, and is prone to miss detection and false detection; (3) manual detection has high labor cost and is difficult to meet the needs of large-scale industrial production; (4) the rapid development of computer vision technology provides an effective solution for industrial defect detection. Early research mainly relied on traditional machine learning methods such as SVM (Support Vector Machine) and the like, which recognized defects through manual design of feature extraction methods. On some standard data sets, these methods can achieve high accuracy; however, such methods are often slow in reasoning, have poor adaptability to defect types, and have limited robustness and generalization ability. In addition, with the rise of deep learning technology, models based on convolutional neural network (CNN) and visual Transformer architecture have shown stronger feature learning ability and higher detection performance in industrial defect detection. Deep models can automatically learn multi-level feature representations from data, recognize more complex and diverse defect types, such as smiling scratches, cracks, structural omissions, and even element misplacement and other logical abnormalities; however, the types of defects of industrial products are diverse and complex, and they appear in different ways in different types of products, and it is difficult to completely collect and label all possible defect types.

[0066] In contrast, it is relatively easy to obtain a large number of high-quality normal sample data. Therefore, unsupervised or normal sample-based anomaly detection methods have become the mainstream direction of research. Such methods usually learn the distribution characteristics of normal samples, construct a "normal model", and when there is a significant difference between the test sample and the model, it is judged as abnormal. This approach not only significantly reduces the dependence on abnormal samples, but also improves the adaptability and scalability of the model in the actual industrial environment. Although researchers have made many exciting achievements in the direction of unsupervised industrial defect detection, the current mainstream methods mostly adopt the "one-class-one-model" strategy, that is, a model is trained for each product category to obtain better detection performance. This approach has obvious limitations: in practical applications, industrial production lines often cover multiple product categories. If an independent model needs to be trained for each product, not only is the training cost high, but also tedious hyperparameter tuning is required for each category, which seriously hinders the large-scale deployment and application of the model. In addition, this strategy is also relatively limited when facing new products. The model cannot be plug-and-play, but must be trained with a sufficient number of normal samples of the new product before it can be put into use. This pre-step not only increases the deployment cycle of the detection system, but also puts higher requirements on sample acquisition. Especially in some scenarios, due to data privacy and commercial confidentiality agreements, normal sample data may be difficult or even completely unavailable, causing unsupervised methods to fail in such scenarios. Therefore, the current unsupervised methods have significant bottlenecks in model generalization ability, training cost, and sample acquisition.

[0067] In view of this, the embodiments of the present application provide an industrial defect detection method based on a visual-language model and related equipment. The method comprises the following steps: obtaining an initial industrial defect detection image dataset; performing image preprocessing on the initial industrial defect detection image dataset to obtain a target industrial defect detection image dataset; training an initial industrial defect detection model based on a pre-trained visual-language model according to the target industrial defect detection image dataset to obtain an optimal industrial defect detection model, and extracting optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model comprises a pre-trained visual-language model, a feature mapping module, and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and a text semantic space, and the text prompt generation module is used to dynamically generate a text prompt; loading the optimal model weights into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model; inputting a to-be-detected image after image preprocessing into the target industrial defect detection model for defect detection to obtain a target defect detection result. The embodiments of the present application train an initial industrial defect detection model based on a pre-trained visual-language model to obtain an optimal industrial defect detection model, extract optimal model weights of the optimal industrial defect detection model, load the optimal model weights into a to-be-detected industrial defect detection model, and then use the industrial defect detection model containing the optimal model weights to detect defects of an industrial image, so that the model can process samples that have not appeared before, that is, the model can adapt to more different types of product detection requirements, and the generalization ability of the model is greatly improved. In addition, the industrial defect detection model of the embodiments of the present application comprises a feature mapping module and a text prompt generation module, the feature mapping module can realize deep alignment of visual features and a text semantic space, thereby enhancing the image-text fusion expression of the model, improving the pixel-level abnormal perception ability, and generating exclusive text prompts for each image through the text prompt generation module, which can significantly enhance the generalization ability of the model, overcome challenges such as unknown categories, difficult-to-describe states, privacy desensitization, and the like in an industrial scene, thereby breaking away from the restrictions of fixed text prompts, and improving the detection efficiency and accuracy of the model.

[0068] The industrial defect detection method based on a vision-language model provided in the embodiments of the present application relates to the technical field of artificial intelligence. The industrial defect detection method based on a vision-language model provided in the embodiments of the present application can be applied to a terminal, can also be applied to a server, and can further be software running in the terminal or the server. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, or the like, but is not limited thereto; the server end can be configured as a stand-alone physical server, can also be configured as a server cluster or a distributed system formed by multiple physical servers, can further be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms, and the server can also be a node server in a blockchain network; and the software can be an application for implementing the industrial defect detection method based on a vision-language model, and the like, but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as a program module. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0070] Please refer to Figure 1 , Figure 1 is an optional flowchart of the industrial defect detection method based on a vision-language model provided in the embodiments of the present application, Figure 1 The method in the flowchart can include, but is not limited to, steps S101 to S105.

[0071] Step S101, obtaining an initial industrial defect detection image data set;

[0072] The initial industrial defect detection image data set refers to an original image data set used for subsequent training of an industrial defect detection model, and is an image data that has not undergone any data processing.

[0073] For the selection of the initial industrial defect detection image dataset, a related public image dataset currently existing in the industrial defect detection scene can be selected for model training, and details are not repeated in the embodiments of the application. It can be understood that the embodiments of the application do not limit this.

[0074] In step S102, the initial industrial defect detection image dataset is preprocessed to obtain a target industrial defect detection image dataset.

[0075] In some embodiments, step S102 can include: performing random cropping processing on the initial industrial defect detection image dataset to obtain a first training image dataset; performing bicubic interpolation scaling processing on the first training image dataset to obtain a second training image dataset; performing image channel unification processing on the second training image dataset to obtain a third training image dataset; performing tensor conversion processing on the third training image dataset to obtain a fourth training image dataset; and performing data standardization processing on the fourth training image dataset to obtain the target industrial defect detection image dataset.

[0076] In a specific implementation, in order to improve the robustness and generalization ability of the model, the input image needs to be standardized and preprocessed. Assuming that the original training image is X raw ∈R H×W×C , first, the original training image is randomly cropped and scaled (RandomResizedCrop). Specifically, the original training image X raw ∈R H×W×C is randomly cropped to a certain proportion of the area to obtain a first sub Figure X 1∈R h×w×C , and the first sub Figure X 1∈R h×w×C is scaled to a fixed size by bicubic interpolation (Bicubicinterpolation) to obtain a second sub Figure X 2∈R n×n×C , wherein the calculation expressions of the first sub Figure X 1 and the second sub Figure X 2 are as follows:

[0077] X1=RadomCrop(X raw ),X1∈R h×w×C

[0078]

[0079] However, in industrial defect detection, the format of the input image can not be a unified RGB format. Some images can be grayscale images, and some can be RGBA. The application requires that the input image be in RGB format. In order to ensure that the input image has a unified RGB three-channel format, the second sub Figure X2, image channel normalization processing is performed to convert the image into an RGB format to obtain a third sub Figure X 3, wherein the third sub Figure X 3 is calculated as follows:

[0080]

[0081] The third sub Figure X 3 is then converted from a pixel matrix to a PyTorch tensor to obtain a fourth sub Figure X 4, wherein the calculation formula for tensor conversion is as follows:

[0082] X4 = ToTensor (X3)

[0083] Finally, the fourth sub Figure X 4 is subjected to a normalization operation to obtain a fifth sub Figure X 5 (i.e., the target industrial defect detection image dataset), wherein the calculation formula for the fifth sub Figure X 5 is as follows:

[0084] X5 = Normalize (X4, mean = μ, std = σ)

[0085] In the embodiments of the present application, image preprocessing is performed on the initial industrial defect detection image dataset, so that the training data and the pre-trained model can better match each other, and the numerical distribution is more stable, thereby helping the model to be trained more effectively.

[0086] Step S103, according to the target industrial defect detection image dataset, an initial industrial defect detection model based on a pre-trained visual-linguistic model is trained to obtain an optimal industrial defect detection model, and the optimal model weight in the optimal industrial defect detection model is extracted; the initial industrial defect detection model includes the pre-trained visual-linguistic model, a feature mapping module and a text prompt generation module, the feature mapping module is used to realize the alignment of visual features and text semantic space, and the text prompt generation module is used to dynamically generate a text prompt;

[0087] In some embodiments, step S103 can include: inputting the target industrial defect detection image dataset into a visual encoder in the initial industrial defect detection model, extracting target visual feature data corresponding to the target industrial defect detection image dataset through a feature mapping module in the visual encoder; wherein the target visual feature data includes first visual feature data, second visual feature data, third visual feature data, and fourth visual feature data; inputting the fourth visual feature data into a text prompt generation module, generating a target text prompt according to the target visual feature data through the text prompt generation module; inputting the target text prompt into a text encoder in the initial industrial defect detection model, generating target text feature data according to the target text prompt through the text encoder; performing global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score; performing local score calculation on the target visual feature data and the target text feature data to obtain a local anomaly score; performing model loss calculation on the global anomaly score and the local anomaly score to obtain a total model training loss; performing back propagation and optimization on the initial industrial defect detection model according to the total model training loss until the model training is completed; performing quality evaluation on the trained initial industrial defect detection model, if the model weight of the trained initial industrial defect detection model reaches a best performance threshold, the trained initial industrial defect detection model is taken as a best industrial defect detection model, and a best model weight in the best industrial defect detection model is extracted.

[0088] In some specific embodiments, the step of performing global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score can include: performing cosine similarity calculation on the first visual feature data and the target text feature data to obtain a first cosine similarity; performing cosine similarity calculation on the second visual feature data and the target text feature data to obtain a second cosine similarity; performing cosine similarity calculation on the third visual feature data and the target text feature data to obtain a third cosine similarity; performing cosine similarity calculation on the fourth visual feature data and the target text feature data to obtain a fourth cosine similarity; performing summation processing on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity to obtain a target cosine similarity; performing average calculation processing on the target cosine similarity to obtain the global anomaly score.

[0089] In some specific embodiments, the step of performing model loss calculation on the global anomaly score and the local anomaly score to obtain a total model training loss can include: performing loss calculation on the global anomaly score and a real global label to obtain a global loss; performing up-sampling processing on the local anomaly score, and performing loss calculation on the up-sampling processed local anomaly score and a real local label to obtain a local loss; determining the total model training loss according to the global loss and the local loss.

[0090] In some specific embodiments, if the model weights of the initial industrial defect detection model trained do not reach the optimal performance threshold, the step of inputting the target industrial defect detection image dataset into the visual encoder of the initial industrial defect detection model is returned to be executed, the target visual feature data corresponding to the target industrial defect detection image dataset is extracted through the feature mapping module in the visual encoder until the model weights of the initial industrial defect detection model trained reach the optimal performance threshold, to obtain the optimal industrial defect detection model, and the optimal model weights in the optimal industrial defect detection model are extracted.

[0091] In the embodiments of the present application, the pre-trained vision-language model (VLM) can adopt a CLIP (Contrastive Language-Image Pretraining) model. The embodiments of the present application construct an end-to-end anomaly detection model based on the PyTorch framework, and the purpose is to build a zero-shot industrial anomaly detection model based on the vision-language model. It can be understood that the industrial defect detection model constructed in the embodiments of the present application is realized based on the pre-trained vision-language model, specifically: the pre-trained vision-language model is adapted, the modules designed in the embodiments of the present application (such as the feature mapping module and the text prompt generation module) are added, and the constructed industrial defect detection model is applied to industrial defect detection, that is, the pre-trained vision-language model is not trained in the embodiments of the present application, but the parameters of the pre-trained vision-language model are "frozen", and the parameters of the modules designed in the embodiments of the present application (such as the feature mapping module and the text prompt generation module) are trained to obtain the optimal model weights.

[0092] Please refer to Figure 2 , Figure 2 is a network structure diagram of an industrial defect detection model provided by the embodiments of the present application, as shown in Figure 2 , the industrial defect detection model constructed in the embodiments of the present application mainly includes a visual encoder (image encoder) and a text encoder (text encoder) and a text prompt generation module. Among them, the visual encoder (image encoder) includes the visual encoding module (stage1-stage4) of the pre-trained vision-language model and the feature mapping module (feature mapping module) designed in the embodiments of the present application. Figure 2 Figure 2 ​The pre-trained visual-language model uses CLIP as its backbone model, comprising Projection Blocks and Output Projection Blocks. The feature mapping module includes three Projection Blocks and one Output Projection Block. The text prompt generation module includes Class Projection and State Projection modules. The text encoder deploys the text encoding module of the pre-trained visual-language model. Figure 2 The CLIP model is adapted to a zero-shot industrial defect detection model for industrial defect detection through the collaborative work between various modules. The model utilizes a Text Transformer and a Deep TextPrompt technique on the text encoder.

[0093] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a mapping block provided in an embodiment of this application, such as... Figure 3 As shown, Figure 3 This paper illustrates the structural diagram of the Projection Block proposed in this application. This module is used to adapt the image encoder of the Visual-Language Model (CLIP) to the text space to enhance the alignment of image and text features. Specifically, this application divides the CLIP image encoder into four stages (Stage 1 to Stage 4), each stage containing several Transformer blocks. In this embodiment, the output features of the last Transformer block in the first three stages are selected and input into the Projection Block for feature mapping. For example, assuming the output features of a certain stage are... N represents the number of N-1 patches into which the image is divided and one additional CLS token, D v The dimension of visual features is denoted as D in CLIP's text encoder output features. text To achieve better image-text alignment, the projection block undergoes a series of linear transformations, the specific process of which is as follows:

[0094] (1) First, f stage_n Mapping to D text Dimension, to obtain The specific calculation formula is as follows:

[0095]

[0096] (2) Then, map back to the original visual dimension to get The specific calculation formula is as follows:

[0097]

[0098] (3) Map to D dimension again to get text The specific calculation formula is as follows:

[0099]

[0100] In addition, the is also proportionally residual fused with the original visual features as the input of the next stage (module / phase), and the specific calculation formula is as follows:

[0101]

[0102] Through this three-layer linear mapping structure, the projection block not only retains the strong visual expression ability of CLIP originally, but also adapts the industrial image data distribution through the learnable feature conversion, so as to improve the model's perception ability to the pixel-level defect area and optimize the matching effect with the text semantics. In addition, for the zero sample setting, in order to prevent the model from overfitting on the training set, the proportional residual fusion method is also used in the embodiments of the present application to enhance the robustness and generalization performance of the model.

[0103] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of an output mapping block provided by the embodiments of the present application; as shown in Figure 4 , similar to the projection block (Projection Block), the output mapping block (Output Projection Block) is also composed of three linear layers, and different from the projection block (Projection Block) is that, since the output mapping block (Output Projection Block) is the last stage (module / phase), it no longer needs residual connection. In addition, the final output visual feature of the output mapping block (Output Projection Block) will be used for calculating the similarity with the text prompt on one hand, and will be used for the input of the class mapping (Class Projection) module and the state mapping (State Projection) module on the other hand.

[0104] ​​In the basic method of related CLIP-based implementation of industrial defect detection, two types of text prompts are usually designed as "a photo of a normal / abnormal <class>“wherein <class>The category name of the object to be detected is represented. After inputting these text prompts into the text encoder of CLIP, the text features of "normal state" and "abnormal state" can be obtained respectively, and then the similarity with the visual features of the image is calculated. If the image has a higher similarity with the abnormal text features, it is determined that the sample is abnormal, and vice versa. However, this method faces many challenges in industrial practice: first, the types of industrial defects are complex and diverse, and a single "normal / abnormal" state word cannot cover all semantics, and the expression ability is limited; second, many industrial parts have no clear and unified category name, especially in application scenarios involving data desensitization or privacy protection, it is often difficult to obtain accurate <class>information. Therefore, the text prompt mode relying on category names and preset state words excessively cannot guarantee the accuracy and generalization of detection. To solve this problem, the present application designs a state mapping (StateProjection) module and a category mapping (Class Projection) module. Please refer to Figure 5 to Figure 6 , Figure 5 is a structural schematic diagram of a category mapping module provided by an embodiment of the present application, Figure 6 is a structural schematic diagram of a state mapping module provided by an embodiment of the present application, as shown in Figure 5 and Figure 6 , the visual features output by the visual encoder are used to guide the construction of the text prompt, so as to improve the expression ability and adaptability of the prompt. The text prompt of the embodiment of the present application is "a photo of a normal / abnormal <class>with <state>", wherein the visual features finally output by the output projection block include one global CLS token feature and N-1 local patch features, the CLS token contains rich global semantic information and can be used to represent the category of the image; and the N-1 patch features contain fine-grained local information and can reflect the potential normal or abnormal state in the image. Based on this, the application proposes to replace the artificially constructed text prompt with visual features: dynamically generating <class>semantics, generated with N-1 patch features <state>semantics, this way not only avoids the complexity and subjectivity of artificial design prompts, but also effectively solves the problem of insufficient semantic coverage. More importantly, this method can generate personalized semantic prompts for each specific sample, thereby improving the expression ability and detection flexibility of CLIP in the task of zero-shot industrial defect detection. In addition, this method retains normal / abnormal, helping the model to have a more explicit decision boundary in the early stage of training, enhancing the robustness of the model.

[0105] In a specific implementation, to further enhance the expression ability of the text prompt, the embodiment of the present application introduces a DDT (Dynamic Descriptor Token) mechanism. As shown in Figure 2 , during the input of the text prompt into the text encoder, the present application selects a token at a specific position in the output of each layer of the Transformer and replaces it with a learnable tensor of the same dimension. This tensor can be dynamically updated during training and has stronger representation ability. Then, the learnable token is transmitted to the next layer of the Transformer along with the remaining output features to achieve more effective semantic guidance and context enhancement.

[0106] In the embodiment of the present application, first, the training set is defined as D train , the test set is defined as D test , since the present application is aimed at the task of zero-shot industrial anomaly detection, the training set and the test set are disjointed, and the training set does not contain any samples in the test set. The industrial defect detection method based on the visual-linguistic model provided by the embodiment of the present application mainly includes a training phase and a test phase (which can also be understood as a detection application phase). Please refer to Figure 7 , Figure 7 is a model training process schematic diagram of an industrial defect detection model provided by the embodiment of the present application, as shown in Figure 7 As shown, the specific steps of the model training process of the industrial defect detection model are as follows: first, obtain an industrial defect detection training data set; second, perform image preprocessing on the industrial defect detection training data set to obtain model training images; third, input the model training images into an industrial defect detection model constructed based on a pre-trained visual-linguistic model (taking CLIP as an example) for training. Specifically, the specific process of the training stage is as follows: first, load the pre-trained weights of CLIP and freeze the related parameters of CLIP, define the mapping block (Projection Block), the output mapping block (Output Projection Block), the class mapping (Class Projection) module, the state mapping (State Projection) module and the learnable token in the DTT technology as learnable parameters; then, input the model training images into the visual branch (visual encoder) and input the text prompt generated by the text prompt generation module into the text branch (text encoder), wherein the visual branch obtains 4 stages of visual features the text branch obtains text features For the text features, N text represents the number of tokens, since EOS contains rich features, only the features of the tokens corresponding to EOS are taken as the text features in the embodiments of the present application Wherein, the abnormal probability calculation formula of global anomaly detection is as follows:

[0107]

[0108] Wherein, represents the abnormal text feature, represents the normal text feature, represents the CLS token of the visual feature;

[0109] Optionally, the abnormal probability calculation formula of local anomaly detection is as follows:

[0110]

[0111] Wherein, represents the patch feature of the visual feature.

[0112] After calculating the abnormal probability P global (i.e. global anomaly score) of global anomaly detection and the abnormal probability P local (local anomaly score) of local anomaly detection, the total loss L is calculated using the BCE, Focal and Dice loss functions. Specifically, the global anomaly score and the real global label (global ground truth) are used to calculate the loss to obtain the global loss L global ; up-sampling the local anomaly score to the image size after image preprocessing, calculating the loss with the real local label (mask) to obtain the local loss L pixel , and finally calculating the total loss L based on the global loss L globa1 and the local loss L pixel , the specific calculation formula is as follows:

[0113] L global = BCE (P global , P global_truth )

[0114] L pixel = Dice (P local , mask) + Focal (P local , mask)

[0115] L = L global + L pixel

[0116] After calculating the total loss L, the model is back-propagated and optimized according to the total loss L to obtain a trained model. At this time, it is necessary to use the test set to evaluate whether the trained model reaches the best performance. If the trained model does not reach the best performance, return to execute the model training step to adjust and optimize the model until the trained model reaches the best performance. When the trained model reaches the best performance, save the model training weight of the best model, and thus the model training phase ends.

[0117] Since the data source of the pre-trained visual-linguistic model (CLIP) is natural images, there is a large gap in data distribution with the industrial field. In addition, CLIP was designed for image classification at the beginning, while defect detection requires pixel-level prediction. Therefore, when directly using CLIP for industrial defect detection, it is often difficult to obtain ideal segmentation results. Therefore, the present application adapts CLIP to zero-shot industrial defect detection by combining the Projection Block, Output Projection Block, Class Projection module, State Projection module, and Deep Text Prompt technology based on the CLIP backbone model. Through the integration of the above modules and technologies, the adaptability gap of CLIP in the industrial scene is made up, so that it has stronger pixel-level anomaly perception and recognition ability, thereby better serving the zero-shot industrial defect detection task.

[0118] As shown in Figure 2 , for any training image (image), after image preprocessing, the image x ∈ R n ×n×3 Specifically, the training process of the industrial defect detection model constructed based on the pre-trained visual-linguistic model is as follows: first, the image x e R n×n×3 after image preprocessing is input into a visual encoder to extract visual features, wherein the visual encoder of CLIP is divided into four stages, each stage is followed by a mapping block (the last stage is followed by an output mapping block), and it is assumed that the feature after the nth visual feature extraction stage is where N represents that the image is divided into N-1 patches plus a cls_token feature, the cls_token feature is rich in global features, and therefore can be used to calculate a global anomaly score, and the remaining patch features have rich local information, and therefore can be used for pixel prediction, D v represents the feature dimension. The visual feature extraction process is as follows: the visual feature is input into the mapping block (if it is the last stage, it is the output mapping block) followed by this stage, and the mapping block will perform three linear mappings on the original visual feature f stage_n The first linear mapping maps the original visual feature f stage_n to the dimension of the text feature D text to obtain the feature The second linear mapping maps the feature back to the dimension of the original visual feature to obtain the feature The output feature of the mapping block will be proportionally connected with the original visual feature to obtain and will be taken as the input of the next stage; the third linear mapping will output a feature with the same dimension as the text feature to obtain that is, the third linear mapping maps the feature to the dimension of the text feature D text to obtain the feature The last output mapping block does not perform residual connection, and the visual feature output by the last stage is generated into a text prompt through a category mapping module and a state mapping module, the generated text prompt is input into a text encoder, and a text feature f text ; then, the cosine similarity is calculated using the text features and the visual features of the four stages, the sum is averaged, the score of the cls_token feature position is taken out, and the global anomaly score is obtained, and the score of the patch feature position is taken out, and the local anomaly score is obtained; then, the global anomaly score and the real global label (global ground truth) are calculated to obtain the loss L global , the local anomaly score is upsampled to the image size after image preprocessing, and the real local label (mask) is calculated to obtain the loss L pixel ; finally, after calculating the anomaly probability P global of global anomaly detection (i.e. global anomaly score) and the anomaly probability P local of local anomaly detection (local anomaly score), the test set needs to be used to evaluate whether the model reaches the best performance, if the model does not reach the best performance, the model is adjusted and optimized based on the calculated loss function until the model reaches the best performance, when the model reaches the best performance, the model training weight of the best model is saved, and thus the model training phase ends.

[0119] Step S104, loading the best model weight into the industrial defect detection model to be detected to obtain a target industrial defect detection model; the industrial defect detection model to be detected and the initial industrial defect detection model have the same model structure.

[0120] Wherein, the model structure of the industrial defect detection model to be detected and the initial industrial defect detection model is the same, and the industrial defect detection model to be detected also includes a visual encoder (image encoder), a text encoder (text encoder) and a text prompt generation module, the visual encoder is deployed with a visual encoding module (stage1-stage4) of a pre-trained visual-language model, and the text encoder is deployed with a text encoding module (stage1-stage4) of a pre-trained visual-language model. Figure 2 Figure 2 ​Text Transformer) in the CLIP model. Among them, the image encoder further comprises a feature mapping module, the feature mapping module comprises three mapping blocks (Projection Block) and an output mapping block (Output Projection Block), the text prompt generation module comprises a class mapping (Class Projection) module and a state mapping (State Projection) module, and the text encoder applies a deep text tuning (Deep Text Prompt) technology. Through the cooperative work of various modules, the CLIP model is adapted to the zero-shot industrial defect detection model for industrial defect detection.

[0121] In the embodiment of the present application, by loading the best model weight extracted in the training stage into the industrial defect detection model to be detected, the target industrial defect detection model containing the best model weight can be used to directly detect the images to be detected for industrial defect detection without the need to train the model again.

[0122] In step S105, the image to be detected after image preprocessing is input into the target industrial defect detection model for defect detection to obtain a target defect detection result.

[0123] Please refer to Figure 8 , Figure 8 is a schematic diagram of an industrial defect detection model test process provided by the embodiment of the present application, as Figure 8 shown, the test / detection process of the industrial defect detection is as follows: first, a test data set is obtained, it should be noted that, Figure 8 the test data set shown in Figure 7 may be the same data set or different data sets, Figure 8 the test data set in Figure 7 is mainly the data set that needs to be detected in the actual application process, and the test data set of Figure 7 may also be used for testing, and the embodiment of the present application does not limit this, and the present application takes the data set that needs to be detected in the actual application process as the test data set (i.e. the image to be detected); then, the test data set is image preprocessed, wherein the image preprocessing step is similar to the processing procedure in the training stage, and the embodiment of the present application does not repeat it here; then, the best model weight saved in the training stage is loaded into the industrial defect detection model to be detected, and the industrial defect detection model containing the best model weight is used to detect the preprocessed test data set for industrial defect detection, and the test set anomaly score and the visualization graph are output, wherein the visualization graph outputs the test set anomaly positioning graph.

[0124] It can be understood that, according to Figure 7 a flow training model, save the best performance weight of the model in the training process; in Figure 8 Only need to use Figure 7 The best performance weight saved in the training process is loaded into the model to perform industrial defect detection.

[0125] The industrial defect detection method based on the visual-language model provided in the embodiments of the present application can be regarded as a zero-shot industrial defect detection method. The method can perform defect detection on samples that have not been processed by the model, and is free from the restriction of the "one category-one model" framework, and also has good detection performance in data privacy and data security scenarios. Specifically, in the data privacy and data security scenarios, the name of the to-be-detected object cannot be known in advance in principle, and the sample of the to-be-detected object cannot be obtained for training. However, the industrial defect detection method provided in the embodiments of the present application does not need the sample of the to-be-detected object for training, and only needs to use the current public data set for model training to detect samples that have not been processed by the model. In addition, the embodiments of the present application do not need to design a text prompt based on the name of the to-be-detected object, and the embodiments of the present application can automatically generate a text prompt according to the visual features of the to-be-detected object.

[0126] The steps S101 to S105 shown in the embodiments of the present application are as follows: an initial industrial defect detection image data set is obtained; image preprocessing is performed on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set; an initial industrial defect detection model based on a pre-trained visual-linguistic model is trained according to the target industrial defect detection image data set to obtain an optimal industrial defect detection model, and an optimal model weight in the optimal industrial defect detection model is extracted; the initial industrial defect detection model includes a pre-trained visual-linguistic model, a feature mapping module and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and text semantic space, and the text prompt generation module is used to dynamically generate a text prompt; the optimal model weight is loaded into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model; a to-be-detected image after image preprocessing is input into the target industrial defect detection model for defect detection to obtain a target defect detection result. In the embodiments of the present application, the initial industrial defect detection model based on the pre-trained visual-linguistic model is trained to obtain the optimal industrial defect detection model, and the optimal model weight of the optimal industrial defect detection model is loaded into the to-be-detected industrial defect detection model, and then the industrial defect detection model containing the optimal model weight is used to detect defects of an industrial image, so that the model can process samples that have not appeared before, that is, the model can adapt to more different types of product detection requirements, and the generalization ability of the model is greatly improved. In addition, the industrial defect detection model in the embodiments of the present application includes a feature mapping module and a text prompt generation module, the feature mapping module can realize deep alignment of visual features and text semantic space, thereby enhancing the expression of the model in the form of image and text, improving the pixel-level abnormal perception ability, and through the text prompt generation module, a special text prompt is generated for each image, which can significantly enhance the generalization ability of the model, and can overcome the challenges of unknown categories, difficult state description, privacy desensitization and the like in an industrial scene, thereby breaking away from the limitation of fixed text prompts and improving the detection efficiency and accuracy of the model.

[0127] In summary, the industrial defect detection method based on the visual-linguistic model provided in the present application uses a pre-trained visual-linguistic model (CLIP) for an industrial defect detection task, and through special module design (i.e., the mapping block, the output mapping block, the category mapping block and the state mapping block designed in the embodiments of the present application), the CLIP is better adapted to the defect detection task, and the CLIP realizes pixel-level prediction. Therefore, the key advantage of the embodiments of the present application lies in that:

[0128] (1) A projection block and an output projection block are designed on the visual encoder to realize the deep alignment of visual features and text semantic space through the cooperative work of the two modules. The projection block is used for three-layer linear transformation and residual fusion of the output features in the intermediate stage of visual feature extraction, to improve the alignment ability of image-text features. The output projection block acts on the final stage of visual feature extraction, outputs global visual features and serves as the input of subsequent modules, to optimize the terminal representation. The two modules work together to enhance the expression of image-text fusion of CLIP and improve the pixel-level anomaly perception ability.

[0129] (2) A class projection module and a state projection module are proposed on the visual-linguistic model to realize a dynamic text prompt generation mechanism through the cooperative work of the two modules. By using CLS token to replace artificial class names and patch token to replace artificial state words, a dedicated text prompt is generated for each image, which significantly enhances the model generalization ability and overcomes the challenges of unknown categories, difficult state description and privacy desensitization in industrial scenarios, and breaks away from the restrictions of fixed text prompts.

[0130] Please refer to Figure 9 The embodiment of the application also provides an industrial defect detection device 900 based on a visual-linguistic model, which can realize the industrial defect detection method based on a visual-linguistic model. The device includes the following modules:

[0131] An image data set acquisition module 901 is configured to acquire an initial industrial defect detection image data set.

[0132] An image preprocessing module 902 is configured to perform image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set.

[0133] A model training module 903 is configured to train an initial industrial defect detection model based on a pre-trained visual-linguistic model according to the target industrial defect detection image data set to obtain an optimal industrial defect detection model and extract optimal model weights in the optimal industrial defect detection model. The initial industrial defect detection model includes the pre-trained visual-linguistic model, a feature mapping module and a text prompt generation module. The feature mapping module is configured to realize the alignment of visual features and text semantic space, and the text prompt generation module is configured to dynamically generate a text prompt.

[0134] The best model weight loading module 904 is configured to load the best model weight into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model;

[0135] The defect detection application module 905 is configured to input a to-be-detected image after image preprocessing into the target industrial defect detection model for defect detection to obtain a target defect detection result.

[0136] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0137] The embodiment of the application further provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0138] It can be understood that the content in the above method embodiments is applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0139] Please refer to Figure 10 , Figure 10 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device includes:

[0140] The processor 1001 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided in the embodiments of the application.

[0141] The memory 1002 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of the application are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the above method of the embodiments of the application.

[0142] The input / output interface 1003 is configured to realize information input and output.

[0143] The communication interface 1004 is configured to realize communication interaction between the device and other devices, and the communication can be realized through a wired manner (for example, a USB, a network cable and the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth and the like).

[0144] The bus 1005 is configured to transmit information between various components (for example, the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004) of the device.

[0145] The processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 are connected to each other through the bus 1005 to realize communication connection between the device.

[0146] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method.

[0147] It can be understood that the contents in the above method embodiments are all applicable to the storage medium embodiment, the storage medium embodiment specifically realizes the functions of the above method embodiments, and the same beneficial effects as the above method embodiments are achieved.

[0148] The embodiment of the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the method.

[0149] It can be understood that the contents in the above method embodiments are all applicable to the program product embodiment, the program product embodiment specifically realizes the functions of the above method embodiments, and the same beneficial effects as the above method embodiments are achieved.

[0150] The memory is a non-transitory computer readable storage medium, and can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, for example, at least one magnetic disk storage device, a flash memory device or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and the remote memory can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0151] The embodiment of the application provides an industrial defect detection method and device based on a visual-language model, an electronic device, a storage medium and a program product. The method comprises the following steps: obtaining an initial industrial defect detection image data set; performing image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set; training an initial industrial defect detection model based on a pre-trained visual-language model according to the target industrial defect detection image data set to obtain an optimal industrial defect detection model, and extracting optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model comprises a pre-trained visual-language model, a feature mapping module and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and a text semantic space, and the text prompt generation module is used to dynamically generate a text prompt; loading the optimal model weights into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model; inputting a to-be-detected image subjected to image preprocessing into the target industrial defect detection model to perform defect detection, and obtaining a target defect detection result. According to the embodiment of the application, the initial industrial defect detection model based on the pre-trained visual-language model is trained to obtain the optimal industrial defect detection model, and the optimal model weights of the optimal industrial defect detection model are loaded into the to-be-detected industrial defect detection model, and then the industrial defect detection model containing the optimal model weights is used to perform defect detection on the industrial image, so that the model can process samples that have not appeared before, that is, the model can adapt to more product detection requirements of different categories, and the generalization ability of the model is greatly improved. In addition, the industrial defect detection model of the embodiment of the application comprises the feature mapping module and the text prompt generation module, the feature mapping module can realize deep alignment of visual features and a text semantic space, thereby enhancing the expression of the model, improving the pixel-level abnormal perception ability, and generating exclusive text prompts for each image through the text prompt generation module, which can significantly enhance the generalization ability of the model, overcome the challenges of unknown categories, difficult state description and privacy desensitization in the industrial scene, thereby breaking away from the limitation of fixed text prompts, and improving the detection efficiency and accuracy of the model.

[0152] The embodiments described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0153] Those skilled in the art can understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the application, and can include more or fewer steps than the drawings, or combine certain steps or different steps.

[0154] The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0155] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the functional modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combinations thereof.

[0156] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims of the foregoing drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so

[0157] It should be understood that in this application, "at least one" means one or more, "multiple" means two or more. "And / or", used to describe the relationship between associated objects, means that there can be three relationships, for example, "A and / or B" can mean: only A, only B, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are a kind of "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other manners. For example, the apparatus embodiments described above are merely illustrative, for example, the division of the above units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, apparatuses or units, and can be electrical, mechanical or other forms.

[0159] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0160] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0161] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0162] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, but this does not limit the scope of the rights of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the rights of the embodiments of the present application.< / state> < / class> < / state> < / class> < / class> < / class> < / class>

Claims

1. A method for industrial defect detection based on vision-language model, characterized in that, The method comprises the following steps: obtaining an initial industrial defect detection image data set; performing image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set; training an initial industrial defect detection model based on a pre-trained visual-linguistic model according to the target industrial defect detection image data set to obtain an optimal industrial defect detection model, and extracting optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model comprises the pre-trained visual-linguistic model, a feature mapping module, and a text prompt generation module, the feature mapping module is used to realize alignment of visual features and text semantic spaces, and the text prompt generation module is used to dynamically generate text prompts; loading the optimal model weights into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model; inputting a to-be-detected image subjected to image preprocessing into the target industrial defect detection model for defect detection to obtain a target defect detection result.

2. The method of claim 1, wherein, The image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set comprises: performing random cropping processing on the initial industrial defect detection image data set to obtain a first training image data set; performing bicubic interpolation scaling processing on the first training image data set to obtain a second training image data set; performing image channel unification processing on the second training image data set to obtain a third training image data set; performing tensor conversion processing on the third training image data set to obtain a fourth training image data set; performing data standardization processing on the fourth training image data set to obtain the target industrial defect detection image data set.

3. The method of claim 1, wherein, The training of the initial industrial defect detection model based on the pre-trained visual-linguistic model according to the target industrial defect detection image data set to obtain the optimal industrial defect detection model and the extraction of the optimal model weights in the optimal industrial defect detection model comprise: inputting the target industrial defect detection image data set into a visual encoder in the initial industrial defect detection model, and extracting target visual feature data corresponding to the target industrial defect detection image data set through a feature mapping module in the visual encoder; wherein the target visual feature data comprises first visual feature data, second visual feature data, third visual feature data, and fourth visual feature data; inputting the fourth visual feature data into a text prompt generation module, and generating a target text prompt according to the target visual feature data through the text prompt generation module; inputting the target text prompt into a text encoder in the initial industrial defect detection model, and generating target text feature data according to the target text prompt through the text encoder; performing global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score; performing local score calculation on the target visual feature data and the target text feature data to obtain a local anomaly score; performing model loss calculation on the global anomaly score and the local anomaly score to obtain a total model training loss; performing back propagation and optimization on the initial industrial defect detection model according to the total model training loss until model training is completed; performing quality evaluation on the initial industrial defect detection model after training, and if the model weight of the initial industrial defect detection model after training reaches an optimal performance threshold, taking the initial industrial defect detection model after training as the optimal industrial defect detection model, and extracting the optimal model weight in the optimal industrial defect detection model.

4. The method of claim 3, wherein, The global score calculation on the target visual feature data and the target text feature data to obtain a global anomaly score comprises: performing cosine similarity calculation on the first visual feature data and the target text feature data to obtain a first cosine similarity; performing cosine similarity calculation on the second visual feature data and the target text feature data to obtain a second cosine similarity; performing cosine similarity calculation on the third visual feature data and the target text feature data to obtain a third cosine similarity; performing cosine similarity calculation on the fourth visual feature data and the target text feature data to obtain a fourth cosine similarity; performing sum processing on the first cosine similarity, the second cosine similarity, the third cosine similarity, and the fourth cosine similarity to obtain a target cosine similarity; performing average calculation processing on the target cosine similarity to obtain the global anomaly score.

5. The method of claim 3, wherein, The model loss calculation on the global anomaly score and the local anomaly score to obtain a total model training loss comprises: performing loss calculation on the global anomaly score and a real global label to obtain a global loss; performing up-sampling processing on the local anomaly score, and performing loss calculation on the up-sampling processed local anomaly score and a real local label to obtain a local loss; determining the total model training loss according to the global loss and the local loss.

6. The method of claim 3, wherein, The method further comprises: if the model weight of the initial industrial defect detection model after training does not reach the optimal performance threshold, returning to perform the step of inputting the target industrial defect detection image data set into the visual encoder in the initial industrial defect detection model, extracting the target visual feature data corresponding to the target industrial defect detection image data set through the feature mapping module in the visual encoder, until the model weight of the initial industrial defect detection model after training reaches the optimal performance threshold, to obtain the optimal industrial defect detection model, and extract the optimal model weight in the optimal industrial defect detection model.

7. An industrial defect detection apparatus based on a vision-linguistic model, characterized by, The device comprises the following modules: an image data set acquisition module configured to acquire an initial industrial defect detection image data set; an image preprocessing module configured to perform image preprocessing on the initial industrial defect detection image data set to obtain a target industrial defect detection image data set; The model training module is configured to train an initial industrial defect detection model based on a pre-trained visual-linguistic model according to the target industrial defect detection image dataset, obtain an optimal industrial defect detection model, and extract optimal model weights in the optimal industrial defect detection model; the initial industrial defect detection model comprises the pre-trained visual-linguistic model, a feature mapping module, and a text prompt generation module; the feature mapping module is configured to realize alignment of visual features and a text semantic space; and the text prompt generation module is configured to dynamically generate a text prompt. The optimal model weight loading module is configured to load the optimal model weights into a to-be-detected industrial defect detection model to obtain a target industrial defect detection model; the to-be-detected industrial defect detection model has the same model structure as the initial industrial defect detection model. The defect detection application module is configured to input a to-be-detected image after image preprocessing into the target industrial defect detection model for defect detection to obtain a target defect detection result.

8. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

Images