Zero sample image anomaly detection positioning method based on pre-training vision-language model

Abstract

The invention provides a zero sample image anomaly detection and positioning method based on a pre-training vision-language model, relates to the technical field of image anomaly detection and positioning, and aims to train a learnable adapter and a learnable text prompt by using a universal vision anomaly detection data set. A normal state text prompt and an abnormal state text prompt are constructed, and a normal state text representation and an abnormal state text representation are respectively obtained. In the aspect of image features, characterization of different levels of an image in a picture encoder is obtained, and image features of neighbor information of different scales are aggregated through window sizes of different scales and are adjusted by an adapter. And finally, calculating the similarity between the text-image features, and obtaining a probability score that the image is classified as abnormal by using a normalization mode. According to the invention, the general abnormal data set is utilized to train the learnable text prompt and the learnable adapter, so that the problem of how to carry out image anomaly detection when the training sample cannot be acquired is solved, the pre-training model can adapt to the target detection data set, and zero sample detection and positioning are effectively carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of image anomaly detection and positioning, and more specifically, to a zero-sample image anomaly detection and positioning method based on a pre-trained visual-language model. Background Art

[0002] The purpose of visual anomaly detection and localization is to detect abnormal images and locate the image areas with abnormalities. It has a wide range of application scenarios. For example, in industrial production, visual anomaly detection and localization can detect defects in production components and control the qualified rate of component production; in medical diagnosis, visual anomaly detection and localization can assist in disease diagnosis and determine the specific location of the lesion.

[0003] In actual application scenarios, abnormal images may have various shapes, colors, and textures. On the one hand, it is difficult to collect comprehensive abnormal samples, and labeling each abnormal sample requires a lot of time and manpower costs. Therefore, the mainstream visual anomaly detection and positioning method is an unsupervised detection method, which only uses normal samples for training and detection. Unsupervised detection methods can perform anomaly detection and positioning through reconstruction, falsification prediction, feature storage comparison, etc. However, this method trains a model for each type of detection object separately, and needs to collect each type of normal sample data for training, which requires a large amount of manpower, time, and storage costs. On the other hand, unsupervised training may lead to situations where training samples cannot be obtained. For example, in medical diagnosis applications, there is a problem that the inspection result images cannot be collected due to patient privacy. In the early stages of industrial production, a large number of component photos cannot be obtained. Therefore, in such scenarios, a zero-sample image anomaly detection method urgently needs to be proposed.

[0004] In the zero-shot image anomaly detection scenario, it is impossible to obtain data samples of the target domain of the detection object. However, due to the widespread existence of data on the Internet, we can always obtain some general visual anomaly detection datasets, such as some public industrial datasets. To perform zero-shot anomaly detection and localization in the target detection domain, the generalization performance of the model used for anomaly detection is very important. If a new model is designed from scratch and trained using a general dataset, the model's detection performance in the target detection domain is often poor. This is because the model designed from scratch does not have any knowledge of the target detection domain and will overfit on the general dataset, while the detection performance in the target domain is poor. Based on this consideration, it is important to select a large pre-trained model base with high robustness. At the same time, how to use a general dataset so that the pre-trained model can adapt to the target detection dataset and effectively perform zero-shot detection and localization is a technical problem that needs to be solved in this field. Summary of the invention

[0005] In order to solve the problem of how to perform zero-shot image anomaly detection and positioning under application requirements where sample image data is difficult to collect, the present invention proposes a zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model. By using a general visual anomaly detection dataset, the pre-trained model can be adapted to the target detection dataset, thereby improving the effect and performance of zero-shot image anomaly detection and positioning.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] A zero-shot image anomaly detection and positioning method based on a pre-trained vision-language model, comprising:

[0008] Using a general anomaly detection dataset to train a learnable adapter and a learnable text prompt, and obtaining a trained learnable adapter and a learnable text prompt;

[0009] Using the trained learnable text prompts, construct normal state text prompts and abnormal state text prompts respectively;

[0010] The text encoder of the pre-trained visual-language model processes the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and outputs the normal state text representation and the abnormal state text representation;

[0011] The layers of the image encoder are divided into different modules, and the image encoder is used to extract the image block feature sequences of the image to be detected in different modules;

[0012] Set the window sizes of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales;

[0013] Use the trained adapter to adjust image features of all image patches and scales;

[0014] Calculating the similarity between the image features and the text features, wherein the text features include normal state text representations and abnormal state text representations; normalizing the similarity to obtain a probability score for classifying the image as abnormal;

[0015] Whether the image is abnormal is determined based on the probability score of the image being classified as abnormal, and the probability score of the image being classified as abnormal is reorganized and then linearly interpolated. According to the linear interpolation result, the abnormal pixels of the image are located.

[0016] Preferably, the learnable adapter includes three linear projection learnable self-attention layers and an output layer, and the learnable weight parameters of the learnable adapter include W Q , W K , W V and W O ,in, are the weight parameters of the self-attention layer that can be learned by three linear projections, W O is the weight parameter of the output layer;

[0017] The learnable text prompt is represented as:

[0018]

[0019] in, Represents the M1th learnable word embedding prompt representation. The learnable text prompt is to map the text into the representation of the word vector. The representation corresponding to a text word is a word vector, and the word vectors at different positions are the learnable text prompt parameters.

[0020] Preferably, before training the learnable adapter and the learnable text prompt using the general anomaly detection dataset, the following steps are performed:

[0021] Use learnable text prompts to construct normal state text prompts and abnormal state text prompts respectively;

[0022] The text encoder of the pre-trained visual-language model processes the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and outputs the normal state text representation and the abnormal state text representation;

[0023] The image encoder is divided into different modules, and the image encoder is used to extract the image block feature sequences of the images in the general anomaly detection dataset in different modules;

[0024] Set the window sizes of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales;

[0025] Use adapters to adjust image features of all image patches and scales;

[0026] Calculating the similarity between the image feature and the text feature, wherein the text feature includes a normal state text representation and an abnormal state text representation;

[0027] The similarities are normalized to obtain the probability scores of images being classified as anomalies in the general anomaly detection dataset.

[0028] Preferably, the normal state text prompt P is constructed n The expression is:

[0029] without defects

[0030] Abnormal status text prompt of the construction P a The expression is:

[0031] with defect

[0032] in, They represent the M1th learnable normal word embedding cue representation and abnormal word embedding cue representation respectively;

[0033] The text encoder of the pre-trained vision-language model that processes the normal state learnable text prompt and the abnormal state learnable text prompt is f T (·), the output normal state text representation T n and abnormal state text representation T a The expressions satisfied are:

[0034] T n =f T (P n ),T a =f r (P a ).

[0035] According to the above technical means, using a text encoder that can learn text prompts and a pre-trained visual-language model, normal and abnormal text representation features can be obtained, which is convenient for similarity calculation with subsequent image features.

[0036] Preferably, the image encoder processes the images in the general anomaly detection dataset to obtain the image semantic information representation [CLS] and the image block representation, which are generally represented as follows:

[0037]

[0038] in, is the image semantic information representation, a i is the image block feature representation, i∈1,2,…,N, N represents the number of image blocks; the hierarchy of the image encoder is divided into l different modules, and the output features of all modules are extracted, the expression is:

[0039]

[0040] a l Represents the image block feature sequence output by the lth module of the picture encoder, including: image semantic information representation And the remaining N image blocks are represented by the image blocks output by the lth module

[0041] Preferably, the window sizes of different scales are set, and the neighbor information of each image block feature sequence at different scales is aggregated to obtain the image features of all image blocks and all scales in the following process:

[0042] Get the width and height of the feature map The one-dimensional image block feature sequence is converted into a two-dimensional image feature map through the reshape() function

[0043] Assume the number of scales is r, the two-dimensional image feature map The image block feature at any position (h,w) is Its neighbor information is: Average pooling is used to aggregate the neighbor information of each image block feature sequence to obtain

[0044] Will Use the reshape() function to reorganize into a one-dimensional sequence Image semantic information representation Stitching to obtain aggregated image features The image features of different levels and scales are expressed as:

[0045] Through the above technical means, richer image feature representation can be obtained.

[0046] Preferably, the process of adjusting the image features of all image blocks and all scales using the adapter is:

[0047] Using three linear projections, the self-attention layer can be learned to transform the image features c lr Mapped to query vector Q lr , key vector K lr and value vector V lr , the expression is:

[0048] Q lr =c lr ·W Q ,K lr =c lr ·W K ,V lr =c lr ·W V

[0049] Use the self-attention mechanism and output weights through the output layer Mapped to the original dimension, the adjusted image features are obtained, expressed as:

[0050]

[0051] Image features at different levels and scales share learnable adapter weight parameters W Q , W K , W W and W O ;

[0052] The adjusted image features are passed through the projection layer f of the image encoder p (·), get image features:

[0053]

[0054] Image features at different levels and scales are represented as: J = {I lr |l,r}}.

[0055] Through the above technical means, image features can be mapped into the text-image joint feature space.

[0056] Preferably, the process of calculating the similarity between the image feature and the text feature is:

[0057] Calculate the similarity between a single image feature and a text feature, including calculating the similarity between a single image feature and a normal state text representation, and calculating the similarity between a single image feature and an abnormal state text representation. The expression is:

[0058]

[0059] Calculate the similarity between a single image feature and the normal state text representation, expression:

[0060]

[0061] Among them, <·> represents the cosine similarity, and the similarity between a single image feature and the normal state text representation and the abnormal state text representation is generally expressed as Sum all l and r to get the overall score Get similarity scores at different levels and scales; normalize the similarity to obtain the probability score of classifying the image as abnormal:

[0062]

[0063] Among them, p CLS represents the probability score of image-level classification as abnormal, is the probability score of classifying N image patches as abnormalities.

[0064] Preferably, when using a general anomaly detection dataset to train a learnable adapter and a learnable text prompt, the label of the image anomaly in the general anomaly detection dataset is y, y∈{0,1}, and the Focal loss is used to construct the loss of image-level anomaly classification, which is expressed as:

[0065] L global =FocalLoss(p CLS ,y)

[0066] Among them, p CLS Represents the probability score of image-level classification as abnormal;

[0067] Focal loss and L1 loss are used to construct the loss for fine-grained image anomaly localization. The expression is:

[0068]

[0069] in, Represents the reorganization value, which is obtained by the probability score P of the abnormality of the image in different modules in the general anomaly detection dataset through the reshape() function operation, p CLS and P are related to the learnable weight parameters of the learnable adapter including W Q , W K , W W and W O , learnable text prompt parameters related; Represents the abnormal pixel mask label corresponding to the abnormal image in the general anomaly detection dataset Downsampling of

[0070] The overall training loss L for training the learnable adapter and learnable textual prompts using the general anomaly detection dataset is:

[0071] L=L global +L local

[0072] When training the learnable adapter and the learnable text prompt, the overall training loss L is used as the training loss function to train the learnable adapter and the learnable text prompt. When the loss function converges, the training is completed, and the learnable weight parameters and the learnable text prompt parameters of the learnable adapter are fixed to obtain the trained learnable adapter and the learnable text prompt.

[0073] According to the above technical means, a learnable adapter and a learnable text prompt are trained using a general anomaly detection dataset, so that the trained learnable adapter and learnable text prompt can be applied to anomaly detection and positioning of zero-sample images.

[0074] Preferably, the trained learnable adapter and the learnable text prompt are used for abnormal detection and positioning of the image to be detected, and the probability score of the image to be detected being classified as abnormal is obtained, and the probability score of the image being classified as abnormal includes the probability score p of the image level being classified as abnormal. CLS The probability score P of the image block being classified as abnormal, the higher the probability score, the greater the possibility of image abnormality, when the probability score is greater than the first set standard, the image is judged to be abnormal; the probability score P of the image block being classified as abnormal is reorganized by the reshape() function to obtain the reorganized value Then reorganize the value Linear interpolation to A pixel mask of a certain size is formed, and position codes in the pixel mask that are larger than a second set standard are abnormal image areas.

[0075] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0076] The present invention proposes a zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model. First, a general visual anomaly detection dataset is used to train a learnable adapter and a learnable text prompt. In terms of text features, the trained learnable text prompt is used to construct a normal state text prompt and an abnormal state text prompt, and the normal state text representation and the abnormal state text representation are obtained respectively. In terms of image features, the representations of different levels of the image in the picture encoder are obtained, and the image features of neighbor information of different scales are aggregated through window sizes of different scales. Then, the adapter is used to adjust the image features of different levels and scales to map them to the common semantic space of text-image. Finally, the similarity between text-image features is calculated, and the probability score of the image classification as abnormal is obtained by normalization. The present invention uses a general abnormal dataset to train a learnable text prompt and a learnable adapter to solve the problem of how to perform image anomaly detection when the training sample cannot be obtained, so that the pre-trained model can be adapted to the target detection dataset and effectively perform zero-shot detection and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram showing a flow chart of a zero-sample image anomaly detection and positioning method based on a pre-trained visual-language model proposed in an embodiment of the present invention;

[0078] Figure 2 A text-image processing flow framework diagram for zero-shot image anomaly detection and localization based on a pre-trained visual-language model proposed in an embodiment of the present invention is shown;

[0079] Figure 3 A schematic diagram showing the aggregation of neighbor information of image features of different scales proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The drawings are for illustrative purposes only and should not be construed as limiting the present patent;

[0081] In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the actual size;

[0082] It is understandable to those skilled in the art that descriptions of certain well-known contents in the drawings may be omitted.

[0083] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0084] The positional relationships described in the drawings are only for illustrative purposes and should not be construed as limiting the present patent;

[0085] Example 1

[0086] This embodiment proposes a zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model. Figure 1 The method comprises the following steps:

[0087] S1: Use the general anomaly detection dataset to train the learnable adapter and the learnable text prompt to obtain the trained learnable adapter and the learnable text prompt;

[0088] S2: Use the trained learnable text prompts to construct normal state text prompts and abnormal state text prompts respectively;

[0089] S3: Use the text encoder of the pre-trained vision-language model to process the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and output the normal state text representation and the abnormal state text representation;

[0090] S4: Divide the hierarchy of the image encoder into different modules, and use the image encoder to extract image block feature sequences in different modules of the image to be detected;

[0091] S5: Set the window sizes of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales;

[0092] S6: Use the trained adapter to adjust the image features of all image patches and all scales;

[0093] S7: Calculate the similarity between the image feature and the text feature, wherein the text feature includes a normal state text representation and an abnormal state text representation; normalize the similarity to obtain a probability score of the image block classification abnormality;

[0094] S8: Determine whether the image is abnormal based on the probability score of the image being classified as abnormal, reorganize the probability score of the image being classified as abnormal and perform linear interpolation, and locate the abnormal pixels of the image based on the linear interpolation result.

[0095] The method proposed in this embodiment uses a text encoder to construct a learnable text prompt to obtain normal and abnormal text prompts, inputs the image to be detected into a picture encoder to obtain image features at different levels, and then obtains image features at different scales through neighbor information aggregation at different scales, and then adjusts the image features through an adapter. Finally, by comparing the similarity of text features and image features, and obtaining the probability score of classifying the image as abnormal by normalization, the problem of how to detect image anomalies when training samples cannot be obtained is solved, so that the pre-trained model can be adapted to the target detection data set, and effectively perform zero-sample detection and positioning.

[0096] Example 2

[0097] In this embodiment, the learnable adapter includes three linear projection learnable self-attention layers and an output layer, and the learnable weight parameters of the learnable adapter include W Q , W K , W V and W O ,in, are the weight parameters of the self-attention layer that can be learned by three linear projections, W O is the weight parameter of the output layer;

[0098] The learnable text prompt is represented as:

[0099]

[0100] in, Represents the M1th learnable word embedding prompt representation. The learnable text prompt is to map the text into the representation of the word vector. The representation corresponding to a text word is a word vector, and the word vectors at different positions are the learnable text prompt parameters.

[0101] Prior to training the learnable adapter and learnable textual prompts using the Generic Anomaly Detection Dataset, the following steps are also performed:

[0102] S101. Use learnable text prompts to construct normal state text prompts and abnormal state text prompts respectively;

[0103] S102. Using the pre-trained visual-language model text encoder to process the normal state learnable text prompt and the abnormal state learnable text prompt, and output the normal state text representation and the abnormal state text representation;

[0104] S103. Divide the hierarchy of the image encoder into different modules, and use the image encoder to extract image block feature sequences in different modules of the image in the general anomaly detection dataset;

[0105] S104. Setting window sizes of different scales, aggregating neighbor information of each image block feature sequence at different scales, and obtaining image features of all image blocks and all scales;

[0106] S105. Using the adapter to adjust image features of all image blocks and all scales;

[0107] S106. Calculate the similarity between the image feature and the text feature, wherein the text feature includes a normal state text representation and an abnormal state text representation;

[0108] S107. Normalize the similarity to obtain a probability score of classifying an image as an anomaly in a general anomaly detection dataset.

[0109] See also Figure 2 In this embodiment, the normal state text prompt P is constructed n The expression is:

[0110] without defects

[0111] Abnormal status text prompt of the construction P a The expression is:

[0112] with defect

[0113] in, They represent the M1th learnable normal word embedding cue representation and abnormal word embedding cue representation respectively;

[0114] The text encoder of the pre-trained vision-language model that processes the normal state learnable text prompt and the abnormal state learnable text prompt is f T (·), the text encoder already exists in the CLIP model, which is a Transformer type text encoder, a general encoder structure, and the output normal state text representation T n and abnormal state text representation T a The expressions satisfied are:

[0115] T n =f T (P n ),T a =f T (P a ).

[0116] By using a text encoder that can learn text prompts and a pre-trained vision-language model, normal and abnormal text representation features can be obtained, which facilitates similarity calculation with subsequent image features.

[0117] In this embodiment, the image encoder already exists in the CLIP model, which is a Vision Transformer type image encoder, which is a general encoder structure. The image encoder processes the images in the general anomaly detection dataset to obtain the image semantic information representation [CLS] and the image block representation, which is represented as a whole:

[0118]

[0119] in, is the image semantic information representation, a i is the image block feature representation, i∈1,2,…,N, N represents the number of image blocks; the Vision Transformer type image encoder has many layers. Because different levels of features contain features of different granularity, the levels of the image encoder are divided into l different modules, and the output features of all modules are extracted. The expression is:

[0120]

[0121] a l Represents the image block feature sequence output by the lth module of the picture encoder, including: image semantic information representation And the remaining N image blocks are represented by the image blocks output by the lth module

[0122] a l Represents the image block feature sequence output by the lth module of the picture encoder, including: image semantic information representation And the remaining N image blocks are represented by the image blocks output by the lth module In this embodiment, all layers of the image encoder are divided into four modules, and the output features of all modules are extracted respectively, and the following can be obtained:

[0123]

[0124] Then, by setting the window sizes of different scales and aggregating the neighbor information of each image block feature sequence at different scales, a richer image feature representation can be obtained. The process of obtaining image features of all image blocks and all scales is as follows:

[0125] Get the width and height of the feature map The one-dimensional image block feature sequence is converted into a two-dimensional image feature map through the reshape() function

[0126] Assume that the number of scales is r. In this embodiment, the neighbor information of three different scales r∈{1,3,5} is aggregated, and the two-dimensional image feature map The image block feature at any position (h,w) is Its neighbor information is: like Figure 3 As shown, average pooling is used to aggregate the neighbor information of each image block feature sequence to obtain

[0127] Will Use the reshape() function to reorganize into a one-dimensional sequence Image semantic information representation Stitching to obtain aggregated image features The image features of different levels and scales are expressed as:

[0128] The learnable adapter adjusts image features at different levels and scales, and can map image features to the text-image joint feature space, making it more suitable for anomaly detection and localization tasks. The adapter consists of a learnable self-attention layer. The process of using the trained adapter to adjust image features of all image blocks and all scales is as follows:

[0129] Using three linear projections, the self-attention layer can be learned to transform the image features c lr Mapped to query vector Q lr , key vector K lr and value vector V lr , the expression is:

[0130] Q lr =c lr ·W Q ,K lr =c lr ·W K ,V lr =c lr ·W V

[0131] Use the self-attention mechanism and output weights through the output layer Mapped to the original dimension, the adjusted image features are obtained, expressed as:

[0132]

[0133] Image features at different levels and scales share learnable adapter weight parameters W Q , W K , W W and W O ;

[0134] The adjusted image features are passed through the projection layer f of the image encoder p (·), get image features:

[0135]

[0136] Image features at different levels and scales are represented as:

[0137] The process of calculating the similarity between image features and text features is:

[0138] Calculate the similarity between a single image feature and a text feature, including calculating the similarity between a single image feature and a normal state text representation, and calculating the similarity between a single image feature and an abnormal state text representation. The expression is:

[0139]

[0140] Calculate the similarity between a single image feature and the normal state text representation, expression:

[0141]

[0142] Among them, <·> represents the cosine similarity, and the similarity between a single image feature and the normal state text representation and the abnormal state text representation is generally expressed as Sum all l and r to get the overall score Get similarity scores at different levels and scales; normalize the similarity to obtain the probability score of classifying the image as abnormal:

[0143]

[0144] Among them, p CLS represents the probability score of image-level classification as abnormal, is the probability score of classifying N image patches as abnormalities.

[0145] The general anomaly detection dataset used in this embodiment is the MVTec AD dataset, which contains 15 kinds of objects, 467 normal images, and 1258 abnormal images. The zero-sample target datasets used for detection include the industrial dataset Visa (962 normal samples, 1200 abnormal samples), BTAD (451 normal samples, 290 abnormal samples), DAGM (6996 normal samples, 1054 abnormal samples), DTD-Synthetic (357 normal samples, 947 abnormal samples) and the medical dataset BrainMRI (98 normal samples, 155 abnormal samples), Br35H (1500 normal samples, 1500 abnormal samples), ISIC (379 abnormal samples), CVC-ColonDB (380 abnormal samples), CVC-ClinicDB (612 abnormal samples).

[0146] When using the general anomaly detection dataset to train the learnable adapter and learnable text prompt, the label of the image anomaly in the general anomaly detection dataset is y, y∈{0,1}, and the focal loss is used to construct the loss of image-level anomaly classification, which is expressed as:

[0147] L global =FocalLoss(p CLS ,y)

[0148] Among them, p CLS Represents the probability score of image-level classification as abnormal;

[0149] Focal loss and L1 loss are used to construct the loss for fine-grained image anomaly localization. The expression is:

[0150]

[0151] in, Represents the reorganization value, which is obtained by the probability score P of N image blocks in the general anomaly detection dataset being classified as abnormal through the reshape() function operation, p CLS and P are related to the learnable weight parameters of the learnable adapter including W Q , W K , W V and W O , learnable text prompt parameters related; Represents the abnormal pixel mask label corresponding to the abnormal image in the general anomaly detection dataset Downsampling of

[0152] The overall training loss L for training the learnable adapter and learnable textual prompts using the general anomaly detection dataset is:

[0153] L=L global +L local

[0154] When training the learnable adapter and the learnable text prompt, the overall training loss L is used as the training loss function to train the learnable adapter and the learnable text prompt. When the loss function converges, the training is completed, and the learnable weight parameters and the learnable text prompt parameters of the learnable adapter are fixed to obtain the trained learnable adapter and the learnable text prompt.

[0155] During the training process, the components of the loss function include p CLS and P, p CLS and P need to contain learnable parameters of the learnable adapter and learnable text prompts. During the training process, W will be adjusted Q , W K , WW and W o , learnable text prompt parameters. The learnable adapter and learnable text prompt are trained using a general anomaly detection dataset, so that the trained learnable adapter and learnable text prompt can be applied to anomaly detection and localization of zero-shot images.

[0156] Example 3

[0157] The trained learnable adapter and learnable text prompt are used for abnormal detection and positioning of the image to be detected, as described in Example 1. For the image to be detected, the trained learnable text prompt is first used to construct the normal state text prompt and the abnormal state text prompt respectively, and then the text encoder of the pre-trained visual-language model is used to process the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and the normal state text representation and the abnormal state text representation are output. The hierarchy of the picture encoder is divided into different modules, and the image encoder is used to extract the image block feature sequence of the image to be detected in different modules, set the window size of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales. Then, the trained adapter is used to adjust the image features of all image blocks and all scales. Calculate the similarity between the image feature and the text feature, and the text feature includes the normal state text representation and the abnormal state text representation; normalize the similarity to obtain the probability score of classifying the image as abnormal.

[0158] In this embodiment, the probability score of an image being classified as abnormal includes the probability score p of the image level being classified as abnormal. CLS and the probability score P of the image block being classified as abnormal,

[0159] The higher the probability score, the greater the possibility of image abnormality. When the probability score is greater than the first set standard, the image is judged to be abnormal. The probability score P of the image block classified as abnormal is reorganized by the reshape() function to obtain the reorganized value Then reorganize the value Linear interpolation to A pixel mask of a certain size is formed, and a position code in the pixel mask that is larger than the second set standard is an abnormal image area, thereby locating the image abnormality.

[0160] The embodiments are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model, characterized in that: include: Using a common anomaly detection dataset to train a learnable adapter and learnable textual cues; Using the trained learnable text prompts, construct normal state text prompts and abnormal state text prompts respectively; The text encoder of the pre-trained visual-language model processes the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and outputs the normal state text representation and the abnormal state text representation; The layers of the image encoder are divided into different modules, and the image encoder is used to extract the image block feature sequences of the image to be detected in different modules; Set the window sizes of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales; Use the trained adapter to adjust image features of all image patches and scales; Calculating the similarity between the image features and the text features, wherein the text features include normal state text representations and abnormal state text representations; normalizing the similarity to obtain a probability score for classifying the image as abnormal; Whether the image is abnormal is determined based on the probability score of the image being classified as abnormal, and the probability score of the image being classified as abnormal is reorganized and then linearly interpolated. According to the linear interpolation result, the abnormal pixels of the image are located.

2. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 1, characterized in that: The learnable adapter includes three linear projection learnable self-attention layers and an output layer. The learnable weight parameters of the learnable adapter include W Q , W K , W V and W O ,in, are the weight parameters of the self-attention layer that can be learned by three linear projections, W O is the weight parameter of the output layer; The learnable text prompt is represented as: in, Represents the M1th learnable word embedding prompt representation. The learnable text prompt is to map the text into the representation of the word vector. The representation corresponding to a text word is a word vector, and the word vectors at different positions are the learnable text prompt parameters.

3. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 2, characterized in that: Before training the learnable adapter and learnable textual hints using the Generic Anomaly Detection dataset, perform the following steps: Use learnable text prompts to construct normal state text prompts and abnormal state text prompts respectively; The text encoder of the pre-trained visual-language model processes the normal state learnable text prompt and the abnormal state learnable text prompt respectively, and outputs the normal state text representation and the abnormal state text representation; The image encoder is divided into different modules, and the image encoder is used to extract the image block feature sequences of the images in the general anomaly detection dataset in different modules; Set the window sizes of different scales, aggregate the neighbor information of each image block feature sequence at different scales, and obtain the image features of all image blocks and all scales; Use adapters to adjust image features of all image patches and scales; Calculating the similarity between the image feature and the text feature, wherein the text feature includes a normal state text representation and an abnormal state text representation; The similarities are normalized to obtain the probability scores of images being classified as anomalies in the general anomaly detection dataset.

4. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 3, characterized in that: Normal status text prompt for the build n The expression is: without defect Abnormal status text prompt of the construction P a The expression is: with defect in, They represent the M1th learnable normal word embedding cue representation and abnormal word embedding cue representation respectively; The text encoder of the pre-trained vision-language model that processes the normal state learnable text prompt and the abnormal state learnable text prompt is f T (·), the output normal state text representation T n and abnormal state text representation T a The expressions satisfied are: T n =f T (P n ),T a =f T (P a )。 5. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 4, characterized in that: The image encoder processes the images in the general anomaly detection dataset to obtain the image semantic information representation [CLS] and image block representation, which are expressed as follows: in, is the image semantic information representation, a i is the image block feature representation, i∈1,2,…,N, N represents the number of image blocks; the hierarchy of the image encoder is divided into l different modules, and the output features of all modules are extracted, the expression is: a l Represents the image block feature sequence output by the lth module of the picture encoder, including: image semantic information representation And the remaining N image blocks are represented by the image blocks output by the lth module 6. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 4, characterized in that: The process of setting window sizes of different scales, aggregating neighbor information of each image block feature sequence at different scales, and obtaining image features of all image blocks and all scales is as follows: Get the width and height of the feature map The one-dimensional image block feature sequence is converted into a two-dimensional image feature map through the reshape() function Assume the number of scales is r, the two-dimensional image feature map The image block feature at any position (h,w) is Its neighbor information is: Average pooling is used to aggregate the neighbor information of each image block feature sequence to obtain Will Use the reshape() function to reorganize into a one-dimensional sequence Image semantic information representation Splicing to obtain aggregated image features The image features of different levels and scales are expressed as:

7. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 6, characterized in that: The process of using the adapter to adjust the image features of all image patches and all scales is: Using three linear projections, the self-attention layer can be learned to transform the image features c lr Mapped to query vector Q lr , key vector K lr and value vector V lr , the expression is: Q lr =c lr ·W Q ,K lr =c lr ·W K ,V lr =c lr ·W V Use the self-attention mechanism and output weights through the output layer Mapped to the original dimension, the adjusted image features are obtained, expressed as: Image features at different levels and scales share learnable adapter weight parameters W Q , W K , W V and W O ; The adjusted image features are passed through the projection layer f of the image encoder p (·), get image features: Image features at different levels and scales are represented as:

8. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 7, characterized in that: The process of calculating the similarity between image features and text features is: Calculate the similarity between a single image feature and a text feature, including calculating the similarity between a single image feature and a normal state text representation, and calculating the similarity between a single image feature and an abnormal state text representation. The expression is: Calculate the similarity between a single image feature and the normal state text representation, expression: Among them, <·> represents the cosine similarity, and the similarity between a single image feature and the normal state text representation and the abnormal state text representation is generally expressed as Sum all l and r to get the overall score Get similarity scores at different levels and scales; normalize the similarity to obtain the probability score of classifying the image as abnormal: Among them, p CLS represents the probability score of image-level classification as abnormal, is the probability score of classifying N image patches as abnormalities.

9. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 8, characterized in that: When using the general anomaly detection dataset to train the learnable adapter and learnable text prompt, the label of the image anomaly in the general anomaly detection dataset is y, y∈{0,1}, and the focal loss is used to construct the loss of image-level anomaly classification, which is expressed as: L global =WordLoss(p CLS ,y) Among them, p CLS Represents the probability score of image-level classification as abnormal; Focalloss and L1loss are used to construct the loss of fine-grained image anomaly localization, and the expression is: in, Represents the reorganization value, which is obtained by the probability score P of N image blocks in the general anomaly detection dataset being classified as abnormal through the reshape() function operation, p CLS and P are related to the learnable weight parameters of the learnable adapter including W Q , W K , W V and W O , learnable text prompt parameters related; Represents the abnormal pixel mask label corresponding to the abnormal image in the general anomaly detection dataset Downsampling of The overall training loss L for training the learnable adapter and learnable textual prompts using the general anomaly detection dataset is: L=L global +L local When training the learnable adapter and the learnable text prompt, the overall training loss L is used as the training loss function to train the learnable adapter and the learnable text prompt. When the loss function converges, the training is completed, and the learnable weight parameters and the learnable text prompt parameters of the learnable adapter are fixed to obtain the trained learnable adapter and the learnable text prompt.

10. The zero-shot image anomaly detection and positioning method based on a pre-trained visual-language model according to claim 9, characterized in that: The trained learnable adapter and learnable text prompt are used for abnormal detection and positioning of the image to be detected, and the probability score of the image to be detected being classified as abnormal is obtained. The probability score of the image being classified as abnormal includes the probability score p of the image level being classified as abnormal. CLS The probability score P of the image block being classified as abnormal, the higher the probability score, the greater the possibility of image abnormality, when the probability score is greater than the first set standard, the image is judged to be abnormal; the probability score P of the image block being classified as abnormal is reorganized by the reshape() function to obtain the reorganized value Then reorganize the value Linear interpolation to A pixel mask of a certain size is formed, and position codes in the pixel mask that are larger than a second set standard are abnormal image areas.

Images