Zero sample image anomaly detection method and device

Abstract

The invention provides a zero sample image anomaly detection method and device, and belongs to the field of computer vision and image anomaly detection. The method comprises the steps that an image of a to-be-detected target is input into a preset zero sample image anomaly detection network, the network comprises an image encoder and a text encoder, and through multi-mode prompt learning, the image encoder outputs local visual features of the image, and the text encoder outputs text features capable of being learned and prompted to be embedded; and calculating the cosine similarity of the local visual features and the text features and interpolating to obtain a similar graph, and measuring the similarity to obtain an abnormal graph so as to obtain an abnormal detection result of the input image. According to the method, the flexibility of feature representation and the generalization ability of prompt are enhanced by constructing multi-modal prompt learning, the complexity of manual design of prompt engineering can be reduced, and a better anomaly detection prompt can be constructed, so that the effect of a zero sample anomaly detection model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and image anomaly detection, and in particular relates to a zero-sample image anomaly detection method and device. Background Art

[0002] Anomaly detection is a key topic in computer vision and is widely used in industrial visual inspection. It detects anomalies in the latter image by comparing the difference between normal images and images to be detected in the feature space. This task is atypical patterns in visual data. Due to the influence of multiple factors such as materials, equipment, processes, and environment during the production process, some industrial products will have "anomalies" on the surface. Industrial product anomalies usually appear as slight deviations in texture, shape, and color and are mixed with normal samples, which increases the difficulty of identifying abnormal products. Most current anomaly detection models are not suitable for industrial production scenarios with multiple categories of anomalies. However, it is unrealistic to build a specific anomaly detection model for each category. Therefore, it is necessary to construct a unified zero-sample anomaly detection model.

[0003] Zero-shot learning based on prompt learning can construct a unified anomaly detection model, which is an improvement on the prompt engineering method. The prompt engineering method is combined with the training of the visual language model to enable the model to learn and extract knowledge from the prompt without additional fine-tuning, thereby transferring the knowledge to downstream tasks. Prompt learning is to design the prompt engineering part into learnable parameters and model the context words of the prompt with learnable vectors. However, this technology currently only uses prompts in a single branch of vision or language to adjust the representation features, which is suboptimal because it does not allow the flexibility of dynamically adjusting the two representation spaces in downstream tasks, limiting the generalization of prompt learning. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a zero-sample image anomaly detection method and device. The present invention can enhance the flexibility of feature representation and the generalization ability of prompts by constructing multimodal prompt learning, reduce the complexity of manually designed prompt engineering, and construct better anomaly detection prompts, thereby improving the effect of the zero-sample anomaly detection model, achieving better detection and positioning effects on the input image, and can be better applied to industrial anomaly detection practice.

[0005] The first embodiment of the present invention provides a zero-sample image anomaly detection method, comprising:

[0006] Obtain an image of a set size of the target to be detected as the image to be detected;

[0007] Inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal cue learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be learned to embed cue;

[0008] The cosine similarity between the local visual feature and the text feature is calculated and interpolated to obtain a similarity map, and an abnormality map is obtained by similarity measurement, thereby obtaining an abnormality detection result of the image to be detected.

[0009] In a specific embodiment of the present invention, the main structure of the zero-sample image anomaly detection network adopts the CLIP model, wherein the image encoder includes 24 layers of visual coding subnetworks, and the text encoder includes 12 layers of text coding subnetworks, each layer of the visual coding subnetwork adopts the VIT model, and each layer of the text coding subnetwork adopts the Transformer model; the visual coding subnetwork adopts dual-path calculation, wherein in the VV attention path, the multi-head attention in the visual coding subnetwork from the 5th to the 20th layer is replaced by VV attention, and the MLP calculation of the layer is removed; a contrast prompt context module is added at the input end of the text encoder to generate prompt input.

[0010] In a specific embodiment of the present invention, the contrast prompt context module generates a context vector and a deep text prompt vector by random initialization; the context prefix vector and the context suffix vector generated by the contrast prompt template slice are concatenated with the context vector to obtain a complete prompt vector; wherein the contrast prompt template consists of three parts: a learnable context prompt, a domain-aware word, and a state prompt:

[0011]

[0012] Among them, Prompt + and Prompt - They are normal prompts and abnormal prompts generated for images; and is a learnable prefix vector, N + and N - is the number of word embeddings learned in normal prompts and abnormal prompts; domain represents the domain prompt word; object and damage object represent the image status, where object is an object, which is used to represent the template input that is not related to the category; damage is damage, which is used to summarize the abnormal state of the image;

[0013] The domain prompt word is concatenated with the state prompt, and then semantically connected with the prefix prompt vector to obtain a complete contrast prompt prompt. After word segmentation, a pair of contrast embedding vectors are obtained, which are further sliced ​​to obtain the prefix vector and suffix vector of the contrast prompt context, and then concatenated with the context vector ctx_vectors to obtain the complete prompts prompt vector. After inputting the prompt vector into the text encoding subnetwork, the text feature f is finally obtained. p .

[0014] In a specific embodiment of the present invention, the first J-1 layers of the visual encoding subnetwork of the image encoder also add deep visual cues; the deep visual cues come from the mapping of deep text cues in the contrast cue context module, and the deep visual cues are spliced ​​with the embedded code of the image input in the image encoder.

[0015] In a specific embodiment of the present invention, the outputs of the 6th, 12th, 18th and 24th visual coding subnetworks of the image encoder are combined to form a local visual feature patch_token corresponding to the input image;

[0016] After the input image passes through each layer of the visual encoding subnetwork of the image encoder in sequence, the image encoder also outputs the global visual feature g corresponding to the input image. p .

[0017] In a specific embodiment of the present invention, before inputting the image to be detected into a preset zero-sample image anomaly detection network, the method further includes:

[0018] Training the zero-shot image anomaly detection network;

[0019] The training of the zero-sample image anomaly detection network comprises:

[0020] 1) Constructing a training set; the training set consists of a normal image of the target to be detected and a corresponding binary segmentation image, and the size of the training set image is consistent with the size of the image to be detected;

[0021] 2) constructing the zero-sample image anomaly detection network and an original CLIP model respectively;

[0022] Among them, in the original CLIP model, the text encoder selects the prompt template of the manual prompt project as input to obtain text features;

[0023] The original CLIP model uses multi-head attention, expressed as follows:

[0024] [Q L ,K L ,V L]=QKVProj(LN(Z L-1 ))

[0025]

[0026] Among them, Z L-1 is the output of the L-1th layer of the image encoder in the original CLIP model, LN is the layer normalization, Proj is the calculated projection output, is the attention output of the Lth layer, MLP is the fully connected layer, Z L is the output of the Lth layer;

[0027] The VV attention adopted by the zero-shot image anomaly detection network is expressed as follows:

[0028] [Q L ,K L ,V L ]=QKVProj(LN(Z L-1 ))

[0029]

[0030] in, is the attention output of the Lth layer of the image encoder in the zero-shot image anomaly detection network, and is also the final output of this layer;

[0031] 3) When training the zero-shot image anomaly detection network, the CLIP backbone of the zero-shot image anomaly detection network is frozen, the backbone uses pre-trained parameters, and then only the parameters of the contrast prompt context module are trained; when training the original CLIP model, the pre-trained parameters are also used;

[0032] After the images of the training set are input into the original CLIP model, the model outputs the visual features g and text features f under non-prompted learning; after the images of the training set are input into the zero-sample image anomaly detection network, the network outputs the visual features g under prompted learning p and text features f p ;

[0033] The zero-sample image anomaly detection network and the original CLIP model are trained separately under the preset loss function constraint, and after the training is completed, the final zero-sample image anomaly detection network is obtained.

[0034] In a specific embodiment of the present invention, the training of the zero-sample image anomaly detection network and the original CLIP model respectively under a preset loss function constraint includes:

[0035] Use loss functions Focalloss, Diceloss and visual text feature constraints L alignThe parameters in the prompt context module are optimized, where the calculation expression of Focalloss is:

[0036] Focal(y true ,y pred )=-α*(1-p true ) γ *y true *log(P pred )

[0037] Among them, p true is the true probability distribution, P pred is the predicted probability distribution of the model; α and γ are hyperparameters used to control the weights of samples of different categories;

[0038] The calculation expression of DiceLoss is:

[0039]

[0040] Among them, y true is the true label, y pred It is the prediction result of the model; the value of Dice Loss is between 0 and 1. The closer the value is to 1, the more similar the model prediction is to the true label;

[0041] The constraints on the features under prompt learning and the visual text features under non-prompt learning are as follows:

[0042]

[0043] A second aspect of the present invention provides a zero-sample image anomaly detection device, comprising:

[0044] An image acquisition module is used to acquire an image of a set size of the target to be detected as the image to be detected;

[0045] A feature extraction module, used for inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal prompt learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be embedded in the learnable prompt;

[0046] The anomaly detection module is used to obtain an anomaly map by calculating the cosine similarity between the local visual feature and the text feature and interpolating the similarity map, and then obtain an anomaly detection result of the image to be detected by using the similarity measurement.

[0047] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0048] at least one processor; and a memory communicatively coupled to the at least one processor;

[0049] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned zero-sample image anomaly detection method.

[0050] To implement the above embodiment, a fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above zero-sample image anomaly detection method.

[0051] Features and beneficial effects of the present invention:

[0052] The present invention introduces a more general prompt learning template, which can automatically construct the prompt project more accurately and comprehensively, and update the prompt parameters as the network gradient is continuously optimized, thereby improving the efficiency of prompt construction and greatly reducing the cost of the prompt project. The present invention only requires some normal sample data for training and learning to achieve good zero-sample anomaly detection results. The present invention significantly improves the zero-sample migration performance of the prompt learning method relative to the non-prompt learning method, and greatly reduces the cost of migrating to downstream tasks based on pre-trained visual language models. The present invention is conducive to empowering industrial quality inspection in practical production and ensuring the consistency of detection of different detection categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 The figure is an overall flow chart of a zero-sample image anomaly detection method according to an embodiment of the present invention.

[0054] Figure 2 This is a structural diagram of a zero-sample image anomaly detection network according to a specific embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram of the principle of multimodal cue learning according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention proposes a zero-sample image anomaly detection method and device, which are further described in detail below in conjunction with the accompanying drawings and specific embodiments:

[0057] The first embodiment of the present invention provides a zero-sample image anomaly detection method, comprising:

[0058] Obtain an image of a set size of the target to be detected as the image to be detected;

[0059] Inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal cue learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be learned to embed cue;

[0060] The cosine similarity between the local visual feature and the text feature is calculated and interpolated to obtain a similarity map, and an abnormality map is obtained by similarity measurement, thereby obtaining an abnormality detection result of the image to be detected.

[0061] In a specific embodiment of the present invention, the overall process of the zero-sample image anomaly detection method is as follows: Figure 1 As shown, it is divided into a training phase and a testing phase, including the following steps:

[0062] 1) Training phase

[0063] 1-1) Constructing a training set

[0064] In this embodiment, the training set consists of normal images of the target to be detected without abnormalities. The training set images are uniformly adjusted to a preset size, the total number of training set images should be no less than 3000, and the original high resolution of each image is recommended to be 700*700, 1024*1024 or 1200*1200.

[0065] The training images used in a specific embodiment of the present invention are from the MvTec and VisA datasets, including 3629 and 6821 training images respectively. The training images in any of the datasets can be used as the training set during training in this embodiment. If the MvTec dataset is used, 1726 images are randomly selected from the training images in the dataset; if the VisA dataset is used, 2176 images are randomly selected from the training images in the dataset.

[0066] In this embodiment, all training images are adjusted to a size of 518*518, and the training set is constructed.

[0067] 1-2) Build a zero-shot image anomaly detection network.

[0068] In this embodiment, the zero-sample image anomaly detection network structure is as follows: Figure 2 As shown in the figure, its main structure adopts the CLIP model, specifically the pre-trained VIT-L / 14@336 model. The network contains two branches, namely the image encoder composed of a 24-layer visual encoding subnetwork and the text encoder composed of a 12-layer text encoding subnetwork. Each layer of the text encoding subnetwork adopts the Transformer model, and each layer of the visual encoding subnetwork adopts the VIT model.

[0069] Furthermore, in the zero-sample image anomaly detection network described in this embodiment, the visual encoding subnetwork adopts dual-path calculation. In the VV attention path, by bypassing the QK retrieval in the multi-head attention, the final attention output is calculated with VV attention, and the MLP calculation of this layer is removed to obtain richer local features. In this embodiment, VV attention is added to the last 20 layers of the visual encoder of this path, that is, the 5th to 24th layers, and the outputs of the 4 layers of visual encoding subnetworks numbered 6, 12, 18, and 24 in the image encoder are extracted to obtain the complete local visual feature patch_token of the input image.

[0070] Furthermore, in the zero-sample image anomaly detection network described in this embodiment, a contrast prompt context module is added to the input end of the text encoder to generate prompt input. The contrast prompt context module generates a context vector and a deep text prompt vector through random initialization. The context prefix vector, context suffix vector and context vector generated by slicing the contrast prompt template are concatenated with the context vector to obtain a complete prompts prompt vector. The contrast prompt template is composed of three parts: learnable context prompts, domain-aware words, and state prompts:

[0071]

[0072] Among them, Prompt + and Prompt - They are normal prompts and abnormal prompts generated for all images. Specifically, and is a learnable prefix vector, N + and N - is the number of word embeddings learned in normal prompts and abnormal prompts, which is set to 12 in a specific implementation of the present invention; "domain" represents domain prompt words, which include words such as "industrial, textural, surface, manufacturing", etc., which means "industrial, texture, surface, manufacturing" here; the last part adopts an object-independent prompt design, and the class name cls of the object is uniformly replaced with object in the state template, thereby shielding the class semantics of the object; damage means "damage" and is used to summarize the abnormal state of the image.

[0073] In this embodiment, the domain prompt word is concatenated with the state prompt, and then semantically connected with the prefix prompt vector to obtain a complete contrast prompt prompt. After word segmentation, a pair of contrast embedding vectors are obtained by projection, and further sliced ​​to obtain the prefix vector and suffix vector of the contrast prompt context, which are concatenated with the context vector ctx_vectors to obtain a complete prompts prompt vector. After the prompt vector is input into the text encoding subnetwork, the prompt learning text feature f is finally obtained. p .

[0074] Furthermore, this embodiment adds deep visual cues to the first J-1 layers of the visual encoding subnetwork of the image encoder. In a specific embodiment of the present invention, J is set to 9. The deep visual cues come from the mapping of deep text cues in the contrast cue context module. The deep visual cues are spliced ​​with the embedded coding of the image input in the image encoder and used as auxiliary cues in the training of the zero-sample image anomaly detection network to encourage the gradient flow of the text encoder and the image encoder and dynamically adjust the flexibility of the two representation spaces. After the input image passes through the image encoder, the output is the corresponding global visual feature g p And local visual features patch_token.

[0075] 1-3) Using the training set obtained in step 1-1), train the zero-sample image anomaly detection network established in step 1-2).

[0076] In this embodiment, during training, the zero-sample image anomaly detection network constructed in step 1-2) and an original CLIP model are initialized respectively. After the image of the training set is input into the original CLIP model, the model outputs the visual features g and text features f of the corresponding image under non-prompted learning; after the image of the training set is input into the zero-sample image anomaly detection network, the network outputs the visual features g and text features f of the corresponding image under prompted learning. p and text features f p In this embodiment, during training, the input batch of training set images is set to 8 images.

[0077] Specifically, in the original CLIP model, the text encoding subnetwork selects the hint template of the artificial hint project as input to obtain the text feature f; the input image enters the image encoding subnetwork to obtain the feature g. The artificial hint template selected in this embodiment is composed of a reference template and a state template. The reference template is such as "a bad photo of a{}, a lowresolution photo of the{}", which means "a bad photo", "a low-resolution photo"; the state template is such as "flawless{}','perfect{}','unblemished{}", etc., which means "flawed", "imperfect", "without flaws". In the visual encoding subnetwork, each batch of training set images is input into the image encoder to obtain the visual feature g.

[0078] It should be noted that in the original CLIP model, when C multi-head attention calculates the attention output, Q and K play the role of associating context, and the QK retrieval mode acts as a global average pool of visual global description to some extent. Among them, the original attention output is as follows:

[0079] [Q L ,K L ,V L ]=QKVProj(LN(Z L-1 ))

[0080]

[0081] Among them, Z L-1 is the output of the L-1th layer of the image encoder in the original CLIP model, LN is the layer normalization, Proj is the calculated projection output, is the attention output of the Lth layer, MLP is the fully connected layer, Z L is the output of the Lth layer.

[0082] The VV attention output calculation used by the zero-shot image anomaly detection network described in this embodiment is as follows:

[0083] [Q L ,K L ,V L ]=QKVProj(LN(Z L-1 ))

[0084]

[0085] in, It is the attention output of the Lth layer of the image encoder in the zero-shot image anomaly detection network, and it is also the final output of this layer.

[0086] In this embodiment, when training the zero-shot image anomaly detection network, the CLIP backbone of the zero-shot image anomaly detection network is frozen, the backbone uses pre-trained parameters, and then only the parameters of the contrast prompt context module are trained. For the original CLIP model, the pre-trained parameters are also used.

[0087] The training set images are respectively input into the zero-sample image anomaly detection network and the original CLIP model to obtain the visual features g under prompt learning. p and text features f p After learning the visual features g and text features f under non-prompt learning, the Adam optimizer is used. The specific training parameters are as follows: the fixed learning rate is 0.001, the bias betas are set to (0.5, 0.999), the training batch size is 8, and the number of iterations is 15 cycles. The training is performed under the constraint of the loss function. The specific method is as follows:

[0088] Use loss functions Focalloss, Diceloss and visual text feature constraints L align To optimize the parameters in the prompt context module. Focalloss is a variant of cross entropy loss, which aims to solve the problem of data imbalance. It gives lower weights to samples that are easy to classify and higher weights to samples that are difficult to classify, thereby encouraging the model to pay more attention to samples that are difficult to classify. The calculation expression is:

[0089] Focal(y true ,y pred )=-α*(1-p true ) γ *y true *log(P pred )

[0090] Among them, p true is the true probability distribution, P pred is the predicted probability distribution of the model. In training, p true That is the image segmentation map label, P pred That is the image anomaly map; α and γ are hyperparameters used to control the weights of samples of different categories.

[0091] DiceLoss is an indicator that measures the similarity between model predictions and true labels. It is usually used for binary classification problems, but can also be used in multi-classification problems. The expression is:

[0092]

[0093] Among them, y true is the true label, y pred is the prediction result of the model. In training, ytrue That is the image segmentation map label, y pred This is the image anomaly map. The value of Dice Loss is between 0 and 1. The closer the value is to 1, the more similar the model prediction is to the true label.

[0094] In addition, in order to guide the prompt features to learn correctly, the features under prompt learning and the features under non-prompt learning are constrained:

[0095]

[0096] In this embodiment, the training is completed after 15 training cycles, and the final zero-sample image anomaly detection network is obtained.

[0097] It should be noted that this embodiment enhances the flexibility of feature representation and the generalization ability of prompts through multi-modal prompt learning. The principle is as follows: Figure 3 As shown in the figure: in the text encoder, learnable cues, domain cues, and state cues are combined to form a unified general domain-aware contrast cues template, and learnable tags are introduced up to a specific depth; in the image encoder, deep visual cues are added up to a specific depth. The deep cues provide the flexibility to learn cues across different hierarchical structures in the encoder structure.

[0098] 2)Testing phase.

[0099] 2-1) Obtain the image to be detected.

[0100] Get a new image of the target to be detected, adjust the image size to the same size as the training set image, and use it as the image to be detected.

[0101] The test images used in this embodiment are from the MvTec and VisA datasets. The test images may include normal images and abnormal images, and the size of the test images is adjusted to 518*518 during image preprocessing.

[0102] In this embodiment, in order to avoid "data leakage", when the model training weights used are obtained by training on the MvTec dataset, the test images of the VisA dataset are used here; when the model training weights used are obtained by training on the VisA dataset, the test images of the MvTec dataset are used here.

[0103] 2-2) Using the zero-sample image anomaly detection network trained in step 1), perform anomaly detection on the image to be detected obtained in step 2-1).

[0104] In this embodiment, the image to be detected obtained in step 2-1) is input into the zero-sample image anomaly detection network trained in step 1). The global visual feature g is obtained in the image encoder. p, local visual features patch_tokens, and get text features f in the text encoder p .

[0105] By calculating the local visual feature patch_tokens and the text feature f p The cosine similarity of the two graphs is interpolated to obtain the similar graph, and the abnormal graph is obtained by similarity measurement; the global visual feature g is calculated. p and text features f p The classification scores are then used to obtain anomaly detection and anomaly location results.

[0106] The embodiments of the present invention were tested on MvTec and VisA, and achieved results of (91.6, 96.3) (81.7, 84.9) in terms of the area under the ROC curve AUROC and accuracy AP indicators at the image level, and achieved results of (88.6, 80.3) (94.5, 86.3) in terms of the area under the ROC curve AUROC and PRO indicators at the pixel level, which are better than the non-prompted learning method.

[0107] To implement the above embodiment, a second aspect of the present invention provides a zero-sample image anomaly detection device, comprising:

[0108] An image acquisition module is used to acquire an image of a set size of the target to be detected as the image to be detected;

[0109] A feature extraction module, used for inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal prompt learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be embedded in the learnable prompt;

[0110] The anomaly detection module is used to obtain an anomaly map by calculating the cosine similarity between the local visual feature and the text feature and interpolating the similarity map, and then obtain an anomaly detection result of the image to be detected by using the similarity measurement.

[0111] It should be noted that the above explanation of the embodiment of a zero-sample image anomaly detection method is also applicable to a zero-sample image anomaly detection device of this embodiment, which will not be repeated here. According to an embodiment of the present invention, a zero-sample image anomaly detection device is proposed, by obtaining an image of a set size of the target to be detected as the image to be detected; the image to be detected is input into a preset zero-sample image anomaly detection network, the zero-sample image anomaly detection network includes an image encoder and a text encoder, through multi-modal prompt learning, the image encoder outputs the multi-level local visual features of the image to be detected, and the text encoder outputs the text features that can be learned to embed prompts; by calculating the cosine similarity of the local visual features and the text features and interpolating to obtain a similarity map, the anomaly map is obtained by similarity measurement, and then the anomaly detection result of the image to be detected is obtained. Therefore, this embodiment enhances the flexibility of feature representation and the generalization ability of prompts by constructing multi-modal prompt learning, which can reduce the complexity of manually designed prompt engineering, and construct better anomaly detection prompts, thereby improving the effect of the zero-sample anomaly detection model, achieving better detection and positioning effects on the input image, and can be better applied to industrial anomaly detection practice.

[0112] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0113] at least one processor; and a memory communicatively coupled to the at least one processor;

[0114] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned zero-sample image anomaly detection method.

[0115] To implement the above embodiment, a fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above zero-sample image anomaly detection method.

[0116] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0117] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes a zero-sample image anomaly detection method of the above embodiment.

[0118] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0120] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0121] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0123] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0125] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0126] The storage medium mentioned above may be a read-only memory, a disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A zero-sample image anomaly detection method, characterized in that: include: Obtain an image of a set size of the target to be detected as the image to be detected; Inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal cue learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be learned to embed cue; The cosine similarity between the local visual feature and the text feature is calculated and interpolated to obtain a similarity map, and an abnormality map is obtained by similarity measurement, thereby obtaining an abnormality detection result of the image to be detected.

2. The method according to claim 1, characterized in that The main structure of the zero-sample image anomaly detection network adopts the CLIP model, wherein the image encoder includes 24 layers of visual coding subnetworks, and the text encoder includes 12 layers of text coding subnetworks, each layer of the visual coding subnetwork adopts the VIT model, and each layer of the text coding subnetwork adopts the Transformer model; the visual coding subnetwork adopts dual-path calculation, wherein in the VV attention path, the multi-head attention in the visual coding subnetwork from the 5th to the 20th layer is replaced by VV attention, and the MLP calculation of this layer is removed; a contrast prompt context module is added at the input end of the text encoder to generate prompt input.

3. The method according to claim 2, characterized in that The contrast prompt context module generates a context vector and a deep text prompt vector by random initialization; the context prefix vector and the context suffix vector generated by the contrast prompt template slice are concatenated with the context vector to obtain a complete prompt vector; wherein the contrast prompt template consists of three parts: a learnable context prompt, a domain-aware word, and a state prompt: Among them, Prompt + and Prompt - They are normal prompts and abnormal prompts generated for images; and is a learnable prefix vector, N + and N - is the number of word embeddings learned in normal prompts and abnormal prompts; domain represents the domain prompt word; object and damage object represent the image status, where object is an object, which is used to represent the template input that is not related to the category; damage is damage, which is used to summarize the abnormal state of the image; The domain prompt word is concatenated with the state prompt, and then semantically connected with the prefix prompt vector to obtain a complete contrast prompt prompt. After word segmentation, a pair of contrast embedding vectors are obtained, which are further sliced ​​to obtain the prefix vector and suffix vector of the contrast prompt context, and then concatenated with the context vector ctx_vectors to obtain the complete prompts prompt vector. After inputting the prompt vector into the text encoding subnetwork, the text feature f is finally obtained. p .

4. The method according to claim 3, characterized in that The first J-1 layers of the visual encoding subnetwork of the image encoder also add deep visual cues; the deep visual cues come from the mapping of deep textual cues in the contrast cue context module, and the deep visual cues are spliced ​​with the embedded code of the image input in the image encoder.

5. The method according to claim 4, characterized in that The outputs of the 6th, 12th, 18th and 24th visual encoding subnetworks of the image encoder are combined to form a local visual feature patch_token corresponding to the input image; After the input image passes through each layer of the visual encoding subnetwork of the image encoder in sequence, the image encoder also outputs the global visual feature g corresponding to the input image. p .

6. The method according to claim 5, characterized in that Before inputting the image to be detected into a preset zero-sample image anomaly detection network, the method further includes: Training the zero-shot image anomaly detection network; The training of the zero-sample image anomaly detection network comprises: 1) Constructing a training set; the training set consists of a normal image of the target to be detected and a corresponding binary segmentation image, and the size of the training set image is consistent with the size of the image to be detected; 2) constructing the zero-sample image anomaly detection network and an original CLIP model respectively; Among them, in the original CLIP model, the text encoder selects the prompt template of the manual prompt project as input to obtain text features; The original CLIP model uses multi-head attention, expressed as follows: [Q L ,K L ,V L ]=QKVProj(LN(Z L-1 )) Among them, Z L-1 is the output of the L-1th layer of the image encoder in the original CLIP model, LN is the layer normalization, Proj is the calculated projection output, is the attention output of the Lth layer, MLP is the fully connected layer, Z L is the output of the Lth layer; The VV attention adopted by the zero-shot image anomaly detection network is expressed as follows: [Q L ,K L ,V L ]=QKVProj(LN(Z L-1 )) in, is the attention output of the Lth layer of the image encoder in the zero-shot image anomaly detection network, and is also the final output of this layer; 3) When training the zero-shot image anomaly detection network, the CLIP backbone of the zero-shot image anomaly detection network is frozen, the backbone uses pre-trained parameters, and then only the parameters of the contrast prompt context module are trained; when training the original CLIP model, the pre-trained parameters are also used; After the images of the training set are input into the original CLIP model, the model outputs the visual features g and text features f under non-prompted learning; after the images of the training set are input into the zero-sample image anomaly detection network, the network outputs the visual features g under prompted learning p and text features f p ; The zero-sample image anomaly detection network and the original CLIP model are trained separately under the preset loss function constraint, and after the training is completed, the final zero-sample image anomaly detection network is obtained.

7. The method according to claim 6, characterized in that The zero-sample image anomaly detection network and the original CLIP model are trained separately under the preset loss function constraint, including: Use loss functions Focal loss, Dice loss and visual text feature constraints L align The parameters in the prompt context module are optimized, where the calculation expression of Focalloss is: Focal(y true ,y pred )=-α*(1-p true ) γ *y true *log(P pred ) Among them, p true is the true probability distribution, P pred is the predicted probability distribution of the model; α and γ are hyperparameters used to control the weights of samples of different categories; The calculation expression of DiceLoss is: Among them, y true is the true label, y pred It is the prediction result of the model; the value of Dice Loss is between 0 and 1. The closer the value is to 1, the more similar the model prediction is to the true label; The constraints on the features under prompt learning and the visual text features under non-prompt learning are as follows:

8. A zero-sample image anomaly detection device, characterized in that: include: An image acquisition module is used to acquire an image of a set size of the target to be detected as the image to be detected; A feature extraction module, used for inputting the image to be detected into a preset zero-shot image anomaly detection network, wherein the zero-shot image anomaly detection network includes an image encoder and a text encoder. Through multimodal prompt learning, the image encoder outputs multi-level local visual features of the image to be detected, and the text encoder outputs text features that can be embedded in the learnable prompt; The anomaly detection module is used to obtain an anomaly map by calculating the cosine similarity between the local visual feature and the text feature and interpolating the similarity map, and then obtain an anomaly detection result of the image to be detected by using the similarity measurement.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images