Zero-shot anomaly detection method, device and equipment based on visual and language models

Abstract

The application provides a zero-shot anomaly detection method and device based on a visual and language model, and relates to the technical field of anomaly detection. The method comprises the following steps: obtaining an inference image and a reference image from the same batch of images; performing double-modal detection on the inference image and the reference image respectively through a first multi-modal model to obtain an initial mask; extracting features of the same batch of images respectively through a second multi-modal model and performing feature aggregation respectively to obtain aggregated inference features and aggregated reference features; filtering noise features from the aggregated reference features according to the initial mask to obtain non-anomalous aggregated reference features, and obtaining an initial anomaly score according to the non-anomalous aggregated reference features and the aggregated inference features; and performing mask refining according to the initial anomaly score and the initial mask to obtain a final inference mask, which is used for realizing zero-shot anomaly detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection, in particular to a zero-shot anomaly detection method and device based on a visual and language model and equipment. BACKGROUND

[0002] Anomaly detection is a key topic in computer vision and is widely used in industrial visual inspection. The model compares the differences between normal images and images to be detected in the feature space for detecting anomalies in the latter. In related technologies, anomaly detection methods mainly use an unsupervised paradigm and still require a large number of normal samples for training. These methods consume a large amount of computing resources and have weak generalization ability.

[0003] With the emergence of zero-shot and few-shot visual language models, such models are applied to zero-shot anomaly detection, and rare classes become important in many application scenarios. The original q-k attention mechanism affects the local semantics, limiting the ability of fine-grained anomaly segmentation. To strengthen the local semantic representation, a self-attention mechanism is introduced. Although the improvement of the self-attention mechanism brings performance improvement, the local semantic representation is still insufficient, and feature matching using the entire test dataset is still needed for anomaly detection. In the actual production environment, testing is generally performed on a batch of samples, and prior knowledge of the test dataset distribution is required to be optimized in advance, but using the entire test dataset in real industrial scenarios is not practical. The method for realizing zero-shot anomaly detection is not perfect in related technologies. SUMMARY

[0004] The problem solved by the present application is how to realize zero-shot anomaly detection.

[0005] To solve the above problems, the present application provides a zero-shot anomaly detection method and device based on a visual and language model and equipment.

[0006] In a first aspect, the present application provides a zero-shot anomaly detection method based on a visual and language model, comprising: obtaining an inference image and a reference image from the same batch of images, wherein the inference image represents an image to be anomaly detected, and the reference image includes other images in the same batch of images except the inference image;

[0007] performing double-modal detection on the inference image and the reference image respectively through a first multi-modal model to obtain an initial mask, wherein the initial mask includes an initial inference anomaly mask and an initial reference anomaly mask;

[0008] extracting features of the same batch of images through a second multi-modal model and performing feature aggregation respectively to obtain aggregated inference features and aggregated reference features;

[0009] According to the initial mask, noise feature filtering is performed on the aggregated reference feature to obtain a non-abnormal aggregated reference feature, and an initial anomaly score is obtained according to the non-abnormal aggregated reference feature and the aggregated inference feature, wherein the initial anomaly score includes an inference anomaly score and a reference anomaly score;

[0010] According to the initial anomaly score and the initial mask, mask refining is performed to obtain a final inference mask, wherein the final inference mask is used to represent an anomaly detection inference result.

[0011] Optionally, before the step of performing bimodal detection on the inference image and the reference image respectively by using the first multi-modal model to obtain an initial mask, the method further includes:

[0012] An initial multi-modal model is determined.

[0013] Residual connections and feedforward modules in the initial multi-modal model are removed.

[0014] Attention features of an intermediate layer in the initial multi-modal model are weighted to replace attention features of a last layer to obtain the first multi-modal model, wherein the intermediate layer includes a specific layer in the initial multi-modal model.

[0015] Optionally, before the step of performing bimodal detection on the inference image and the reference image respectively by using the first multi-modal model to obtain an initial mask, the method further includes:

[0016] A normal text prompt template and an abnormal text prompt template are constructed.

[0017] Normal text features and abnormal text features in the normal text prompt template and the abnormal text prompt template are extracted respectively by using the first multi-modal model.

[0018] Optionally, the text features include the normal text features and the abnormal text features, and the step of performing bimodal detection on the inference image and the reference image respectively by using the first multi-modal model to obtain an initial mask includes:

[0019] Inference image features and reference image features of the inference image and the reference image are extracted respectively by using the first multi-modal model.

[0020] Inference similarity and reference similarity between the inference image features and the text features and between the reference image features and the text features are calculated respectively.

[0021] The initial mask is determined according to the inference similarity and the reference similarity.

[0022] Optionally, the extracting features of the same batch of images respectively by the second multi-modal model and performing feature aggregation respectively to obtain aggregated inference features and aggregated reference features comprises:

[0023] For each image, extracting stage patch features by the second multi-modal model, wherein the stage patch features are obtained from each layer output of a visual encoder of the second multi-modal model;

[0024] performing feature aggregation on the stage patch features of the same image number and the same visual encoder layer within a preset neighborhood area to obtain aggregated features;

[0025] obtaining the aggregated inference features according to the aggregated features corresponding to the inference image, and obtaining the aggregated reference features according to the aggregated features corresponding to the reference image.

[0026] Optionally, the filtering noise features from the aggregated reference features according to the initial mask to obtain non-anomalous aggregated reference features, and obtaining initial anomaly scores according to the non-anomalous aggregated reference features and the aggregated inference features comprises:

[0027] filtering out the aggregated reference features with abnormal features according to the initial mask, and retaining non-anomalous aggregated reference features;

[0028] calculating feature distances between the aggregated inference features and the non-anomalous aggregated reference features, and taking the smallest feature distance as the initial anomaly score.

[0029] Optionally, the refining the mask according to the initial anomaly score and the initial mask to obtain a final inference mask comprises:

[0030] generating average anomaly scores according to the extracted stage patch features in the inference image and the reference image, wherein the average anomaly scores include inference average anomaly scores and reference average anomaly scores;

[0031] averaging the average anomaly scores with initial anomaly scores respectively to obtain refined inference anomaly scores and refined reference anomaly scores under each neighborhood area;

[0032] performing binary processing on the refined inference anomaly scores and the refined reference anomaly scores that exceed a refined threshold to obtain an inference intermediate mask and a reference intermediate mask;

[0033] performing collaborative voting on the inference intermediate mask and the reference intermediate mask respectively to obtain a final inference mask and a final reference mask.

[0034] Optionally, the collaborative voting of the inference intermediate mask and the reference intermediate mask respectively to obtain the final inference mask and the final reference mask comprises:

[0035] obtaining the inference intermediate mask and the reference intermediate mask under each preset neighborhood area;

[0036] collaborative voting of the inference intermediate mask under each preset neighborhood area to obtain the final inference mask;

[0037] collaborative voting of the reference intermediate mask under each preset neighborhood area to obtain the final reference mask.

[0038] In a second aspect, the present application further provides a zero-shot anomaly detection device based on a visual and language model, comprising:

[0039] an acquisition module configured to acquire an inference image and a reference image from the same batch of images, wherein the inference image represents an image to be detected for anomaly, and the reference image comprises other images in the same batch of images except the inference image;

[0040] a first detection module configured to perform double-modal detection on the inference image and the reference image respectively by a first multi-modal model to obtain an initial mask, wherein the initial mask comprises an initial inference anomaly mask and an initial reference anomaly mask;

[0041] a second detection module configured to extract features of the same batch of images and perform feature aggregation respectively by a second multi-modal model to obtain aggregated inference features and aggregated reference features;

[0042] a noise feature filtering module configured to perform noise feature filtering on the aggregated reference features according to the initial mask to obtain non-anomalous aggregated reference features, and obtain an initial anomaly score according to the non-anomalous aggregated reference features and the aggregated inference features, wherein the initial anomaly score comprises an inference anomaly score and a reference anomaly score;

[0043] a refining module configured to perform mask refining according to the initial anomaly score and the initial mask to obtain a final inference mask, wherein the final inference mask is used to represent an anomaly detection inference result.

[0044] In a third aspect, the present application further provides an electronic device comprising a memory and a processor;

[0045] the memory is configured to store a computer program;

[0046] the processor is configured to implement the zero-shot anomaly detection method based on a visual and language model as described in the first aspect when executing the computer program.

[0047] The beneficial effects of the zero-shot anomaly detection method based on the visual and language model of the present application are:

[0048] The inference image and the reference image are obtained from the same batch of images, the similarity between the images is utilized, and the anomaly detection can be directly performed without additional training data, which is beneficial for realizing zero-shot anomaly detection in the absence of specific domain samples. The first multi-modal model is used for double-modal detection of the inference image and the reference image to generate an initial mask, and the multi-modal information such as images and texts is combined to enhance the local semantic representation, which is beneficial for fine-grained anomaly segmentation and provides a basis for subsequent refinement. The second multi-modal model is used for extracting and aggregating the features of the same batch of images to capture information of different scales and levels, improve the detection comprehensiveness and accuracy, and strengthen the understanding ability of the model to complex patterns. According to the initial mask, the noise in the aggregated reference features is filtered, the information of normal patterns is retained, the influence of noise is reduced, the robustness and reliability of the model are improved, the model can focus on the real abnormal situation, and the detection accuracy is improved. The inference image is compared with the purified reference features to obtain reliable anomaly scores to evaluate the possibility of each pixel or region to appear abnormal, which provides a basis for further optimization. The mask refinement is performed based on the anomaly scores and the initial mask, and the inference mask accurately reflecting the anomaly detection result is finally obtained, realizing efficient and accurate zero-shot anomaly detection. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flowchart of the zero-shot anomaly detection method based on the visual and language model of the embodiment of the present application is shown in the figure.

[0050] Figure 2 The flowchart of step S200 of the zero-shot anomaly detection method based on the visual and language model of the embodiment of the present application is shown in the figure.

[0051] Figure 3 The flowchart of steps S300-S500 of the zero-shot anomaly detection method based on the visual and language model of the embodiment of the present application is shown in the figure.

[0052] Figure 4 The example diagram of the electronic device of the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be realized in various forms, and should not be interpreted as being limited to the embodiments described herein, on the contrary, these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of protection of the present application.

[0054] It should be understood that each step described in the method embodiments of the present application can be performed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0055] The term "comprising" and variations thereof as used herein are open-ended, that is "including, but not limited to"; the term "based on" is, at least in part based on; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optional" means "optional in at least some embodiments". Related definitions are given throughout the description. It is to be noted that the concepts "first", "second", etc. mentioned in the present application are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0056] It should be noted that the modification of "one" or "multiple" mentioned in the present application is illustrative rather than limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as "one or more".

[0057] The names of the messages or information exchanged between the devices in the embodiments of the present application are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0058] Anomaly detection is widely used in different fields and plays an important role. It is used to identify abnormal situations in samples and determine their positions. Traditional anomaly detection methods mainly use unsupervised paradigm and rely on a large number of normal samples for training. However, these methods consume a lot of computing resources and have weak generalization ability.

[0059] Zero-shot anomaly detection has started to attract attention and can be directly inferred without target domain data. The development of visual-linguistic pre-training models and their strong zero-shot generalization ability have promoted the application of this technology in zero-shot anomaly detection. Multimodal models are used for anomaly detection, estimating anomaly probability through the similarity of image and text features. However, some studies have pointed out that the original q-k attention mechanism affects local semantics, limiting the ability of fine-grained anomaly segmentation. In order to strengthen the local semantic representation, the v-v attention mechanism is introduced.

[0060] Although the improvement of the self-attention mechanism improves the performance, it weakens the generalization ability of the model to some extent. Since most of the pixels in the test data set are normal, the entire test data set is used for feature matching for anomaly detection. In the actual production environment, the test is generally carried out on a batch of samples, and it is not realistic to use the entire test data set, and a large amount of computing resources are required. In addition, the prior knowledge of the distribution of the test data set before the test requires post-optimization. The mainstream anomaly detection method usually involves fine-tuning the model on one data set and then testing on another data set, which may have the risk of data leakage.

[0061] To solve the above problems, the present application proposes a zero-shot anomaly detection method without training, which combines the principles of zero-shot anomaly detection and the idea of feature matching. By regarding the images in the same batch as mutual references, limited reference information can be collected. This method uses the relationship between images in the same batch to avoid the traditional training process while ensuring the effectiveness of anomaly detection. In other words, with the help of the comparison between images in the same batch, the anomaly detection method of the present application can complete the anomaly detection task without additional training.

[0062] In view of the problems existing in the above related technology, the embodiment provides a zero-shot anomaly detection method and device based on a visual and language model and equipment.

[0063] As shown in Figure 1 , the zero-shot anomaly detection method based on a visual and language model provided by the embodiment of the present application comprises:

[0064] Step S100, acquiring an inference image and a reference image from the same batch of images, wherein the inference image represents an image to be detected, and the reference image includes other images in the same batch of images except the inference image.

[0065] From the images in the same batch, the inference image to be detected and the reference image are obtained, wherein, in order to ensure zero-shot detection, the inference image and the reference image in the embodiment of the present application are selected from the same batch of images, for example, when the same batch of images has five images, image 1 is used as the inference image, and at least one of image 2, image 3, image 4 and image 5 is used as the reference image; when image 2 is used as the inference image, at least one of image 1, image 3, image 4 and image 5 is used as the reference image.

[0066] As shown in Figure 2 , step S200, performing double-modal detection on the inference image I u and the reference image I v respectively through a first multi-modal model to obtain an initial mask, wherein the initial mask includes an initial inference anomaly mask M uand the initial reference abnormality mask M v .

[0067] A multi-modal model is a model capable of processing and integrating input information from two or more different data types (i.e., modalities). In embodiments of the present application, a multi-modal model is used to process input data of text type and image type.

[0068] The inference image and the reference image are detected by the first multi-modal model, and the corresponding initial inference abnormality mask and initial reference abnormality mask are obtained respectively. The initial inference abnormality mask and the initial reference abnormality mask are used to represent the probability that the inference image and the reference image are classified as normal or abnormal. The proxy layer is used to restore the semantic correlation between patches, and the attention weight thereof is derived from the intermediate layer. Wherein, M u and M v represent the inference mask and the reference mask respectively, and VisualEncoder represents the visual encoder.

[0069] Optionally, the first multi-modal model is an improved CLIP model, i.e., a SeCLIP model.

[0070] As shown in Figure 3 , at step S300, the features of the same batch of images are extracted by the second multi-modal model respectively, and the features are aggregated respectively to obtain aggregated inference features and aggregated reference features.

[0071] The second multi-modal model optimized or adjusted is used to process image data, and visual features are extracted from the same batch of images. These features not only include global information, but also contain local information at different levels, which helps to better understand the image content and better capture local details and semantic information in the image.

[0072] Since the outputs of different layers of the multi-modal model have different attention, in order to understand the local semantics and increase the generalization of the model, by aggregating the local information of different scales at different stages (outputs of different layers), a more rich feature representation is generated to enhance the robustness and richness of the feature representation, thereby improving the accuracy of anomaly detection. The reference image provides a benchmark for normal patterns, and by comparing the aggregated features of the inference image and the reference image, the abnormal area can be more effectively identified.

[0073] Optionally, the second multi-modal model is an improved CLIP model, i.e., a FiCLIP model.

[0074] Step S400, filtering noise features from the aggregated reference features according to the initial mask to obtain non-anomalous aggregated reference features, and obtaining an initial anomaly score according to the non-anomalous aggregated reference features and the aggregated inference features, wherein the initial anomaly score includes an inference anomaly score and a reference anomaly score.

[0075] When applying the feature matching method for anomaly detection, it is necessary to ensure that the reference image does not contain any patch similar to the anomaly patch in the inference image. However, when applied to zero-shot anomaly detection, the uncertainty of whether the reference features are normal may introduce some interference. The similarity between the anomaly patches in the reference image and the inference image leads to a decrease in the anomaly score of the inference image, which may cause an increase in the false negative rate in anomaly detection. Therefore, the abnormal features in the aggregated reference features are filtered out, and the filtered non-anomalous aggregated reference features are used as a reference to calculate the initial anomaly score.

[0076] By filtering out the abnormal features in the reference image through the initial mask, the reference features only contain information of normal patterns, thereby reducing the influence of noise on subsequent calculations. By comparing the inference image with clean non-anomalous reference features, the abnormal area can be more accurately identified, avoiding false positives and false negatives. After removing the noise features, the model can better capture the true abnormal information, improving the robustness and reliability of the overall detection. The initial anomaly score provides an abnormal probability distribution, which helps to more accurately locate and describe the abnormal area in the subsequent steps.

[0077] Step S500, refining the mask according to the initial anomaly score and the initial mask to obtain a final inference mask, wherein the final inference mask is used to represent the anomaly detection inference result.

[0078] The initial anomaly score is calculated by comparing the differences between the inference image and the non-anomalous reference features, indicating the likelihood of each pixel or patch position being abnormal. This score provides a preliminary estimate of potential abnormal areas in the image, helping to quickly locate areas that may have problems and laying the foundation for subsequent analysis. The generated initial mask serves as an important input for the subsequent refinement step, guiding how to optimize the anomaly detection result.

[0079] Combining different neighborhood sizes and different stage anomaly scores can capture multi-scale and hierarchical information, which helps to enhance the comprehensiveness and accuracy of detection. The binary mask generated based on the preliminary anomaly score indicates which areas may be abnormal, guiding the noise feature filtering and refinement process. By filtering out the abnormal features in the reference image, the reference features only contain information of normal patterns, reducing the influence of noise on subsequent calculations and improving the robustness of the model.

[0080] The initial mask provides a starting point for the refinement step, allowing the model to focus on areas most likely to be abnormal, thus optimizing more efficiently. Through preliminary screening, most obviously normal areas can be excluded, reducing the probability of false positives and making the final results more reliable. This method not only simplifies the processing flow, but also improves detection efficiency and the quality of the results.

[0081] In this embodiment, by obtaining inference images and reference images from the same batch of images, the similarity between images is utilized to perform anomaly detection directly without additional training data, facilitating zero-shot anomaly detection in the absence of domain-specific samples.

[0082] As shown in Figure 2 , the first multi-modal model is used to perform dual-modal detection on the inference images I u and the reference images I v , respectively, to obtain initial masks. The multi-modal model combines image and text features (i.e., Text in Figure 2 ) or other modal information to enhance local semantic representation, facilitating fine-grained anomaly segmentation. The generated initial masks provide preliminary anomaly location estimates, laying the foundation for subsequent refinement steps and reducing computational resource consumption.

[0083] The second multi-modal model is used to extract features from the same batch of images and perform feature aggregation, respectively. Feature aggregation can capture information at different scales and levels, improving the comprehensiveness and accuracy of detection. This process enhances the model's understanding of complex patterns in images, facilitating more accurate anomaly identification.

[0084] According to the initial mask, noise feature filtering is performed on the aggregated reference features to obtain non-anomalous aggregated reference features. After filtering out the abnormal features in the reference images, the aggregated reference features only contain normal pattern information, reducing the impact of noise on subsequent calculations and improving the robustness and reliability of the model. This step allows the model to focus on true anomalies, improving detection accuracy.

[0085] According to the non-anomalous aggregated reference features and the aggregated inference features, initial anomaly scores are obtained. By comparing the inference images with the purified reference features, more reliable anomaly scores, i.e., inference anomaly scores and reference anomaly scores, can be obtained. These scores are used to evaluate the likelihood of each pixel or region being abnormal, providing a basis for further optimization.

[0086] Mask refinement is performed according to the initial anomaly score and the initial mask to obtain a final inference mask. The refinement process further optimizes the anomaly detection result, eliminates most obviously normal regions, reduces the false positive probability, and makes the final result more reliable. The final inference mask accurately represents the inference result of anomaly detection, and realizes efficient and accurate zero-shot anomaly detection.

[0087] Optionally, before the initial mask is obtained by performing bimodal detection on the inference image and the reference image respectively through the first multi-modal model, the method further comprises:

[0088] An initial multi-modal model is determined.

[0089] The residual connection and the feedforward module in the initial multi-modal model are removed.

[0090] The attention features of an intermediate layer of the initial multi-modal model are weighted to replace the attention features of a last layer to obtain the first multi-modal model, wherein the intermediate layer comprises a specific layer in the initial multi-modal model.

[0091] In an embodiment, the initial multi-modal model is a CLIP model. Since CLIP is trained for a classification task, it pays more attention to global semantic information. However, anomaly classification and segmentation require understanding of local semantics. Therefore, it is necessary to restore its semantic relevance to local patterns. The CLIP visual encoder based on ViT is composed of a series of attention blocks. Each block generates a feature representation X i ∈R B ×(HW+1)×D , where i represents a layer index, B represents a batch size, D represents a dimension, and HW represents a number of local patch tokens.

[0092] In order to maintain the generalization ability of CLIP, the changes are limited to the last layer. By visualizing the attention map of CLIP, it can be found that the attention map of the intermediate layer has better local semantics. Therefore, the attention of the intermediate layer Attn inter is used to replace the attention map Attn -1 of the last layer:

[0093] Attn -1 =Attn inter , X attn =Proj(Attn inter·v-1 ),

[0094] where Proj represents a projection layer.

[0095] The residual connection has a significant negative impact on the performance of dense segmentation tasks, while the impact of the feedforward module is minimal. In the embodiment of the application, the output X final ∈R B×(HW+1)×DCan be defined as:

[0096] X final =X attn ,

[0097] Wherein, X attn Indicates the feature representation after the modified attention mechanism processing.

[0098] Optionally, before the step of performing double-modal detection on the inference image and the reference image respectively through the first multi-modal model to obtain an initial mask, the method further comprises:

[0099] Constructing a normal text prompt template and an abnormal text prompt template.

[0100] Extracting normal text features and abnormal text features in the normal text prompt template and the abnormal text prompt template through the first multi-modal model.

[0101] Optionally, the text features include the normal text features and the abnormal text features, and the step of performing double-modal detection on the inference image and the reference image respectively through the first multi-modal model to obtain an initial mask comprises:

[0102] Extracting inference image features and reference image features of the inference image and the reference image through the first multi-modal model.

[0103] Calculating inference similarity and reference similarity between the inference image features and the text features, and between the reference image features and the text features, respectively.

[0104] Determining the initial mask according to the inference similarity and the reference similarity.

[0105] In order to utilize the multi-modal capability of CLIP for zero-shot anomaly detection, the similarity between text features and visual features is calculated to determine whether it is abnormal. Anomaly detection involves the concepts of normal and abnormal, so the text prompt template can be designed as "a [extra information] [abnormal category] [image category] image", wherein the extra information represents additional information about the image, such as rotation or cropping; the abnormal category represents whether the image is abnormal, including abnormal or normal; and the image category represents the category of the image, such as a person image and an object image.

[0106] Using the text encoder of CLIP to extract the text features of the text prompt template, obtaining normal text features and abnormal text features respectively, and obtaining the final text features F t ∈R 2×C , wherein C represents the dimension of the features. For abnormal classification, the global visual features Fc ∈R B×C (ie, reasoning image features and reference image features, when the processed image is a reasoning image, F c is the probability that the image is classified as normal or abnormal, Cls prob ∈R B×2 which can be represented as:

[0107] CLS prob = Softmax(F c · F t ),

[0108] The abnormal classification score CLS score ∈R B represents the probability that the image category is abnormal. For abnormal segmentation, remove the [CLS] tag and map the dimension to C, finally obtain the local patch feature F s ∈R B×HW×C . The abnormal segmentation probability SEG prob ∈R B×H×W×2 can be represented as:

[0109] SEG prob = Softmax(F s · F t ),

[0110] The abnormal segmentation score Seg score ∈R B×HW represents the probability that the patch is abnormal.

[0111] For two images in the same batch, SeCLIP-AD, the first multi-modal model, calculates the similarity between the text and visual features to generate the corresponding abnormal scores CLS prob and the mask SEG prob , the mask SEG prob is the initial mask.

[0112] Optionally, as shown in Figure 3 , the extracting features of the same batch of images by the second multi-modal model and aggregating features respectively to obtain aggregated reasoning features and aggregated reference features comprises:

[0113] For each image, the stage patch features are extracted by the second multi-modal model, wherein the stage patch features are obtained from each layer output of the visual encoder of the second multi-modal model.

[0114] The stage patch features with the same image number and the same visual encoder layer within the preset neighborhood area are aggregated to obtain aggregated features.

[0115] The aggregated inference feature is obtained according to the aggregated feature corresponding to the inference image, and the aggregated reference feature is obtained according to the aggregated feature corresponding to the reference image.

[0116] Features of B unlabeled test images D = {I u} are extracted simultaneously using a second multi-modal model, where u = 1,..., B. For a CLIP encoder with L layers, multi-stage patch features where i ∈ {0, 1,..., L} represents the layers of the CLIP visual encoder.

[0117] By aggregating adjacent features, the detection accuracy of various size anomalies is improved. Specifically, the patch label can be reshaped into H x W x D. Average pooling is used to aggregate the stage patch features in the r x r neighborhood of the current position to obtain the aggregated aggregated feature and which is then reshaped into and where u represents the number of inference images, v represents the number of reference images, i represents the number of visual encoder layers, and r represents the neighborhood area.

[0118] where the stage patch feature represents the feature obtained by the visual encoder at different layers and In an embodiment, the preset neighborhood area has multiple, for example, the preset neighborhood area r = 1, 3, 5. That is, the patch label features in the r x r neighborhood of the current position are obtained when the preset neighborhood area is 1, 3 and 5 respectively.

[0119] Optionally, the noise feature filtering of the aggregated reference feature according to the initial mask obtains a non-anomalous aggregated reference feature, and the initial anomaly score is obtained according to the non-anomalous aggregated reference feature and the aggregated inference feature, comprising:

[0120] According to the initial mask, the aggregated reference feature with abnormal features is filtered out, and the non-anomalous aggregated reference feature and the corresponding mask are retained.

[0121] Calculate the feature distance between the aggregated inference feature and the non-anomalous aggregated reference feature, and take the smallest feature distance as the initial anomaly score.

[0122] The aggregated reference feature with abnormal features is filtered out by the initial mask, represented as:

[0123]

[0124] where M ∈ R HW represents a mask, P a represents an anomaly probability, Pn represents a normal probability.

[0125] The normal feature can be defined as:

[0126]

[0127] wherein, represents a filtered aggregated reference feature, N' represents the length of the normal patch, and M represents the filtered mask.

[0128] The initial anomaly score is represented as:

[0129]

[0130] wherein, represents the initial anomaly score.

[0131] Optionally, as shown in Figure 3 mask refinement according to the initial anomaly score and the initial mask to obtain a final inference mask includes:

[0132] In the inference image and the reference image, an average anomaly score is generated according to the extracted stage patch features, wherein the average anomaly score includes an inference average anomaly score and a reference average anomaly score.

[0133] The average anomaly score is averaged with the initial anomaly score respectively to obtain a refined inference anomaly score and a refined reference anomaly score under each neighborhood area.

[0134] The refined inference anomaly score and the refined reference anomaly score that exceed a refinement threshold are binarized to obtain an inference intermediate mask and a reference intermediate mask.

[0135] The inference intermediate mask and the reference intermediate mask are respectively subjected to collaborative voting to obtain a final inference mask and a final reference mask.

[0136] Features extracted from multiple stages and different aggregation types and use the anomaly score to refine the anomaly mask M. Specifically, the average anomaly score, i.e., the refined inference anomaly score and the refined reference anomaly score, is generated using the patch features of all stages. For example, in an embodiment of the present application, the average anomaly score is generated through the patch features of four stages, represented as:

[0137]

[0138] wherein m represents the number of stages, u represents the number of inference images, v represents the number of reference images, i represents the number of layers of the multi-modal model, and there are L layers in total, denotes a refined inference anomaly score, denotes a refined reference anomaly score.

[0139] The intermediate masks include an inference intermediate mask and a reference intermediate mask, and the intermediate mask is defined as:

[0140]

[0141] wherein M inter denotes an intermediate mask, denotes an anomaly score of an image under a neighborhood r, and μ denotes a hyperparameter, i.e., a refined threshold, and the value is set according to specific requirements.

[0142] Optionally, the hyperparameter μ = 0.57.

[0143] Then, a collaborative voting is applied to the intermediate mask Minter to refine a final inference mask M u and a final reference mask M v .

[0144] Optionally, the collaborative voting of the inference intermediate mask and the reference intermediate mask respectively to obtain the final inference mask and the final reference mask includes:

[0145] Obtaining the inference intermediate mask and the reference intermediate mask under each preset neighborhood area.

[0146] Collaborative voting is performed on the inference intermediate mask under each preset neighborhood area to obtain the final inference mask.

[0147] Collaborative voting is performed on the reference intermediate mask under each preset neighborhood area to obtain the final reference mask.

[0148] The inference intermediate mask and the reference intermediate mask are calculated for each preset neighborhood area respectively, and the inference intermediate mask and the reference intermediate mask under each neighborhood area are collaboratively voted to iteratively obtain the final inference mask and the final reference mask.

[0149] The embodiment of the application provides a zero-shot anomaly detection device based on a visual and language model, which comprises:

[0150] An acquisition module is configured to acquire an inference image and a reference image from the same batch of images, wherein the inference image represents an image to be detected for anomaly, and the reference image includes other images in the same batch of images except the inference image.

[0151] The first detection module is configured to perform double-modal detection on the inference image and the reference image respectively by using a first multi-modal model to obtain an initial mask, wherein the initial mask includes an initial inference abnormality mask and an initial reference abnormality mask.

[0152] The second detection module is configured to extract features of the same batch of images respectively by using a second multi-modal model and perform feature aggregation respectively to obtain aggregated inference features and aggregated reference features.

[0153] The noise feature filtering module is configured to perform noise feature filtering on the aggregated reference features according to the initial mask to obtain non-abnormal aggregated reference features, and obtain an initial abnormality score according to the non-abnormal aggregated reference features and the aggregated inference features, wherein the initial abnormality score includes an inference abnormality score and a reference abnormality score.

[0154] The refining module is configured to perform mask refining according to the initial abnormality score and the initial mask to obtain a final inference mask, wherein the final inference mask is used to represent an abnormality detection inference result.

[0155] As shown in Figure 4 The electronic device 400 provided by the embodiment of the present application includes a memory 410 and a processor 420; the memory 410 is used to store a computer program; the processor 420 is used to implement the xxx method as described above when the computer program is executed.

[0156] Alternatively, the electronic device 400 includes a memory 410 and a processor 420 coupled to the memory 410; the memory 410 is configured to store a computer program; the processor 420 is configured to perform the following operations when the computer program is executed:

[0157] An inference image and a reference image are obtained from the same batch of images, wherein the inference image represents an image to be detected for abnormalities, and the reference image includes other images in the same batch of images except the inference image;

[0158] Double-modal detection is performed on the inference image and the reference image respectively by using a first multi-modal model to obtain an initial mask, wherein the initial mask includes an initial inference abnormality mask and an initial reference abnormality mask.

[0159] Features of the same batch of images are extracted respectively by using a second multi-modal model, and feature aggregation is performed respectively to obtain aggregated inference features and aggregated reference features.

[0160] According to the initial mask, noise feature filtering is performed on the aggregated reference feature to obtain a non-abnormal aggregated reference feature, and an initial abnormal score is obtained according to the non-abnormal aggregated reference feature and the aggregated inference feature, wherein the initial abnormal score includes an inference abnormal score and a reference abnormal score;

[0161] According to the initial abnormal score and the initial mask, mask refining is performed to obtain a final inference mask, wherein the final inference mask is used to represent an abnormality detection inference result.

[0162] The embodiment of the application provides a computer readable storage medium, and the storage medium stores a computer program.

[0163] Alternatively, a non-volatile computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following operations:

[0164] An inference image and a reference image are obtained from the same batch of images, wherein the inference image represents an image to be detected for abnormality, and the reference image includes other images in the same batch of images except the inference image;

[0165] An initial mask is obtained by performing double-modal detection on the inference image and the reference image respectively through a first multi-modal model, wherein the initial mask includes an initial inference abnormal mask and an initial reference abnormal mask;

[0166] Features of the same batch of images are extracted and aggregated respectively through a second multi-modal model to obtain aggregated inference features and aggregated reference features;

[0167] According to the initial mask, noise feature filtering is performed on the aggregated reference feature to obtain a non-abnormal aggregated reference feature, and an initial abnormal score is obtained according to the non-abnormal aggregated reference feature and the aggregated inference feature, wherein the initial abnormal score includes an inference abnormal score and a reference abnormal score;

[0168] According to the initial abnormal score and the initial mask, mask refining is performed to obtain a final inference mask, wherein the final inference mask is used to represent an abnormality detection inference result.

[0169] An electronic device 400, which can be a server or a client of the present application, will now be described, which is an example of a hardware device that can be applied to aspects of the present application. The electronic device 400 is intended to represent various forms of digital electronic computer devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device 400 can also represent various forms of mobile devices such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0170] The electronic device 400 includes a computing unit that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) or a computer program loaded into a random access memory (RAM) from a storage unit. In the RAM, various programs and data required for device operation can also be stored. The computing unit, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0171] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), or a random access memory (RAM), etc. In this application, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application. In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0172] Although the present application is disclosed as above, the protection scope of the present application is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application, and these changes and modifications will fall within the protection scope of the present application.

Claims

1. A zero-shot anomaly detection method based on vision and language models, characterized in that, The method comprises the following steps: obtaining an inference image and a reference image from the same batch of images, wherein the inference image represents an image to be detected for abnormalities, and the reference image comprises other images in the same batch of images except the inference image; performing image modality detection on the inference image and the reference image respectively through a first multi-modal model, performing text modality detection on the inference image and the reference image respectively through the first multi-modal model according to a normal text prompt template and an abnormal text prompt template respectively, and obtaining an initial mask, wherein the initial mask comprises an initial inference abnormality mask and an initial reference abnormality mask, the initial inference abnormality mask is used to represent the probability that the inference image and the reference image are classified as normal, and the initial reference abnormality mask is used to represent the probability that the inference image and the reference image are classified as abnormal; extracting features of the same batch of images through a second multi-modal model and performing feature aggregation respectively, and obtaining aggregated inference features and aggregated reference features; performing noise feature filtering on the aggregated reference features according to the initial mask to obtain non-abnormal aggregated reference features, and obtaining an initial abnormality score according to the non-abnormal aggregated reference features and the aggregated inference features, wherein the initial abnormality score comprises an inference abnormality score and a reference abnormality score; performing mask refining according to the initial abnormality score and the initial mask to obtain a final inference mask, wherein the final inference mask is used to represent an abnormality detection inference result.

2. The vision and language model based zero-shot anomaly detection method of claim 1, wherein, Before the step of performing image modality detection on the inference image and the reference image respectively through a first multi-modal model, performing text modality detection on the inference image and the reference image respectively through the first multi-modal model according to a normal text prompt template and an abnormal text prompt template, and obtaining an initial mask, the method further comprises the following steps: determining an initial multi-modal model; removing residual connections and feedforward modules in the initial multi-modal model; weighting attention features of an intermediate layer of the initial multi-modal model to replace attention features of a last layer to obtain the first multi-modal model, wherein the intermediate layer comprises a specific layer in the initial multi-modal model.

3. The vision and language model based zero-shot anomaly detection method of claim 1, wherein, Before the step of performing image modality detection on the inference image and the reference image respectively through a first multi-modal model, performing text modality detection on the inference image and the reference image respectively through the first multi-modal model according to a normal text prompt template and an abnormal text prompt template, and obtaining an initial mask, the method further comprises the following steps: constructing the normal text prompt template and the abnormal text prompt template; extracting normal text features and abnormal text features in the normal text prompt template and the abnormal text prompt template through the first multi-modal model respectively.

4. The method of claim 3, wherein the visual and language model based zero-shot anomaly detection method is characterized by, The text features comprise the normal text features and the abnormal text features, and the step of performing image modality detection on the inference image and the reference image respectively through a first multi-modal model, performing text modality detection on the inference image and the reference image respectively through the first multi-modal model according to a normal text prompt template and an abnormal text prompt template, and obtaining an initial mask comprises the following steps: extracting inference image features and reference image features of the inference image and the reference image through the first multi-modal model respectively; calculate an inference similarity and a reference similarity between the inference image feature and the text feature, and between the reference image feature and the text feature, respectively; determine the initial mask according to the inference similarity and the reference similarity.

5. The vision and language model based zero-shot anomaly detection method of claim 1, wherein, the extracting features of the same batch of images by the second multi-modal model and aggregating the features respectively to obtain aggregated inference features and aggregated reference features comprises: for each image, extracting stage patch features by the second multi-modal model, wherein the stage patch features are obtained from the output of each layer of the visual encoder of the second multi-modal model; aggregating the stage patch features of the same image number and the same visual encoder layer within a preset neighborhood area to obtain aggregated features; obtaining the aggregated inference features according to the aggregated features corresponding to the inference images, and obtaining the aggregated reference features according to the aggregated features corresponding to the reference images.

6. The vision and language model based zero-shot anomaly detection method of claim 1, wherein, the filtering noise features from the aggregated reference features according to the initial mask to obtain non-anomalous aggregated reference features, and obtaining an initial anomaly score according to the non-anomalous aggregated reference features and the aggregated inference features comprises: filtering out the aggregated reference features with abnormal features according to the initial mask to retain non-anomalous aggregated reference features; calculating the feature distance between the aggregated inference features and the non-anomalous aggregated reference features, and taking the smallest feature distance as the initial anomaly score.

7. The vision and language model based zero-shot anomaly detection method of claim 1, wherein, the mask refining according to the initial anomaly score and the initial mask to obtain a final inference mask comprises: generating average anomaly scores according to the extracted stage patch features in the inference images and the reference images, wherein the average anomaly scores include inference average anomaly scores and reference average anomaly scores; averaging the average anomaly scores with initial anomaly scores respectively to obtain refined inference anomaly scores and refined reference anomaly scores under each neighborhood area; performing binary processing on the refined inference anomaly scores and the refined reference anomaly scores that exceed a refined threshold to obtain an inference intermediate mask and a reference intermediate mask; performing collaborative voting on the inference intermediate mask and the reference intermediate mask respectively to obtain a final inference mask and a final reference mask.

8. The vision and language model based zero-shot anomaly detection method of claim 7, wherein, the collaborative voting on the inference intermediate mask and the reference intermediate mask respectively to obtain a final inference mask and a final reference mask comprises: obtaining the inference intermediate mask and the reference intermediate mask under each preset neighborhood area; performing collaborative voting on the inference intermediate mask under each preset neighborhood area to obtain the final inference mask; performing collaborative voting on the reference intermediate mask under each preset neighborhood area to obtain the final reference mask.

9. A zero-shot anomaly detection apparatus based on vision and language models, comprising: comprises: an obtaining module, configured to obtain inference images and reference images from the same batch of images, wherein the inference images represent images to be detected for anomalies, and the reference images include other images in the same batch of images except the inference images; The first detection module is configured to perform image modality detection on the inference image and the reference image respectively by using a first multi-modal model, perform text modality detection according to a normal text prompt template and an abnormal text prompt template respectively by using the first multi-modal model, and obtain an initial mask, wherein the initial mask includes an initial inference abnormal mask and an initial reference abnormal mask, the initial inference abnormal mask is used to represent a probability that the inference image and the reference image are classified as normal, and the initial reference abnormal mask is used to represent a probability that the inference image and the reference image are classified as abnormal. The second detection module is configured to extract features of the same batch of images respectively by using a second multi-modal model and aggregate the features respectively, and obtain aggregated inference features and aggregated reference features. The noise feature filtering module is configured to perform noise feature filtering on the aggregated reference features according to the initial mask, obtain non-abnormal aggregated reference features, and obtain an initial abnormal score according to the non-abnormal aggregated reference features and the aggregated inference features, wherein the initial abnormal score includes an inference abnormal score and a reference abnormal score. The refining module is configured to perform mask refining according to the initial abnormal score and the initial mask, and obtain a final inference mask, wherein the final inference mask is used to represent an abnormal detection inference result.

10. An electronic device, comprising: The device comprises a memory and a processor. The memory is configured to store a computer program. The processor is configured to implement the zero-shot abnormality detection method based on a visual and language model when the computer program is executed. The device comprises a memory and a processor. The memory is configured to store a computer program. The processor is configured to implement the zero-shot abnormality detection method based on a visual and language model when the computer program is executed.

Images