Multi-modal small sample data driven anomaly detection method and system and storage medium

Abstract

The invention belongs to the technical field of medical image processing, and particularly relates to a multi-modal small sample data driven anomaly detection method and system and a storage medium. The system of the present invention comprises: an input module configured to input a test image; the character description module is configured to input the test image into an abnormal diffusion-free model to obtain character description; the image customization module is configured to convert the test image into a series of normal images in normal distribution to obtain a customized image; and the prediction module is configured to respectively compare the test image with the customized image, compare the test image with the text description, and compare the test image with a normal image sample set to obtain three comparison scores, and fuse the three comparison scores to obtain a final prediction result. According to the method, more accurate prediction performance can be realized on the premise that only a small number of normal image samples are applied, and the method has a good application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a multimodal small sample data driven anomaly detection method, system and storage medium. Background Art

[0002] Anomaly detection (AD) has attracted a lot of attention due to its wide applicability in various fields such as industrial defect detection, medical diagnosis, video surveillance, manufacturing inspection, etc. This high interest is mainly attributed to its reliance on positive samples and the adoption of an unsupervised learning paradigm, in which it usually learns from the distribution of positive samples to identify anomalies by detecting outliers. Traditional AD methods, including autoencoder-based, GAN-based, and knowledge-based methods, have emerged recently. In addition, some diffusion-based methods have also emerged recently. Although most of these methods do not require annotated data, they require a large number of normal samples in the training phase to effectively capture the distribution of normal samples. This requirement for large sample size data has also severely restricted its development in various fields.

[0003] Recently, there have been many explorations in the field of small samples. Especially with the rise of visual language pre-trained large models (VLM), many works on small sample anomaly detection based on VLM have achieved state-of-the-art results. WinCLIP first uses pre-trained CLIP to complete small sample anomaly detection by carefully designing text prompts and image feature comparisons. AnomalyGPT eliminates the disadvantage of manually setting thresholds and supports multi-round dialogues. InCTRL uses contextual learning to achieve general anomaly detection.

[0004] However, the latest work on small sample anomaly detection in this category uses a rough direct match and feature comparison between several normal images (reference images) and test samples in terms of vision, which will affect the accuracy of the results and make it difficult to generalize the samples to other complex fields. This is because the direct comparison between the test image and several small sample reference images may be very different, and the prediction results obtained by each test sample will be affected by the position, angle, size, color, etc. of the randomly selected reference images (several small sample normal images). Secondly, due to the small number of reference images, only a few images are not representative enough as independent references, and of course, there is no in-depth exploration of their feature distribution, which makes it difficult to meet accurate feature comparisons and easily leads to unstable results.

[0005] Therefore, for anomaly detection tasks, how to design models and methods to expand the sample size of small sample data so that the model can obtain accurate results based on small sample data as reference images is still an important research topic in this field. Summary of the invention

[0006] In view of the problems in the prior art, the present invention provides a multimodal small sample data driven anomaly detection method, system and storage medium.

[0007] A multimodal small sample data driven anomaly detection system, comprising: The input module is configured to input a test image and a positive sample text description of the corresponding category; An image customization module is configured to transform the test image into a series of normal images with normal distribution using a non-abnormal diffusion model to obtain a customized image; The prediction module is configured to respectively compare the test image with the customized image, compare the test image with the text description, and compare the test image with a normal image sample set to obtain three comparison scores, and fuse the three comparison scores to obtain a final prediction result.

[0008] Preferably, the test image and the normal image are medical diagnostic images.

[0009] Preferably, the anomaly-free diffusion model is a model parameterized by θ on class o , through a series of normal image-text demonstration pairs consisting of C o To constrain and fine-tune the training of the anomaly-free diffusion model, the formula is as follows: Where θ is the parameter of the customized anomaly-free diffusion model on category o, o is the category, x n,t is the normal reference image input After t noise addition steps, the potential version c is the normal image x n A brief text description of C o For image-text demonstration pairs, is the expected value, ( , ) is a normal image and text pair, is the standard Gaussian noise added to the noisy image, and t is the noise adding time step.

[0010] Preferably, in the image customization module, the method of converting the test image into a series of normal images with normal distribution comprises the following steps: Step 1: Pass the test image through the image encoder of the non-abnormal diffusion model, and then go through the noise adding process to obtain a noisy data z t ; Step 2: Add the positive sample text description of the corresponding category as a condition to guide the non-abnormal diffusion model to z t Perform progressive denoising; Step 3: The denoised z tThe image is converted into the customized image through the image decoder.

[0011] Preferably, in the prediction module, three comparison scores are obtained by the CLIP model.

[0012] Preferably, the comparison score S obtained by comparing the test image with the custom image is p The calculation method is as follows: in, l represents the number of layers, and n represents the number of feature extraction blocks. represents the cosine similarity function, Represents the test image x Extracted from l Layer features, represents the features extracted from the custom image; And / or, using the same CLIP image encoder to extract multi-level features from the normal sample set, and storing these features in a repository M; comparing the test image with the normal image sample set to obtain a comparison score S N The calculation method is as follows: Among them, m represents the sample features sampled from the repository M. Representative l Characteristics of the layer; And / or, a comparison score S obtained by comparing the test image with a normal image sample set text The calculation method is as follows: in, represents a set of normal and abnormal prompts obtained through the CLIP text encoder, and softmax(·) is the average softmax score across multiple levels.

[0013] Preferably, the formula for fusing the three comparison scores is: in, is the score after fusion, and α and β are weights respectively.

[0014] Preferably, the normal image The average value of is used as the threshold to get the final prediction result.

[0015] The present invention also provides a method for performing anomaly detection using the above-mentioned multimodal small sample data driven anomaly detection system, comprising the following steps: Inputting the test image into a non-abnormal diffusion model to obtain a text description; Converting the test image into a series of normal images with normal distribution to obtain a customized image; The test image is respectively compared with the customized image, the test image is compared with the text description, and the test image is compared with a normal image sample set to obtain three comparison scores, and the three comparison scores are fused to obtain a final prediction result.

[0016] The present invention also provides a computer-readable storage medium, on which is stored: a computer program for implementing the above method.

[0017] The present invention constructs a preferred model structure and method for anomaly detection driven by small sample data. Specifically, more series of normal images with normal distribution are constructed through test images, thereby effectively expanding the normal image samples that can be compared. In the process of obtaining the final anomaly detection result, the present invention combines three aspects of information, respectively comparing the test image with the customized image, comparing the test image with the text description, and comparing the test image with the normal image sample set. Finally, the three comparison results are fused to obtain the final prediction result.

[0018] According to the technical solution of the present invention, it is possible to more accurately realize anomaly detection of small sample size tasks, and due to the use of triple contrast anomaly reasoning, the method has better stability and robustness. For example, in the field of medical diagnosis, the present invention can use only a few (small sample) unlabeled normal medical images for model learning, and can efficiently, accurately and stably identify different modalities (for example: MRI, CT, OCT), different categories of medical diseases (for example: fundus, brain, lungs), and can accurately segment their parts. Therefore, the present invention has a good application prospect.

[0019] Obviously, according to the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, other various forms of modification, replacement or change may be made.

[0020] The above contents of the present invention are further described in detail below through specific implementation methods in the form of embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples. All technologies realized based on the above contents of the present invention belong to the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0022] It should be noted that the algorithms of data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structure, circuit connection, etc. not specifically described can all be implemented through the contents disclosed in the prior art.

[0023] Embodiment Small sample data driven anomaly detection method and system The system of this embodiment includes: The input module is configured to input a test image and a positive sample text description of the corresponding category; An image customization module is configured to transform the test image into a series of normal images with normal distribution using a non-abnormal diffusion model to obtain a customized image; The prediction module is configured to respectively compare the test image with the customized image, compare the test image with the text description, and compare the test image with a normal image sample set to obtain three comparison scores, and fuse the three comparison scores to obtain a final prediction result.

[0024] The system of this embodiment can be used for anomaly detection in various fields. Taking the detection of medical images as an example, only a few (small sample) unlabeled normal medical images can be used for model learning to efficiently, accurately and stably identify different modalities (for example: MRI, CT, OCT), different types of medical diseases (for example: fundus, brain, lungs), and accurately segment their parts.

[0025] Specifically, the method of applying the above system to perform anomaly detection is as follows: Figure 1 As shown, the following steps are included: Step S1, input the test image into the non-abnormal diffusion model to obtain a text description: The method is first based on the stable diffusion model, and then a custom anomaly-free diffusion model based on dreambooth (trained with a small number of normal images and their text descriptions) is used to mine the distribution of reference images. In order to better align categories with their text descriptions, a series of demonstrations Co are provided for the diffusion model, using a small sample of reference images paired with the corresponding text of a specific class.

[0026] The specific steps are as follows: For each class, a series of small sample reference images (i.e., 2-8 normal images) and their matching text c are prepared, for example: a normal [category] image, where [category] can be replaced by retina, brain, skin, etc., depending on the category of the input image, for customized training of the diffusion model.

[0027] The anomaly-free diffusion model is a model parameterized by θ on class o , through a series of normal image-text demonstration pairs consisting of Co To constrain and fine-tune the training of the anomaly-free diffusion model, the formula is as follows: Where θ is the parameter of the customized anomaly-free diffusion model on category o, o is the category, x n,t is the normal reference image input After t noise addition steps, the potential version c is the normal image x n A brief text description of C o For image-text demonstration pairs, is the expected value, ( , ) is a normal image and text pair, ϵ is the standard Gaussian noise added to the noisy image, and t is the noise adding time step.

[0028] In summary, the dimensions of the image and the latent image are the same throughout the process. Therefore, the anomaly-free diffusion model learns the distribution of normal images under the text prompt c, which can be used in the process of converting customized images in the subsequent steps.

[0029] Step S2, converting the test image into a series of normal images with normal distribution to obtain a customized image: This embodiment proposes a customized conversion strategy from one to normal in order to convert a test image into a positive sample distribution similar to it. It adaptively maintains the characteristics of its original normal part, while the abnormal part gradually transforms the normal part. These processes are all completed using the anomaly-free diffusion model obtained in the previous step. Specifically, text prompts are first designed to maximize the retention of information in the normal area of ​​the test image while converting the abnormal area to a normal state. In order to mitigate the impact of other factors (such as contrast, image quality), all potential normal states are simulated using these prompts, and a template list is curated for various potential image physical states.

[0030] The specific process of conversion is as follows: Step 1: Pass the test image through the image encoder of the non-abnormal diffusion model, and then go through the noise adding process to obtain a noisy data z t ; Step 2: Add the positive sample text description of the corresponding category as a condition to guide the non-abnormal diffusion model to z t Perform progressive denoising; Step 3: The denoised z t The image is converted into the customized image through the image decoder.

[0031] Step S3, respectively comparing the test image with the customized image, comparing the test image with the text description, and comparing the test image with a normal image sample set to obtain three comparison scores, and fusing the three comparison scores to obtain a final prediction result: This step obtains three comparison scores through the CLIP model.

[0032] Comparison between test image and custom image: In this embodiment, the image encoder of the CLIP model is divided into n multi-feature extraction blocks to extract multi-layer features, and then the comparison results are compared on the multi-layer features. , this CLIP image encoder extracts their features respectively and , and finally compare the scores The calculation is as follows: in, l represents the number of layers, and n represents the number of feature extraction blocks. represents the cosine similarity function, Represents the test image x Extracted from l Layer features, represents the features extracted from the custom image; Comparison between test images and normal image sample sets: The data pool of normal images (normal image sample set) is a non-anomaly sample pool, which consists of normal reference images and generated normal images, which better represent the distribution of positive samples. Among them, the generated normal images are normal samples generated from the above-mentioned non-anomaly diffusion model, and are included in the data pool for the prediction task in this step, because many previous studies have demonstrated the ability of diffusion models to synthesize high-fidelity images. Then, the same CLIP image encoder is used to extract multi-level features from the non-anomaly samples in the data pool, and these features are stored in the repository M. The prediction score S between the test image and the data pool sample N It can be expressed as: Among them, m represents the sample features sampled from the repository M. Representative l Characteristics of the layer; Comparison of test images and text descriptions: To calculate the anomaly score between test images and text prompts, the text prompts are divided into two types: normal text and abnormal text. The goal is to cover more possible states for these objects to better model various potential conditions of the image. Specifically, the text feature F text∈R2×d is a set of normal and abnormal cues obtained by the CLIP text encoder and then compared with the features in the test image: in, represents a set of normal and abnormal prompts obtained through the CLIP text encoder, and softmax(·) is the average softmax score across multiple levels.

[0033] The final prediction result is obtained by combining the scores of the above three branches. The formula for fusing the three comparison scores is: in, is the score after fusion, α and β are weights respectively. As a preferred method, the default .

[0034] In this way, each test image gets an anomaly score ， The higher the anomaly score, the more likely it is an abnormal (lesion) image. The higher the pixel-level anomaly value in a test image, the greater the possibility that the area is a lesion area. In this embodiment, the average value of the abnormal value of the normal sample set image is used as the threshold to obtain the final lesion detection and segmentation results.

[0035] This embodiment further verifies the effectiveness of the above system and method in the medical field, industrial field and semantic field. In the medical field, we cover datasets of various modalities, such as magnetic resonance imaging (MRI), computed tomography (CT) and optical coherence tomography (OCT). Specifically, medical datasets include OCT2017, BrainMRI, HeadCT and RESC. In the industrial field, we utilize multiple datasets, including MVTec-AD, Visa, KSDD, AFID and ELPV. For semantic anomaly detection, we use two datasets: MNIST and CIFAR-10.

[0036] The performance of anomaly detection is evaluated by calculating the value of the receiver operating characteristic curve (AUROC). The larger the AUROC value, the better the model performance.

[0037] We compare our proposed method with various state-of-the-art anomaly detection (AD) methods reported in the literature, including PaDiM, PatchCore, RegAD, CoOp, WinCLIP, and InCTRL.

[0038] This example selects 2, 4, and 8 samples as the training set to verify the test set, and the resolution is set to 240. All experiments are completed using PyTorch on an NVIDIA GeForce RTX 4090 GPU.

[0039] The experimental results are as follows: 1. The AUROC results of the method in this embodiment compared with PaDiM, PatchCore, RegAD, WinCLIP and InCTRL on medical data sets (OCT2017, BrainMRI, HeadCT and RESC) are as follows: Table 1 It can be concluded from the table that the method of this embodiment has improvements on the four medical data sets.

[0040] 2. The AUROC results of the comparison methods of this embodiment, PaDiM, PatchCore, RegAD, WinCLIP and InCTRL on industrial data sets (MVTec-AD, Visa, KSDD, AFID and ELPV) are as follows: Table 2 It can be concluded from the table that, in most cases, the method of this embodiment is the best on the five industrial data sets.

[0041] 3. The AUROC results of the method in this embodiment compared with other methods on semantic datasets (MNIST and CIFAR-10) are as follows: Table 3 It can be concluded from the table that the method of this embodiment is optimal under various settings in the semantic data.

[0042] From the above embodiments, it can be seen that the present invention constructs an anomaly detection method and system for small sample size tasks, which can achieve more accurate prediction performance under the premise of using only a small number of normal image samples. Therefore, the present invention has a good application prospect.

Claims

1. A multimodal small sample data driven anomaly detection system, characterized in that: include: The input module is configured to input a test image and a positive sample text description of the corresponding category; An image customization module is configured to convert the test image into a series of normal images with normal distribution using a non-abnormal diffusion model to obtain a customized image; The prediction module is configured to respectively compare the test image with the customized image, compare the test image with the text description, and compare the test image with a normal image sample set to obtain three comparison scores, and fuse the three comparison scores to obtain a final prediction result.

2. The multimodal small sample data driven anomaly detection system according to claim 1, characterized in that: The test image and the normal image are medical diagnosis images.

3. The multimodal small sample data driven anomaly detection system according to claim 1, characterized in that: The anomaly-free diffusion model is a model parameterized by θ on class o , through a series of normal image-text demonstration pairs consisting of C o To constrain and fine-tune the training of the anomaly-free diffusion model, the formula is as follows: Where θ is the parameter of the customized anomaly-free diffusion model on category o, o is the category, x n,t is the normal reference image input After t noise addition steps, the potential version c is the normal image x n A brief text description of C o For image-text demonstration pairs, is the expected value, ( , ) is a normal image and text pair, is the standard Gaussian noise added to the noisy image, and t is the noise adding time step.

4. The multimodal small sample data driven anomaly detection system according to claim 1, characterized in that: In the image customization module, the method for converting the test image into a series of normal images with normal distribution comprises the following steps: Step 1: Pass the test image through the image encoder of the non-abnormal diffusion model, and then go through the noise adding process to obtain a noisy data z t ; Step 2: Add the positive sample text description of the corresponding category as a condition to guide the non-abnormal diffusion model to z t Perform progressive denoising; Step 3: The denoised z t The image is converted into the customized image through the image decoder.

5. The multimodal small sample data driven anomaly detection system according to claim 1, characterized in that: In the prediction module, three comparison scores are obtained through the CLIP model.

6. The multimodal small sample data driven anomaly detection system according to claim 5, characterized in that: The comparison score S obtained by comparing the test image with the custom image p The calculation method is as follows: in, l represents the number of layers, n represents the number of feature extraction blocks, represents the cosine similarity function, Represents the test image x Extracted from l Layer features, represents the features extracted from the custom image; And / or, using the same CLIP image encoder to extract multi-level features from the normal sample set, and storing these features in a repository M; comparing the test image with the normal image sample set to obtain a comparison score S N The calculation method is as follows: Among them, m represents the sample features sampled from the repository M. Representative l Characteristics of the layer; and / or, comparing the test image with a normal image sample set to obtain a comparison score S text The calculation method is as follows: in, represents a set of normal and abnormal prompts obtained through the CLIP text encoder, and softmax(·) is the average softmax score across multiple levels.

7. The multimodal small sample data driven anomaly detection system according to claim 6, characterized in that: The formula for fusing the three contrast scores is: in, is the score after fusion, and α and β are weights respectively.

8. The multimodal small sample data driven anomaly detection system according to claim 7, characterized in that: The normal image The average value of is used as the threshold to obtain the final lesion detection and segmentation results.

9. A method for performing anomaly detection using the multimodal small sample data driven anomaly detection system according to any one of claims 1 to 8, characterized in that: The steps include: Inputting the test image into a non-abnormal diffusion model to obtain a text description; Converting the test image into a series of normal images with normal distribution to obtain a customized image; The test image is respectively compared with the customized image, the test image is compared with the text description, and the test image is compared with a normal image sample set to obtain three comparison scores, and the three comparison scores are fused to obtain a final prediction result.

10. A computer-readable storage medium, characterized in that: Stored thereon is a computer program for implementing the method described in claim 9.

Images