A multi-scale feature aggregation and semantic guidance based zero-shot anomaly detection method

Abstract

The application discloses a kind of multi-scale feature aggregation and semantic guiding zero sample exception detection method, wherein the exception image detection model in the method is obtained by the following steps: obtaining image training set, image reference set and text category feature;Multi-scale window feature extraction is carried out on image reference set and image training set, to obtain image feature set;According to text category feature, multi-scale semantic alignment is carried out on first image feature set, to obtain semantic guiding prediction representation;Multi-scale similarity aggregation is carried out on image feature set, to obtain exception segmentation prediction representation;According to semantic guiding prediction representation and exception segmentation prediction representation, the parameter of initialized exception image detection model is updated, to obtain trained exception image detection model.The method can effectively reduce the sample data amount required during model training, improve model training efficiency, and improve the detection accuracy of the model in the few-sample scenario, with high scalability.The present application relates to the technical field of computer vision.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance. Background Technology

[0002] In the field of automated industrial quality control, anomaly detection technology is crucial for improving manufacturing efficiency and product reliability. This technology is widely used in various industrial sectors such as aerospace, automotive, pharmaceuticals, and electronics, involving a great diversity of objects and their defects, including variations in color, texture, size, and other dimensions.

[0003] Currently, traditional anomaly detection techniques mainly rely on models customized for specific tasks. Good anomaly detection results can only be obtained by training specific models. This requires a large amount of sample data, has low training efficiency, low accuracy in scenarios with few samples, and low scalability.

[0004] Therefore, the problems existing in the current technology still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance. This method can effectively reduce the amount of sample data required for model training and improve model training efficiency, while also improving the detection accuracy of the model in low-sample scenarios and exhibiting high scalability.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0008] In a first aspect, embodiments of this application provide a zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance, including:

[0009] Acquire the visual image to be detected;

[0010] The visual image is input into a trained abnormal image detection model to obtain the image detection result;

[0011] The trained abnormal image detection model is obtained through the following steps:

[0012] Acquire the image dataset, as well as the preset text prompt templates and preset image reference sets;

[0013] The image dataset is preprocessed to obtain an image training set, and the text prompt template is subjected to text feature extraction to obtain text category features;

[0014] Multi-scale window feature extraction is performed on the image training set to obtain a first image feature set, and multi-scale window feature extraction is performed on the image reference set to obtain a second image feature set;

[0015] Based on the text category features, multi-scale semantic alignment is performed on the first image feature set to obtain a semantically guided prediction representation;

[0016] Based on the second image feature set, multi-scale similarity aggregation is performed on the first image feature set to obtain an anomaly segmentation prediction representation;

[0017] Based on the semantically guided prediction representation and the anomaly segmentation prediction representation, the parameters of the initialized anomaly image detection model are updated to obtain a trained anomaly image detection model.

[0018] In addition, the multi-scale feature aggregation and semantically guided zero-shot anomaly detection method according to the above embodiments of this application may also have the following additional technical features:

[0019] Furthermore, in one embodiment of this application, the preprocessing of the image dataset to obtain the image training set includes:

[0020] The image dataset is feature-scaled to obtain a scaled image set;

[0021] The image scaling set is interpolated and adjusted to obtain the image training set.

[0022] Furthermore, in one embodiment of this application, the step of extracting text features from the text prompt template to obtain text category features includes:

[0023] Get a text prompt learner;

[0024] Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, which is used to represent feature information that is unrelated to the semantics of the object.

[0025] Text extraction is performed on the text representation information to obtain the text category features.

[0026] Furthermore, in one embodiment of this application, multi-scale window feature extraction is performed on the target dataset to obtain a target feature set, including:

[0027] Obtain a preset number of sliding windows, each with a different size of its active area;

[0028] Based on all the sliding windows, window mapping is performed on each target image in the target dataset to obtain the target feature set, which is used to characterize window feature maps of different scales corresponding to each target image in the target dataset;

[0029] The target dataset is either an image training set or an image reference set.

[0030] Furthermore, in one embodiment of this application, the step of performing multi-scale semantic alignment on the first image feature set based on the text category features to obtain a semantically guided prediction representation includes:

[0031] Based on the text category features, the first image feature set is aligned to obtain the category-aligned text features and the category-aligned first image feature set;

[0032] Based on the text features aligned to the categories, matrix multiplication is performed on the first image feature set aligned to the categories to obtain anomaly window scores at different scales;

[0033] The abnormal window scores are aggregated using a harmonic average to obtain the semantically guided prediction representation.

[0034] Furthermore, in one embodiment of this application, the step of performing multi-scale similarity aggregation on the first image feature set based on the second image feature set to obtain an anomaly segmentation prediction representation includes:

[0035] Based on the first image feature set, first window feature maps of different scales are obtained, and based on the second image feature set, second window feature maps of different scales are obtained;

[0036] Based on the second window feature map, the similarity of the first window feature map at the corresponding scale is calculated to obtain the window feature similarity.

[0037] The anomaly segmentation prediction representation is obtained by integrating the window feature similarity.

[0038] Further, in one embodiment of this application, the step of updating the parameters of the initialized anomaly image detection model based on the semantically guided prediction representation and the anomaly segmentation prediction representation to obtain a trained anomaly image detection model includes:

[0039] Based on the anomaly segmentation prediction representation, the semantically guided prediction representation is expanded to obtain a few-sample anomaly score map.

[0040] Based on the few-sample anomaly score map, the parameters of the initialized anomaly image detection model are updated to obtain the trained anomaly image detection model.

[0041] Secondly, embodiments of this application provide a zero-shot anomaly detection system based on multi-scale feature aggregation and semantic guidance, including:

[0042] The acquisition module is used to acquire the visual image to be detected;

[0043] The processing module is used to input the visual image into the trained abnormal image detection model to obtain the image detection result;

[0044] The trained abnormal image detection model is obtained through the following steps:

[0045] Acquire the image dataset, as well as the preset text prompt templates and preset image reference sets;

[0046] The image dataset is preprocessed to obtain an image training set, and the text prompt template is subjected to text feature extraction to obtain text category features;

[0047] Multi-scale window feature extraction is performed on the image training set to obtain a first image feature set, and multi-scale window feature extraction is performed on the image reference set to obtain a second image feature set;

[0048] Based on the text category features, multi-scale semantic alignment is performed on the first image feature set to obtain a semantically guided prediction representation;

[0049] Based on the second image feature set, multi-scale similarity aggregation is performed on the first image feature set to obtain an anomaly segmentation prediction representation;

[0050] Based on the semantically guided prediction representation and the anomaly segmentation prediction representation, the parameters of the initialized anomaly image detection model are updated to obtain a trained anomaly image detection model.

[0051] Thirdly, embodiments of this application also provide an electronic device, including:

[0052] At least one processor;

[0053] At least one memory for storing at least one program;

[0054] When the at least one program is executed by the at least one processor, the at least one processor implements the method of the first aspect described above.

[0055] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the method of the first aspect described above.

[0056] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0057] This application discloses a zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance. The method involves acquiring a visual image to be detected; inputting the visual image into a trained anomaly image detection model to obtain an image detection result; wherein the trained anomaly image detection model is obtained through the following steps: acquiring an image dataset, a preset text prompt template, and a preset image reference set; preprocessing the image dataset to obtain an image training set, and extracting text features from the text prompt template to obtain text category features; extracting multi-scale window features from the image training set to obtain a first image feature set, and extracting multi-scale window features from the image reference set to obtain a second image feature set; performing multi-scale semantic alignment on the first image feature set based on the text category features to obtain a semantic guidance prediction representation; performing multi-scale similarity aggregation on the first image feature set based on the second image feature set to obtain an anomaly segmentation prediction representation; and updating the parameters of the initialized anomaly image detection model based on the semantic guidance prediction representation and the anomaly segmentation prediction representation to obtain a trained anomaly image detection model. This detection method is based on multi-scale semantic alignment, extracting semantically guided predictive representations that are closely aligned with image visual features and linguistic descriptions. This enables anomaly segmentation detection under unlabeled and zero-sample conditions, effectively reducing the amount of sample data required for model training and improving training efficiency and scalability. Furthermore, this method effectively integrates the intuitive visual information of normal reference images with the advantages of language-driven prediction based on anomaly segmentation predictive representations, significantly improving detection accuracy in low-sample scenarios. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0059] Figure 1 A flowchart illustrating a multi-scale feature aggregation and semantically guided zero-shot anomaly detection method provided in this application embodiment;

[0060] Figure 2 A schematic diagram illustrating the training process of an abnormal image detection model provided in an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of the structure of a zero-shot anomaly detection system with multi-scale feature aggregation and semantic guidance provided in an embodiment of this application;

[0062] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] Currently, traditional anomaly detection techniques mainly rely on models customized for specific tasks. Good anomaly detection results can only be obtained by training specific models. This requires a large amount of sample data, has low training efficiency, low accuracy in scenarios with few samples, and low scalability.

[0066] In view of this, embodiments of the present invention provide a zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance. This detection method is based on multi-scale semantic alignment, extracting semantically guided predictive representations that are closely aligned with image visual features and language descriptions, thereby achieving anomaly segmentation detection under unlabeled and zero-shot conditions. It can effectively reduce the amount of sample data required for model training, improving training efficiency and scalability. Furthermore, this detection method also effectively integrates the intuitive visual information of normal reference images with the predictive advantages of language-driven methods based on anomaly segmentation predictive representations, effectively improving detection accuracy in low-sample scenarios.

[0067] Reference Figure 1 and Figure 2 In this embodiment of the application, a zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance includes:

[0068] Step 110: Obtain the visual image to be detected;

[0069] Step 120: Input the visual image into the trained anomaly image detection model to obtain the image detection result;

[0070] In this embodiment of the application, any visual image in the industrial field can be input into a trained abnormal image detection model for anomaly detection to generate an image detection result corresponding to the visual image. The image detection result is used to indicate whether there are defects or other abnormalities in the detected object in the visual image.

[0071] The trained abnormal image detection model is obtained through the following steps:

[0072] Step 130: Obtain the image dataset, as well as the preset text prompt template and the preset image reference set;

[0073] In this embodiment, the image dataset can be a collection of actual visual images acquired by an image acquisition device in the industrial field; the preset image reference set can be a collection of normal state images corresponding to the industrial field, which are used to represent the normal state of objects in the image; the text prompt template can be a learnable text prompt library, which includes normal learnable word embeddings and abnormal learnable word embeddings.

[0074] Step 140: Preprocess the image dataset to obtain an image training set, and extract text features from the text prompt template to obtain text category features;

[0075] In some embodiments, step 140, preprocessing the image dataset to obtain an image training set, includes:

[0076] A1. Perform feature scaling on the image dataset to obtain a scaled image set;

[0077] A2. Interpolate and adjust the image scaling set to obtain the image training set.

[0078] In this embodiment, step A1 may involve normalizing each RGB image in the image dataset to adjust the pixel value of each RGB image to the [0, 1] interval. Then, based on pre-calculated channel mean and standard deviation, the pixel values ​​of each channel of the RGB image are standardized to obtain an image scaling set. This reduces the difference between the red, green, and blue channels of the RGB image, thereby improving the stability and generalization ability of the model training. Step A2 may involve adjusting the image size of each RGB image in the image scaling set based on bicubic interpolation to obtain an image training set. This training set includes several training images, where the shorter sides of all training images have the same length.

[0079] In some embodiments, step 140, extracting text features from the text prompt template to obtain text category features, includes:

[0080] B1. Obtain the text prompt learner;

[0081] B2. Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, which is used to represent feature information that is not related to the semantics of the object;

[0082] B3. Extract text from the text representation information to obtain the text category features.

[0083] In this embodiment, the text prompt learner can be a Prompt learner, which is an 8-layer model, with each layer consisting of 4×768 Parameter objects, and is learned and optimized through backpropagation. Step B2 can involve inputting the text prompt template into the text prompt learner to obtain the text representation information output by the text prompt learner.

[0084] It is understood that the abnormal image detection model in the embodiments of this application can be a multimodal pre-trained model (Contrastive Language-Image Pre-training, CLIP), and step B3 can be to input the text representation information into the text feature extractor in the CLIP model to obtain the text category features.

[0085] It should be noted that the text prompt learner in this embodiment can be trained based on an auxiliary dataset, which includes normal images and abnormal images. Specifically, the total loss function of the text prompt learner can be expressed as:

[0086]

[0087] Among them, L total L is the total loss function; global λ is the cross-entropy loss between visual embeddings of normal or anomalous images from the auxiliary training set and object-independent text embeddings; λ is a hyperparameter; M k This is the kth intermediate layer; For the set of all intermediate layers; This represents the local loss of the k-th intermediate layer.

[0088] Step 150: Perform multi-scale window feature extraction on the image training set to obtain a first image feature set, and perform multi-scale window feature extraction on the image reference set to obtain a second image feature set;

[0089] In some embodiments, multi-scale window feature extraction is performed on the target dataset to obtain a target feature set, including:

[0090] C1. Obtain a preset number of sliding windows, each of which has a different size of its active area;

[0091] C2. Based on all the sliding windows, perform window mapping on each target image in the target dataset to obtain the target feature set, which is used to characterize window feature maps of different scales corresponding to each target image in the target dataset;

[0092] In this embodiment, the target training set can be an image training set or an image reference set. This embodiment takes an image training set as the target training set as an example. Specifically, for a training image in the image training set, step C1 can be to obtain a sliding window group, which includes several sliding windows, each of which is a binary mask with a different size of activation region.

[0093] It is understandable that, for a certain sliding window and training image, the window mapping in step C2 can be to perform an element-set multiplication operation between the sliding window and the training image, thereby retaining only the image information within the sliding window, and using an image encoder based on the CLIP model, which is used to map the local image region to a high-dimensional feature space, thereby obtaining the feature vector (i.e., window feature map) of the training image within the sliding window. The contents of the remaining training images and the remaining sliding windows are similar to the above, and can be easily deduced by analogy, so this application will not elaborate further here.

[0094] Step 160: Based on the text category features, perform multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation;

[0095] In some embodiments, step 160, performing multi-scale semantic alignment on the first image feature set according to the text category features to obtain a semantically guided prediction representation, includes:

[0096] D1. Based on the text category features, perform category alignment on the first image feature set to obtain the category-aligned text features and the category-aligned first image feature set;

[0097] D2. Based on the text features aligned to the categories, perform matrix multiplication on the first image feature set aligned to the categories to obtain anomaly window scores at different scales;

[0098] D3. Perform harmonious average aggregation on the abnormal window scores to obtain the semantically guided prediction representation.

[0099] In this embodiment, step D1 can be based on the KAN model (Kolmogorov-Arnold Networks) to align the text features representing normal and / or abnormal categories with each window feature map in the first image feature set, thereby obtaining text features and the first image feature set that are more suitable for anomaly judgment.

[0100] Understandably, step D2 can involve matrix multiplication of each window feature map and text feature in the first image feature set to obtain window anomaly scores at different scales. Furthermore, for a given pixel location (i,j) in the image, multiple sliding windows may cover that pixel location (i,j). Therefore, the harmonic averaging method in step D3 can be used to aggregate the window anomaly scores covering that pixel location (i,j). Its equivalent expression can be:

[0101]

[0102] in, The sum of the outliers at pixel position (i,j) after aggregation; w uv Let (u,v) be the feature map of the window at pixel position (i,j); It is the window anomaly score of the window feature map (u,v).

[0103] It should be noted that the same applies to other pixel positions covered by multiple sliding windows. After obtaining the window anomaly scores corresponding to the pixel positions covered by multiple sliding windows, these window anomaly scores can be integrated with the window anomaly scores corresponding to only one sliding window to obtain the semantically guided prediction representation.

[0104] Step 170: Based on the second image feature set, perform multi-scale similarity aggregation on the first image feature set to obtain an anomaly segmentation prediction representation;

[0105] In some embodiments, step 170, performing multi-scale similarity aggregation on the first image feature set based on the second image feature set to obtain an anomaly segmentation prediction representation, includes:

[0106] E1. Based on the first image feature set, obtain first window feature maps of different scales, and based on the second image feature set, obtain second window feature maps of different scales.

[0107] E2. Based on the second window feature map, perform similarity calculation on the first window feature map at the corresponding scale to obtain the window feature similarity;

[0108] E3. Integrate the window feature similarity to obtain the anomaly segmentation prediction representation.

[0109] In this embodiment of the application, step E1 can be based on the scale attribute corresponding to the first window feature map, classifying the set of first window feature maps at different scales (i.e., the first image feature set) to obtain the first window feature map under the scale attribute. The content of the second window feature map is similar to the content of the first window feature map, and can be easily deduced by analogy.

[0110] It is understandable that step E2 can be to calculate the cosine similarity between the first window feature map and the second window feature map of the same scale, so as to obtain the window feature similarity corresponding to each scale; step E3 can be to determine the anomaly segmentation prediction representation of each pixel position of the training image based on the window feature similarity at each scale.

[0111] Specifically, for the anomaly segmentation prediction representation at pixel position (i,j), its equivalent formula can be expressed as:

[0112]

[0113] Among them, M ij Let r be the anomaly segmentation prediction representation at pixel position (i,j); r is the feature vector of the normal state image; R is the set of feature vectors of the normal state image; F ij Let be the feature vector of the actual visual image at pixel position (i,j); <> is the cosine similarity function.

[0114] Step 180: Update the parameters of the initialized abnormal image detection model according to the semantic-guided prediction representation and the abnormal segmentation prediction representation to obtain the trained abnormal image detection model.

[0115] In some embodiments, step 180, updating the parameters of the initialized anomaly image detection model based on the semantically guided prediction representation and the anomaly segmentation prediction representation to obtain a trained anomaly image detection model, includes:

[0116] F1. Based on the anomaly segmentation prediction representation, the semantically guided prediction representation is expanded to obtain a few-sample anomaly score map.

[0117] F2. Based on the few-sample anomaly score map, update the parameters of the initialized anomaly image detection model to obtain the trained anomaly image detection model.

[0118] In this embodiment, step F1 may involve fusing the anomaly segmentation prediction representation and the semantically guided prediction representation to obtain a few-sample anomaly score map that combines visual cues from the normal state image with the semantically guided prediction representation; then, based on the obtained few-sample anomaly score map, the initialized anomaly image detection model is trained and updated to obtain a trained anomaly image detection model.

[0119] The following describes in detail, with reference to the accompanying drawings, a multi-scale feature aggregation and semantically guided zero-sample anomaly detection system proposed according to an embodiment of this application.

[0120] Reference Figure 3 The zero-shot anomaly detection system based on multi-scale feature aggregation and semantic guidance proposed in this application includes:

[0121] The acquisition module 101 is used to acquire the visual image to be detected;

[0122] Processing module 102 is used to input the visual image into a trained abnormal image detection model to obtain image detection results;

[0123] The trained abnormal image detection model is obtained through the following steps:

[0124] Acquire the image dataset, as well as the preset text prompt templates and preset image reference sets;

[0125] The image dataset is preprocessed to obtain an image training set, and the text prompt template is subjected to text feature extraction to obtain text category features;

[0126] Multi-scale window feature extraction is performed on the image training set to obtain a first image feature set, and multi-scale window feature extraction is performed on the image reference set to obtain a second image feature set;

[0127] Based on the text category features, multi-scale semantic alignment is performed on the first image feature set to obtain a semantically guided prediction representation;

[0128] Based on the second image feature set, multi-scale similarity aggregation is performed on the first image feature set to obtain an anomaly segmentation prediction representation;

[0129] Based on the semantically guided prediction representation and the anomaly segmentation prediction representation, the parameters of the initialized anomaly image detection model are updated to obtain a trained anomaly image detection model.

[0130] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0131] Reference Figure 4 This application also provides an electronic device, including:

[0132] At least one processor 201;

[0133] At least one memory 202 is used to store at least one program;

[0134] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above-described method embodiments.

[0135] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0137] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0139] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0140] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0142] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0143] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0144] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0145] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0146] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance, characterized in that, include: Acquire the visual image to be detected; The visual image is input into the trained anomaly image detection model to obtain the image detection result; The trained abnormal image detection model is obtained through the following steps: Obtain the image dataset, as well as the preset text prompt templates and preset image reference sets; The image dataset is preprocessed to obtain an image training set, and the text prompt template is subjected to text feature extraction to obtain text category features; Multi-scale window feature extraction is performed on the image training set to obtain a first image feature set, and multi-scale window feature extraction is performed on the image reference set to obtain a second image feature set; Based on the text category features, multi-scale semantic alignment is performed on the first image feature set to obtain a semantically guided prediction representation; Based on the second image feature set, multi-scale similarity aggregation is performed on the first image feature set to obtain an anomaly segmentation prediction representation; Based on the semantic-guided prediction representation and the anomaly segmentation prediction representation, the parameters of the initialized anomaly image detection model are updated to obtain the trained anomaly image detection model. The step of performing multi-scale semantic alignment on the first image feature set based on the text category features to obtain a semantically guided prediction representation includes: Based on the text category features, the first image feature set is aligned to obtain the category-aligned text features and the category-aligned first image feature set; Based on the text features aligned to the categories, matrix multiplication is performed on the first image feature set aligned to the categories to obtain anomaly window scores at different scales; The abnormal window scores are aggregated using a harmonic average to obtain the semantically guided prediction representation; The step of updating the parameters of the initialized anomaly image detection model based on the semantically guided prediction representation and the anomaly segmentation prediction representation to obtain a trained anomaly image detection model includes: Based on the anomaly segmentation prediction representation, the semantically guided prediction representation is expanded to obtain a few-sample anomaly score map. Based on the few-sample anomaly score map, the parameters of the initialized anomaly image detection model are updated to obtain the trained anomaly image detection model.

2. The detection method according to claim 1, characterized in that, The preprocessing of the image dataset to obtain the image training set includes: The image dataset is feature-scaled to obtain a scaled image set; The image scaling set is interpolated and adjusted to obtain the image training set.

3. The detection method according to claim 1, characterized in that, The step of extracting text features from the text prompt template to obtain text category features includes: Get a text prompt learner; Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, which is used to represent feature information that is unrelated to the semantics of the object. Text extraction is performed on the text representation information to obtain the text category features.

4. The detection method according to claim 1, characterized in that, Multi-scale window feature extraction is performed on the target dataset to obtain the target feature set, including: Obtain a preset number of sliding windows, each with a different size of its active area; Based on all the sliding windows, window mapping is performed on each target image in the target dataset to obtain the target feature set, which is used to characterize window feature maps of different scales corresponding to each target image in the target dataset; The target dataset is either an image training set or an image reference set.

5. The detection method according to claim 1, characterized in that, The step of performing multi-scale similarity aggregation on the first image feature set based on the second image feature set to obtain an anomaly segmentation prediction representation includes: Based on the first image feature set, first window feature maps of different scales are obtained, and based on the second image feature set, second window feature maps of different scales are obtained; Based on the second window feature map, the similarity of the first window feature map at the corresponding scale is calculated to obtain the window feature similarity. The anomaly segmentation prediction representation is obtained by integrating the window feature similarity.

6. A zero-shot anomaly detection system based on multi-scale feature aggregation and semantic guidance, characterized in that, include: The acquisition module is used to acquire the visual image to be detected; The processing module is used to input the visual image into the trained abnormal image detection model to obtain the image detection result; The trained abnormal image detection model is obtained through the following steps: Obtain the image dataset, as well as the preset text prompt templates and preset image reference sets; The image dataset is preprocessed to obtain an image training set, and the text prompt template is subjected to text feature extraction to obtain text category features; Multi-scale window feature extraction is performed on the image training set to obtain a first image feature set, and multi-scale window feature extraction is performed on the image reference set to obtain a second image feature set; Based on the text category features, multi-scale semantic alignment is performed on the first image feature set to obtain a semantically guided prediction representation; Based on the second image feature set, multi-scale similarity aggregation is performed on the first image feature set to obtain an anomaly segmentation prediction representation; Based on the semantic-guided prediction representation and the anomaly segmentation prediction representation, the parameters of the initialized anomaly image detection model are updated to obtain the trained anomaly image detection model. The step of performing multi-scale semantic alignment on the first image feature set based on the text category features to obtain a semantically guided prediction representation includes: Based on the text category features, the first image feature set is aligned to obtain the category-aligned text features and the category-aligned first image feature set; Based on the text features aligned to the categories, matrix multiplication is performed on the first image feature set aligned to the categories to obtain anomaly window scores at different scales; The abnormal window scores are aggregated using a harmonic average to obtain the semantically guided prediction representation; The step of updating the parameters of the initialized anomaly image detection model based on the semantically guided prediction representation and the anomaly segmentation prediction representation to obtain a trained anomaly image detection model includes: Based on the anomaly segmentation prediction representation, the semantically guided prediction representation is expanded to obtain a few-sample anomaly score map. Based on the few-sample anomaly score map, the parameters of the initialized anomaly image detection model are updated to obtain the trained anomaly image detection model.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-5.

Images