Zero sample anomaly detection method based on multi-scale feature aggregation and semantic guidance

Abstract

The invention discloses a zero sample anomaly detection method based on multi-scale feature aggregation and semantic guidance, and an abnormal image detection model in the method is obtained through the following steps: obtaining an image training set, an image reference set and text category features; performing multi-scale window feature extraction on the image reference set and the image training set to obtain an image feature set; according to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain semantic guidance prediction representation; performing multi-scale similarity aggregation on the image feature set to obtain an abnormal segmentation prediction representation; and according to the semantic guidance prediction representation and the abnormal segmentation prediction representation, performing parameter updating on the initialized abnormal image detection model to obtain a trained abnormal image detection model. According to the method, the sample data volume required during model training can be effectively reduced, the model training efficiency is improved, meanwhile, the detection precision of the model in a few-sample scene is improved, and the expansibility is high. The invention relates to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance. Background Art

[0002] In the field of automated industrial quality control, anomaly detection technology is crucial to improving manufacturing efficiency and product reliability. This technology is widely used in a variety of industrial fields such as aerospace, automotive, pharmaceuticals and electronics, involving a great diversity of objects and their defects, including changes in multiple dimensions such as color, texture and size.

[0003] At present, traditional anomaly detection technology mainly relies on models customized for specific tasks. Better anomaly detection results can only be achieved by training specific models. The amount of sample data required is large, the training efficiency is low, the accuracy is low in small sample scenarios, and the scalability is not high.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] To this end, one purpose of an embodiment of the present invention is to provide a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance, which can effectively reduce the amount of sample data required for model training and improve the model training efficiency, while improving the detection accuracy of the model in few-sample scenarios and has high scalability.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance, including:

[0009] Acquire a visual image to be detected;

[0010] Inputting the visual image into a trained abnormal image detection model to obtain an image detection result;

[0011] The trained abnormal image detection model is obtained by the following steps:

[0012] Obtain an image dataset, a preset text prompt template, and a preset image reference set;

[0013] Preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features;

[0014] Performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set;

[0015] According to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation;

[0016] According to the second image feature set, performing multi-scale similarity aggregation on the first image feature set to obtain an abnormal segmentation prediction representation;

[0017] According to the semantically guided prediction representation and the abnormal segmentation prediction representation, the parameters of the initialized abnormal image detection model are updated to obtain a trained abnormal image detection model.

[0018] In addition, the zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance according to the above embodiment of the present application may also have the following additional technical features:

[0019] Furthermore, in one embodiment of the present application, the preprocessing of the image data set to obtain an image training set includes:

[0020] Performing feature scaling on the image data set to obtain an image scaling set;

[0021] Interpolation adjustment is performed on the image scaling set to obtain the image training set.

[0022] Furthermore, in one embodiment of the present application, the extracting text features from the text prompt template to obtain text category features includes:

[0023] Get the text prompt learner;

[0024] Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, where the text representation information is used to represent feature information that is unrelated to the semantics of the object;

[0025] Perform text extraction on the text representation information to obtain the text category feature.

[0026] Furthermore, in one embodiment of the present application, multi-scale window feature extraction is performed on the target data set to obtain a target feature set, including:

[0027] Acquire a plurality of preset sliding windows, each of which has an activation area of ​​a different size;

[0028] According to all the sliding windows, window mapping is performed on each target image in the target data set to obtain the target feature set, where the target feature set is used to characterize window feature maps of different scales corresponding to each target image in the target data set;

[0029] The target data set is an image training set or an image reference set.

[0030] Further, in one embodiment of the present application, performing multi-scale semantic alignment on the first image feature set according to the text category feature to obtain a semantically guided prediction representation includes:

[0031] According to the text category features, the first image feature set is category-aligned to obtain the category-aligned text features and the category-aligned first image feature set;

[0032] According to the text features after the category alignment, matrix multiplication is performed on the first image feature set after the category alignment to obtain abnormal window scores of different scales;

[0033] The abnormal window scores are harmoniously averaged and aggregated to obtain the semantically guided prediction representation.

[0034] Further, in one embodiment of the present application, performing multi-scale similarity aggregation on the first image feature set according to the second image feature set to obtain an abnormal segmentation prediction representation includes:

[0035] Obtaining first window feature maps of different scales according to the first image feature set, and obtaining second window feature maps of different scales according to the second image feature set;

[0036] According to the second window feature map, similarity calculation is performed on the first window feature map of the corresponding scale to obtain window feature similarity;

[0037] The window feature similarities are integrated to obtain the abnormal segmentation prediction representation.

[0038] Further, in one embodiment of the present application, the parameter updating of the initialized abnormal image detection model according to the semantic guidance prediction representation and the abnormal segmentation prediction representation to obtain a trained abnormal image detection model includes:

[0039] According to the abnormal segmentation prediction representation, performing sample expansion on the semantic guided prediction representation to obtain a few-sample abnormal score map;

[0040] According to the few-sample anomaly score map, the parameters of the initialized abnormal image detection model are updated to obtain the trained abnormal image detection model.

[0041] In a second aspect, the embodiment of the present application provides a zero-sample anomaly detection system with multi-scale feature aggregation and semantic guidance, including:

[0042] An acquisition module, used for acquiring a visual image to be detected;

[0043] A processing module, used for inputting the visual image into a trained abnormal image detection model to obtain an image detection result;

[0044] The trained abnormal image detection model is obtained by the following steps:

[0045] Obtain an image dataset, a preset text prompt template, and a preset image reference set;

[0046] Preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features;

[0047] Performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set;

[0048] According to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation;

[0049] According to the second image feature set, performing multi-scale similarity aggregation on the first image feature set to obtain an abnormal segmentation prediction representation;

[0050] According to the semantically guided prediction representation and the abnormal segmentation prediction representation, the parameters of the initialized abnormal image detection model are updated to obtain a trained abnormal image detection model.

[0051] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0052] at least one processor;

[0053] at least one memory for storing at least one program;

[0054] When the at least one program is executed by the at least one processor, the at least one processor implements the method of the first aspect described above.

[0055] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the method of the first aspect when executed by the processor.

[0056] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0057] The embodiment of the present application discloses a zero-sample anomaly detection method with multi-scale feature aggregation and semantic guidance, wherein the detection method obtains a visual image to be detected; inputs the visual image into a trained abnormal image detection model to obtain an image detection result; wherein the trained abnormal image detection model is obtained by the following steps: obtaining an image data set, a preset text prompt template and a preset image reference set; preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features; performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set; performing multi-scale semantic alignment on the first image feature set according to the text category features to obtain a semantically guided prediction representation; performing multi-scale similarity aggregation on the first image feature set according to the second image feature set to obtain an abnormal segmentation prediction representation; and updating parameters of the initialized abnormal image detection model according to the semantically guided prediction representation and the abnormal segmentation prediction representation to obtain a trained abnormal image detection model. This detection method is based on multi-scale semantic alignment, extracting semantically guided prediction representations that closely align image visual features with language descriptions, thereby achieving anomaly segmentation detection without label annotation and under zero-sample conditions, which can effectively reduce the amount of sample data required for model training and improve training efficiency and scalability. In addition, this detection method is also based on anomaly segmentation prediction representation and anomaly segmentation prediction representation, effectively integrating the intuitive visual information of normal reference images and the prediction advantages driven by language, effectively improving the detection accuracy in few-sample scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 A flowchart of a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a training process of an abnormal image detection model provided in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the structure of a zero-sample anomaly detection system with multi-scale feature aggregation and semantic guidance provided in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] At present, traditional anomaly detection technology mainly relies on models customized for specific tasks. Better anomaly detection results can only be achieved by training specific models. The amount of sample data required is large, the training efficiency is low, the accuracy is low in small sample scenarios, and the scalability is not high.

[0066] In view of this, the embodiment of the present invention provides a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance. The detection method is based on multi-scale semantic alignment, and extracts semantic-guided prediction representations that closely align image visual features with language descriptions, thereby realizing anomaly segmentation detection under label-free and zero-sample conditions, which can effectively reduce the amount of sample data required for model training and improve training efficiency and scalability. In addition, the detection method is also based on anomaly segmentation prediction representation and anomaly segmentation prediction representation, effectively integrating the intuitive visual information of normal reference images and the prediction advantages driven by language, effectively improving the detection accuracy in small sample scenarios.

[0067] Reference Figure 1 and Figure 2 In an embodiment of the present application, a zero-sample anomaly detection method based on multi-scale feature aggregation and semantic guidance includes:

[0068] Step 110: obtaining a visual image to be detected;

[0069] Step 120: input the visual image into a trained abnormal image detection model to obtain an image detection result;

[0070] In an embodiment of the present application, any visual image in the industrial field can be input into a trained abnormal image detection model for abnormality detection to generate an image detection result corresponding to the visual image. The image detection result is used to indicate whether the detection object in the visual image has defects or other abnormal conditions.

[0071] The trained abnormal image detection model is obtained by the following steps:

[0072] Step 130: acquiring an image data set, a preset text prompt template, and a preset image reference set;

[0073] In an embodiment of the present application, the image data set may be a collection of actual visual images acquired in an industrial field based on image acquisition equipment; the preset image reference set may be a collection of normal state images corresponding to the industrial field, and the normal state images are used to characterize the normal conditions of objects in the image; the text prompt template may be a learnable text prompt library, which includes normal learnable word embeddings and abnormal learnable word embeddings.

[0074] Step 140: preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features;

[0075] In some embodiments, the step 140 of preprocessing the image data set to obtain an image training set includes:

[0076] A1. Perform feature scaling on the image data set to obtain an image scaling set;

[0077] A2. Perform interpolation adjustment on the image scaling set to obtain the image training set.

[0078] In an embodiment of the present application, step A1 may be to normalize each RGB image in the image data set, so as to adjust each pixel value in the RGB image to the interval [0, 1]; then, based on the pre-calculated channel mean and standard deviation, the pixel value of each channel of the RGB image is standardized to obtain an image scaling set, which can reduce the degree of difference between the red, green and blue channels of the RGB image, thereby improving the stability and generalization ability of the model training. Step A2 may be based on bicubic interpolation, to adjust the image size of each RGB image in the image scaling set to obtain an image training set, which includes a plurality of training images, and the lengths between the shorter sides in all training images are the same.

[0079] In some embodiments, the step 140 of extracting text features from the text prompt template to obtain text category features includes:

[0080] B1. Get the text prompt learner;

[0081] B2. Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, where the text representation information is used to represent feature information that is unrelated to the semantics of the object;

[0082] B3. Perform text extraction on the text representation information to obtain the text category feature.

[0083] In the embodiment of the present application, the text prompt learner may be a Prompt learner, which is an 8-layer structure model, each layer consisting of a Parameter object of size 4×768, and is learned and optimized by a back propagation algorithm. Step B2 may be inputting the text prompt template into the text prompt learner to obtain the text representation information output by the text prompt learner.

[0084] It can be understood that the abnormal image detection model in the embodiment of the present application can be a multimodal pre-training model (Contrastive Language-Image Pre-training, CLIP), and step B3 can be to input the text representation information into the text feature extractor in the CLIP model to obtain text category features.

[0085] It should be noted that the text prompt learner in the embodiment of the present application can be trained based on an auxiliary data set, and the auxiliary training set includes normal images and abnormal images. Specifically, the total loss function of the text prompt learner can be expressed as:

[0086]

[0087] Among them, L total is the total loss function; L global is the cross entropy loss between the visual embedding of normal or abnormal images from the auxiliary training set and the object-independent text embedding; λ is a hyperparameter; M k is the kth intermediate layer; is the set of all middle layers; is the local loss of the kth intermediate layer.

[0088] Step 150: performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set;

[0089] In some embodiments, multi-scale window feature extraction is performed on the target data set to obtain a target feature set, including:

[0090] C1. Acquire a plurality of preset sliding windows, wherein the size of the activation area of ​​each sliding window is different;

[0091] C2. Performing window mapping on each target image in the target data set according to all the sliding windows to obtain the target feature set, where the target feature set is used to characterize window feature maps of different scales corresponding to each target image in the target data set;

[0092] In the embodiment of the present application, the target training set may be an image training set or an image reference set. The embodiment of the present application takes the target training set as an image training set as an example. Specifically, for a certain training image in the image training set, step C1 may be to obtain a sliding window group, the sliding window group includes a plurality of sliding windows, each sliding window is a binary mask, and the size of the activation area of ​​the binary mask is different.

[0093] It can be understood that, for a certain sliding window and training image, the window mapping in step C2 can be an element set multiplication operation between the sliding window and the training image, so as to retain only the image information within the sliding window, and an image encoder based on the CLIP model, which is used to map the local image area to a high-dimensional feature space, so as to obtain a feature vector (i.e., a window feature map) about the training image in the sliding window. The contents of the remaining training images and the remaining sliding windows are similar to the aforementioned contents and can be simply deduced by analogy, and the present application will not go into details here.

[0094] Step 160: performing multi-scale semantic alignment on the first image feature set according to the text category features to obtain a semantically guided prediction representation;

[0095] In some embodiments, the step 160 of performing multi-scale semantic alignment on the first image feature set according to the text category feature to obtain a semantically guided prediction representation includes:

[0096] D1. According to the text category features, the first image feature set is category-aligned to obtain the category-aligned text features and the category-aligned first image feature set;

[0097] D2. According to the text features after the category alignment, matrix multiplication is performed on the first image feature set after the category alignment to obtain abnormal window scores of different scales;

[0098] D3. Perform harmonious average aggregation on the abnormal window scores to obtain the semantically guided prediction representation.

[0099] In an embodiment of the present application, step D1 can be based on the KAN model (Kolmogorov-Arnold Networks) to align the text features representing normal categories and / or abnormal categories with each window feature map in the first image feature set, so as to obtain text features and the first image feature set that are more suitable for abnormality judgment.

[0100] It can be understood that step D2 can be to perform matrix multiplication on each window feature map and text feature in the first image feature set to obtain window anomaly scores at different scales. In addition, for a certain pixel position (i, j) in the image, there may be multiple sliding windows covering the pixel position (i, j), so the harmonious average method in step D3 can be used to aggregate the window anomaly scores covering the pixel position (i, j), and its equivalent expression can be:

[0101]

[0102] in, is the window anomaly score at the aggregated pixel position (i, j); w uv is the window feature map (u,v) at pixel position (i,j); is the window anomaly score of the window feature map (u,v).

[0103] It should be noted that the same is true for the other pixel positions covered by multiple sliding windows. After obtaining the window anomaly scores corresponding to the pixel positions covered by multiple sliding windows, the window anomaly scores can be integrated with the window anomaly scores corresponding to only one sliding window to obtain the semantically guided prediction representation.

[0104] Step 170: performing multi-scale similarity aggregation on the first image feature set according to the second image feature set to obtain an abnormal segmentation prediction representation;

[0105] In some embodiments, the step 170 of performing multi-scale similarity aggregation on the first image feature set according to the second image feature set to obtain an abnormal segmentation prediction representation includes:

[0106] E1. Obtaining first window feature maps of different scales according to the first image feature set, and obtaining second window feature maps of different scales according to the second image feature set;

[0107] E2. Calculate the similarity of the first window feature map of the corresponding scale according to the second window feature map to obtain window feature similarity;

[0108] E3. Integrate the window feature similarities to obtain the abnormal segmentation prediction representation.

[0109] In an embodiment of the present application, step E1 can be based on the scale attribute corresponding to the first window feature map, and classify the set of first window feature maps of different scales (i.e., the first image feature set) to obtain the first window feature map under the attribute of scale. The content of the second window feature map is similar to that of the aforementioned first window feature map, and can be simply deduced by analogy.

[0110] It can be understood that step E2 can be to calculate the cosine similarity between the first window feature map and the second window feature map of the same scale to obtain the window feature similarity corresponding to each scale; step E3 can be based on the window feature similarity at each scale to determine the abnormal segmentation prediction representation of each pixel position of the training image.

[0111] Specifically, for the abnormal segmentation prediction representation at the pixel position (i, j), its equivalent formula can be expressed as:

[0112]

[0113] Among them, M ij is the abnormal segmentation prediction representation at the pixel position (i, j); r is the feature vector of the normal state image; R is the set of feature vectors of the normal state image; F ij is the feature vector of the actual visual image at the pixel position (i, j); <> is the cosine similarity function.

[0114] Step 180: Update the parameters of the initialized abnormal image detection model according to the semantic guidance prediction representation and the abnormal segmentation prediction representation to obtain a trained abnormal image detection model.

[0115] In some embodiments, the step 180 of updating the parameters of the initialized abnormal image detection model according to the semantic guidance prediction representation and the abnormal segmentation prediction representation to obtain a trained abnormal image detection model includes:

[0116] F1. According to the abnormal segmentation prediction representation, the semantic guided prediction representation is sample expanded to obtain a few-sample abnormal score map;

[0117] F2. According to the few-sample anomaly score map, update the parameters of the initialized abnormal image detection model to obtain the trained abnormal image detection model.

[0118] In an embodiment of the present application, step F1 can be to associate and fuse the abnormal segmentation prediction representation and the semantically guided prediction representation to obtain a few-sample abnormality score map that combines the visual clues of the normal state image and the semantically guided prediction representation; then, based on the obtained few-sample abnormality score map, the initialized abnormal image detection model is trained and updated to obtain a trained abnormal image detection model.

[0119] A zero-sample anomaly detection system with multi-scale feature aggregation and semantic guidance proposed according to an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0120] Reference Figure 3 , a zero-sample anomaly detection system based on multi-scale feature aggregation and semantic guidance proposed in an embodiment of the present application includes:

[0121] An acquisition module 101 is used to acquire a visual image to be detected;

[0122] A processing module 102 is used to input the visual image into a trained abnormal image detection model to obtain an image detection result;

[0123] The trained abnormal image detection model is obtained by the following steps:

[0124] Obtain an image dataset, a preset text prompt template, and a preset image reference set;

[0125] Preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features;

[0126] Performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set;

[0127] According to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation;

[0128] According to the second image feature set, performing multi-scale similarity aggregation on the first image feature set to obtain an abnormal segmentation prediction representation;

[0129] According to the semantically guided prediction representation and the abnormal segmentation prediction representation, the parameters of the initialized abnormal image detection model are updated to obtain a trained abnormal image detection model.

[0130] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] Reference Figure 4 , the embodiment of the present application further provides an electronic device, including:

[0132] at least one processor 201;

[0133] At least one memory 202, used to store at least one program;

[0134] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0135] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0137] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0138] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0139] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0140] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0141] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0142] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0143] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0144] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0145] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0146] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A zero-shot anomaly detection method based on multi-scale feature aggregation and semantic guidance, characterized in that: include: Acquire a visual image to be detected; Inputting the visual image into a trained abnormal image detection model to obtain an image detection result; The trained abnormal image detection model is obtained by the following steps: Obtain an image dataset, a preset text prompt template, and a preset image reference set; Preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features; Performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set; According to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation; According to the second image feature set, performing multi-scale similarity aggregation on the first image feature set to obtain an abnormal segmentation prediction representation; According to the semantically guided prediction representation and the abnormal segmentation prediction representation, the parameters of the initialized abnormal image detection model are updated to obtain a trained abnormal image detection model.

2. The detection method according to claim 1, characterized in that: The preprocessing of the image data set to obtain an image training set includes: Performing feature scaling on the image data set to obtain an image scaling set; Interpolation adjustment is performed on the image scaling set to obtain the image training set.

3. The detection method according to claim 1, characterized in that: The extracting text features from the text prompt template to obtain text category features includes: Get the text prompt learner; Based on the text prompt learner, text learning is performed on the text prompt template to obtain text representation information, where the text representation information is used to represent feature information that is unrelated to the semantics of the object; Perform text extraction on the text representation information to obtain the text category feature.

4. The detection method according to claim 1, characterized in that: Perform multi-scale window feature extraction on the target data set to obtain the target feature set, including: Acquire a plurality of preset sliding windows, each of which has an activation area of ​​a different size; According to all the sliding windows, window mapping is performed on each target image in the target data set to obtain the target feature set, where the target feature set is used to characterize window feature maps of different scales corresponding to each target image in the target data set; The target data set is an image training set or an image reference set.

5. The detection method according to claim 1, characterized in that: The step of performing multi-scale semantic alignment on the first image feature set according to the text category feature to obtain a semantically guided prediction representation includes: According to the text category features, the first image feature set is category-aligned to obtain the category-aligned text features and the category-aligned first image feature set; According to the text features after the category alignment, matrix multiplication is performed on the first image feature set after the category alignment to obtain abnormal window scores of different scales; The abnormal window scores are harmoniously averaged and aggregated to obtain the semantically guided prediction representation.

6. The detection method according to claim 1, characterized in that: The step of performing multi-scale similarity aggregation on the first image feature set according to the second image feature set to obtain an abnormal segmentation prediction representation includes: Obtaining first window feature maps of different scales according to the first image feature set, and obtaining second window feature maps of different scales according to the second image feature set; According to the second window feature map, similarity calculation is performed on the first window feature map of the corresponding scale to obtain window feature similarity; The window feature similarities are integrated to obtain the abnormal segmentation prediction representation.

7. The detection method according to claim 1, characterized in that: The step of updating the parameters of the initialized abnormal image detection model according to the semantic guidance prediction representation and the abnormal segmentation prediction representation to obtain a trained abnormal image detection model includes: According to the abnormal segmentation prediction representation, performing sample expansion on the semantic guided prediction representation to obtain a few-sample abnormal score map; According to the few-sample anomaly score map, the parameters of the initialized abnormal image detection model are updated to obtain the trained abnormal image detection model.

8. A zero-shot anomaly detection system based on multi-scale feature aggregation and semantic guidance, characterized in that: include: An acquisition module, used for acquiring a visual image to be detected; A processing module, used for inputting the visual image into a trained abnormal image detection model to obtain an image detection result; The trained abnormal image detection model is obtained by the following steps: Obtain an image dataset, a preset text prompt template, and a preset image reference set; Preprocessing the image data set to obtain an image training set, and extracting text features from the text prompt template to obtain text category features; Performing multi-scale window feature extraction on the image training set to obtain a first image feature set, and performing multi-scale window feature extraction on the image reference set to obtain a second image feature set; According to the text category features, performing multi-scale semantic alignment on the first image feature set to obtain a semantically guided prediction representation; According to the second image feature set, performing multi-scale similarity aggregation on the first image feature set to obtain an abnormal segmentation prediction representation; According to the semantically guided prediction representation and the abnormal segmentation prediction representation, the parameters of the initialized abnormal image detection model are updated to obtain a trained abnormal image detection model.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images