A zero-shot anomaly detection method and apparatus based on multimodal cue learning

Abstract

This application discloses a zero-shot anomaly detection method and apparatus based on multimodal cue learning. The method includes: a frequency domain dynamic cue module modeling an image to obtain spectral features; fusing the spectral features with a multimodal cue learning template to obtain text cue; inputting the text cue into an encoder to obtain initial text features; a cross-attention guided anomaly map generation module updating the initial text features to obtain target text features; calculating an anomaly map by combining the target text features with staged image features; a dual-branch pooling attention module pooling the staged image features to obtain local image features; adding the local image features with category representation features to obtain global image features; and calculating an anomaly score by combining the global image features with the initial text features. This application can reduce the model's dependence on fixed cue, enhance the ability to distinguish different anomaly patterns, and improve the accuracy of anomaly localization, and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a zero-shot anomaly detection method and apparatus based on multimodal cue learning. Background Technology

[0002] In related technologies, there are image sample anomaly detection methods designed to automatically identify abnormal regions in images that deviate from normal patterns, such as scratches, cracks, and other defects on the surface of industrial products, or lesions in medical images. However, current image sample anomaly detection methods generally rely on manually designed text prompts. Different categories of objects have significant differences in visual features such as appearance and texture details. Fixed text features cannot dynamically adapt to the feature distribution of different categories, resulting in a decrease in the model's discriminative ability when detecting across categories.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The embodiments of this application aim to at least partially address one of the technical problems in the related art. Therefore, the main objective of the embodiments of this application is to propose a zero-shot anomaly detection method and apparatus based on multimodal cue learning, which can reduce the model's dependence on specific product cuees, enhance the ability to distinguish different anomaly patterns, and improve the accuracy and robustness of anomaly localization.

[0005] To achieve the above objectives, one aspect of this application proposes a zero-shot anomaly detection method based on multimodal cue learning, the method comprising the following steps: The image to be detected is input into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; The frequency domain dynamic prompting module models the global frequency domain features of the image to be detected to obtain target spectral features, and then fuses the target spectral features with the multimodal prompting learning template to obtain target text prompts. The target text prompt is input into the text encoder to obtain the initial text features; The cross-attention guided anomaly graph generation module dynamically updates the initial text features to obtain target text features, and calculates the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly graph. The staged image features are processed by bi-branch pooling attention module to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features. The cosine similarity between the global image features and the initial text features is calculated to obtain an image-level anomaly score.

[0006] In some embodiments, the step of modeling the global frequency domain features of the image to be detected through the frequency domain dynamic prompting module to obtain target spectral features includes: The frequency domain dynamic prompting module performs Fourier transform processing on the image to be detected to obtain an initial frequency domain representation. The frequency domain dynamic prompting module performs feature decomposition on the initial frequency domain representation to obtain initial spectral features; wherein, the initial spectral features include amplitude spectrum and phase spectrum; The frequency domain dynamic prompting module uses a convolution module to spatially model the initial spectral features to obtain the target spectral features; wherein, the target spectral features include amplitude spectrum features and phase spectrum features.

[0007] In some embodiments, the multimodal cue learning template includes a learnable embedding vector, and the step of fusing the target spectral features with the multimodal cue learning template to obtain the target text cue includes: The frequency domain dynamic prompting module adds the target spectral features to the learnable embedding vector to obtain a fused category representation. The frequency domain dynamic prompting module replaces the original content in the learnable embedding vector with the fused category representation to obtain the target text prompt.

[0008] In some embodiments, the step of dynamically updating the initial text features through the cross-attention guided anomaly graph generation module to obtain target text features includes: The initial text features are converted into text query vectors through the convolutional layers in the cross-attention guided anomaly graph generation module. The convolutional layer in the cross-attention guided anomaly graph generation module converts several image features contained in the staged image features into corresponding keys and values. The cross-attention-guided anomaly graph generation module, based on the cross-attention mechanism, dynamically updates the initial text features according to the text query vector and the keys and values ​​corresponding to several image features contained in the staged image features, to obtain the target text features.

[0009] In some embodiments, the step of calculating the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map includes: The cross-attention guided anomaly graph generation module performs cosine similarity calculation on the target text features and several image features contained in the staged image features to obtain initial image anomaly graphs corresponding to several image features. The cross-attention guided anomaly map generation module calculates the average of several initial image anomaly maps to obtain the target image anomaly map.

[0010] In some embodiments, the step of performing bi-branch pooling processing on the staged image features through the bi-branch pooling attention module to obtain local image features includes: The dual-branch pooling attention module uses a linear layer to map several image features contained in the staged image features to the joint feature space, and then performs channel-dimensional concatenation in the joint feature space to obtain multi-layer fusion features. The multi-layer fusion features are subjected to global average pooling through the dual-branch pooling attention module to obtain global average pooling features of the image. The multi-layer fusion features are subjected to global max pooling through the dual-branch pooling attention module to obtain global max pooling features of the image. The dual-branch pooling attention module inputs the global average pooling features and the global max pooling features of the image into the multilayer perceptron to generate target channel weights. The dual-branch pooling attention module multiplies the target channel weights and the multi-layer fusion features element-wise, and adds the result of the element-wise multiplication to the multi-layer fusion features to obtain weighted image features. The local image features are obtained by performing global average pooling on the weighted image features through the dual-branch pooling attention module.

[0011] To achieve the above objectives, another aspect of this application proposes a zero-shot anomaly detection device based on multimodal cue learning, the device comprising the following modules: The image input module is used to input the image to be detected into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic prompting module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; The frequency domain dynamic prompting construction module is used to model the global frequency domain features of the image to be detected through the frequency domain dynamic prompting module to obtain the target spectral features, and to fuse the target spectral features with the multimodal prompting learning template to obtain the target text prompt; A text prompt encoding module is used to input the target text prompt into the text encoder to obtain initial text features; The image anomaly generation module is used to dynamically update the initial text features by guiding the anomaly generation module through the cross attention to obtain target text features, and to calculate the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly. A dual-branch pooling processing module is used to perform dual-branch pooling processing on the staged image features through the dual-branch pooling attention module to obtain local image features; The image-level anomaly score calculation module is used to add the local image features to the category representation features generated by the image encoder to obtain global image features, and to calculate the cosine similarity between the global image features and the initial text features to obtain the image-level anomaly score.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0015] The embodiments of this application include at least the following beneficial effects: This application provides a zero-shot anomaly detection method and apparatus based on multimodal cue learning. This scheme inputs the image to be detected into a trained anomaly detection network. The anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a two-branch pooling attention module. The frequency domain dynamic cue module models the global frequency domain features of the image to be detected to obtain target spectral features, and fuses the target spectral features with a multimodal cue learning template to obtain target text cue. The target text cue is input into the text encoder to obtain initial text features. The cross-attention guided anomaly map generation module dynamically updates the initial text features to obtain target text features, and performs cosine similarity calculation between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The two-branch pooling attention module performs two-branch pooling processing on the staged image features to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features, and the cosine similarity calculation between the global image features and the initial text features is performed to obtain an image-level anomaly score. This application embodiment integrates deep frequency domain features of images into text prompts through a frequency domain dynamic prompt module, reducing the model's reliance on specific product prompts. A cross-attention-guided anomaly map generation module dynamically updates text features using fine-grained image features, achieving more accurate image-text alignment and generating accurate anomaly maps. A dual-branch pooling attention module fuses multi-layered contextual information to obtain local image features, thereby calculating image-level anomaly scores and enhancing the ability to distinguish different anomaly patterns. In summary, this application embodiment leverages the strong generalization ability of visual language models, introducing a multimodal prompt learning mechanism combining a frequency domain dynamic prompt module, a cross-attention-guided anomaly map generation module, and a dual-branch pooling attention module. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the steps of a zero-shot anomaly detection method based on multimodal cue learning provided in an embodiment of this application. Figure 2 This is a schematic diagram of the network structure of a zero-shot anomaly detection method based on multimodal cue learning provided in an embodiment of this application; Figure 3 This is a flowchart illustrating a zero-shot anomaly detection method based on multimodal cue learning provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of a zero-shot anomaly detection device based on multimodal cue learning provided in an embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0018] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0019] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] A typical approach to this technique is as follows: First, textual prompts are generated using a fixed prompt template. These prompts, along with the image to be detected, are input into the text encoder and image encoder of a visual language model (CLIP, Contrastive Language-Image Pre-Training), respectively, to obtain text embeddings and multi-level image features. Then, the cosine similarity between the multi-level image features and the text embeddings is calculated to generate an anomaly map. The multiple anomaly maps are then averaged to obtain the final predicted anomaly map. Finally, the cosine similarity between the text embeddings and the category embedding features obtained from the image encoder is calculated to obtain the final anomaly score. The main drawback of this approach is that all detection categories use a fixed prompt template. However, different categories of objects differ significantly in visual features such as appearance, texture details, etc. Fixed text features cannot dynamically adapt to the feature distribution of different categories, leading to a decrease in the model's discriminative ability during cross-category detection. Furthermore, this method relies heavily on global similarity to calculate the anomaly score, limiting the accuracy of anomaly detection. Moreover, this approach not only requires high domain knowledge and incurs high design costs, but also struggles to accurately characterize the complex structural features of images.

[0022] In view of this, this application provides a zero-shot anomaly detection method and apparatus based on multimodal cue learning. This scheme involves inputting the image to be detected into a trained anomaly detection network. The anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a two-branch pooling attention module. The frequency domain dynamic cue module models the global frequency domain features of the image to be detected to obtain target spectral features, and fuses these target spectral features with a multimodal cue learning template to obtain target text cue. The target text cue is input into the text encoder to obtain initial text features. The cross-attention guided anomaly map generation module dynamically updates the initial text features to obtain target text features, and performs cosine similarity calculation between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The two-branch pooling attention module performs two-branch pooling processing on the staged image features to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features, and the cosine similarity between the global image features and the initial text features is calculated to obtain an image-level anomaly score. This application embodiment integrates deep frequency domain features of images into text prompts through a frequency domain dynamic prompt module, reducing the model's reliance on specific product prompts. A cross-attention-guided anomaly map generation module dynamically updates text features using fine-grained image features, achieving more accurate image-text alignment and generating accurate anomaly maps. A dual-branch pooling attention module fuses multi-layered contextual information to obtain local image features, thereby calculating image-level anomaly scores and enhancing the ability to distinguish different anomaly patterns. In summary, this application embodiment leverages the strong generalization ability of visual language models, introducing a multimodal prompt learning mechanism combining a frequency domain dynamic prompt module, a cross-attention-guided anomaly map generation module, and a dual-branch pooling attention module. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization.

[0023] The zero-shot anomaly detection method based on multimodal cue learning provided in this application relates to the field of computer technology. This method can be applied to terminals, servers, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the zero-shot anomaly detection method based on multimodal cue learning, but is not limited to the above forms.

[0024] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0025] Please see Figure 1 , Figure 1 This is an optional flowchart of the zero-shot anomaly detection method based on multimodal cue learning provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0026] Step S101: Input the image to be detected into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic prompting module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; Please see Figure 2 , Figure 2This is a schematic diagram of the network structure of a zero-shot anomaly detection method based on multimodal cue learning provided in an embodiment of this application, as shown below. Figure 2 As shown, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly graph generation module, and a dual-branch pooling attention module. In this embodiment, the image encoder and text encoder are the visual encoder and text encoder of the CLIP (Contrastive Language–Image Pre-training) model.

[0027] In this embodiment, to enhance the model's ability to perceive specific semantic information of images, a frequency domain dynamic prompting module is introduced to generate prompts using frequency domain dynamic features. Since the introduction of frequency domain features into the text prompts complicates the alignment of text embeddings with fine-grained image features, this embodiment proposes a cross-attention-based guidance mechanism to dynamically update text features and generate anomaly maps. This enables precise localization of anomalous regions, effectively improving the accuracy and robustness of anomaly localization. To further aggregate fine-grained image features and enhance anomaly detection capabilities, this embodiment proposes a dual-branch pooling attention module. The core objective of this module is to combine the fused local image features with the category representation output by the image encoder to calculate an image-level anomaly score.

[0028] Step S102: The global frequency domain features of the image to be detected are modeled by the frequency domain dynamic prompting module to obtain the target spectral features, and the target spectral features are fused with the multimodal prompting learning template to obtain the target text prompt; The modal cue learning template includes learnable embedding vectors. Regarding text cue construction, the modal cue learning template used in this embodiment is in the form of "a photo of a [state] [".] ][ ]…[ ]”, where “a photo of a” is fixed text, and [state] represents words describing the state of the target (such as good or damaged). ][ ]…[ ] is a set of learnable embedding vectors used to encode category-specific semantic information and is initialized with a standard normal distribution. In this embodiment, i is 2.

[0029] In some embodiments, the step of modeling the global frequency domain features of the image to be detected using the frequency domain dynamic prompting module to obtain the target spectral features may include: performing a Fourier transform on the image to be detected using the frequency domain dynamic prompting module to obtain an initial frequency domain representation; performing feature decomposition on the initial frequency domain representation using the frequency domain dynamic prompting module to obtain initial spectral features; wherein the initial spectral features include amplitude spectrum and phase spectrum; and using a convolution module to spatially model the initial spectral features using the frequency domain dynamic prompting module to obtain the target spectral features; wherein the target spectral features include amplitude spectrum features and phase spectrum features.

[0030] In some embodiments, the step of fusing the target spectral features with a multimodal cue learning template to obtain a target text cue may include: adding the target spectral features to a learnable embedding vector through a frequency domain dynamic cue module to obtain a fused category representation; and replacing the original content in the learnable embedding vector with the fused category representation through the frequency domain dynamic cue module to obtain the target text cue.

[0031] In the specific implementation, firstly, the image to be detected, I, is input into the anomaly detection network. Then, through the frequency domain dynamic prompting module, the image to be detected, I, undergoes Fourier transform processing to obtain the initial frequency domain representation. Next, the initial frequency domain representation is processed through the frequency domain dynamic prompting module. Feature decomposition is performed to obtain initial spectral features (including amplitude spectrum A and phase spectrum P). Then, through the frequency domain dynamic cueing module, a convolution module is used to spatially model the initial spectral features to obtain the target spectral features (including amplitude spectrum features). and phase spectrum characteristics Furthermore, through the frequency domain dynamic prompting module, the target spectral characteristics (i.e., amplitude spectrum characteristics) are displayed. and phase spectrum characteristics ) and learnable embedding vectors ( and Adding them together yields the fusion category representation (i.e. and ),in: , The frequency domain dynamic prompting module represents the fusion category ( and Replace the original content in the learnable embedding vector. and The target text prompt is "a photo of a [state][ ] [ ]".

[0032] Step S103: Input the target text prompt into the text encoder to obtain initial text features; In the specific implementation, the target text prompt will be "a photo of a [state][ ] [ The initial text features are obtained by inputting them into the text encoder of the CLIP model. .

[0033] Step S104: The initial text features are dynamically updated through the cross-attention guided anomaly map generation module to obtain target text features, and the cosine similarity between the target text features and the staged image features generated by the image encoder is calculated to generate a target image anomaly map. In some embodiments, the step of dynamically updating the initial text features through the cross-attention-guided anomaly graph generation module to obtain the target text features may include: converting the initial text features into a text query vector through the convolutional layer in the cross-attention-guided anomaly graph generation module; converting several image features contained in the staged image features into corresponding keys and values ​​through the convolutional layer in the cross-attention-guided anomaly graph generation module; and dynamically updating the initial text features based on the cross-attention mechanism, according to the text query vector and the keys and values ​​corresponding to the several image features contained in the staged image features, to obtain the target text features.

[0034] In some embodiments, the step of calculating the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map may include: using a cross-attention-guided anomaly map generation module to calculate the cosine similarity between the target text features and several image features contained in the staged image features to obtain initial image anomaly maps corresponding to several image features; and using a cross-attention-guided anomaly map generation module to average several initial image anomaly maps to obtain a target image anomaly map.

[0035] like Figure 2 As shown, the anomaly detection network includes an image encoder. The image encoder used in this embodiment is the visual encoder of the pre-trained visual language model CLIP. This image encoder consists of a 24-layer Transformer structure. In this embodiment, the feature outputs of layers 6, 12, 18, and 24 are extracted and labeled as the first stage, second stage, third stage, and fourth stage, respectively. The image feature outputs obtained in the first, second, third, and fourth stages are respectively denoted as... , , , Furthermore, this image encoder performs global processing on the input image and outputs a category representation feature. .

[0036] In the specific implementation, firstly, the convolutional layers in the anomaly graph generation module are guided by cross-attention to process the initial text features. Convert to text query vector The convolutional layers in the anomaly map generation module, guided by cross-attention, will generate phased image features. Several image features contained in (i=1, 2, 3, 4) , , , Convert them to their corresponding keys respectively. Sum The anomaly graph generation module, guided by cross-attention, is based on a cross-attention mechanism and uses text query vectors. and staged image features The keys corresponding to several image features contained Sum For initial text features Dynamic updates are performed to obtain the target text features. The anomaly graph generation module is guided by cross-attention to analyze the target text features. Stage-based image features The cosine similarity of several image features is calculated to obtain the initial image anomaly map corresponding to each image feature. The anomaly map generation module is guided by cross-attention to generate anomaly maps from several initial images. An average calculation is performed to obtain the anomaly map of the target image.

[0037] Step S105: Perform bi-branch pooling processing on the staged image features through the bi-branch pooling attention module to obtain local image features; In some embodiments, step S105 may include: using a dual-branch pooling attention module, mapping several image features contained in the staged image features to a joint feature space using a linear layer, and concatenating them along the channel dimension in the joint feature space to obtain multi-layer fusion features; using a dual-branch pooling attention module, performing global average pooling on the multi-layer fusion features to obtain global average pooling features of the image; using a dual-branch pooling attention module, performing global max pooling on the multi-layer fusion features to obtain global max pooling features of the image; using a dual-branch pooling attention module, inputting the global average pooling features and the global max pooling features of the image into a multilayer perceptron to generate target channel weights; using a dual-branch pooling attention module, multiplying the target channel weights and the multi-layer fusion features element-wise, and adding the result obtained after element-wise multiplication to the multi-layer fusion features to obtain weighted image features; using a dual-branch pooling attention module, performing global average pooling on the weighted image features to obtain local image features.

[0038] In the specific implementation, a dual-branch pooling attention module is used, and a linear layer is employed to process the phased image features. The included image features are mapped to a joint feature space, and then concatenated along the channel dimension in the joint feature space to obtain multi-layer fused features. ; Multi-layer fusion features are processed through a dual-branch pooling attention module. Perform global average pooling to obtain the global average pooling features of the image. ; Multi-layer fusion features are processed through a dual-branch pooling attention module. Perform global max pooling to obtain the global max pooling features of the image. The image's global average pooling features are processed through a dual-branch pooling attention module. and global max pooling features of images Input to a multilayer perceptron to generate target channel weights. The target channel weights are adjusted using a dual-branch pooling attention module. and multi-layer fusion features Perform element-wise multiplication, and then combine the results of element-wise multiplication with the multi-layer fusion features. Add them together to obtain the weighted image features. The weighted image features are processed through a dual-branch pooling attention module. Perform global average pooling to obtain local image features. (i.e., the local context representation of fusion).

[0039] Step S106: Add the local image features to the category representation features generated by the image encoder to obtain global image features, and calculate the cosine similarity between the global image features and the initial text features to obtain an image-level anomaly score.

[0040] In the specific implementation, local image features are used. Category representation features generated by the image encoder Add them together to obtain the global image features. and global image features Compared with initial text features Cosine similarity is calculated to obtain the image-level anomaly score S.

[0041] Steps S101 to S106 as illustrated in this embodiment involve inputting the image to be detected into a trained anomaly detection network. The anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cueing module, a cross-attention guided anomaly map generation module, and a two-branch pooling attention module. The frequency domain dynamic cueing module models the global frequency domain features of the image to be detected to obtain target spectral features, and fuses these target spectral features with a multimodal cueing learning template to obtain target text cueing. The target text cueing is input into the text encoder to obtain initial text features. The cross-attention guided anomaly map generation module dynamically updates the initial text features to obtain target text features, and performs cosine similarity calculation between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The two-branch pooling attention module performs two-branch pooling processing on the staged image features to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features, and cosine similarity calculation is performed between the global image features and the initial text features to obtain an image-level anomaly score. This application embodiment integrates deep frequency domain features of images into text prompts through a frequency domain dynamic prompt module, reducing the model's reliance on specific product prompts. A cross-attention-guided anomaly map generation module dynamically updates text features using fine-grained image features, achieving more accurate image-text alignment and generating accurate anomaly maps. A dual-branch pooling attention module fuses multi-layered contextual information to obtain local image features, thereby calculating image-level anomaly scores and enhancing the ability to distinguish different anomaly patterns. In summary, this application embodiment leverages the strong generalization ability of visual language models, introducing a multimodal prompt learning mechanism combining a frequency domain dynamic prompt module, a cross-attention-guided anomaly map generation module, and a dual-branch pooling attention module. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization.

[0042] To explain in detail the principle of the technical solution of the present invention, the overall process of the present invention will be described below with reference to some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and should not be regarded as a limitation of the present invention.

[0043] This application provides a zero-shot anomaly detection method based on multimodal cue learning, which can automatically identify abnormal regions in images that deviate from normal patterns without training on specific categories of samples. Examples include scratches and cracks on the surface of industrial products, or lesions in medical images. Current methods generally rely on manually designed text prompts, which not only require high domain knowledge and are costly to design, but also struggle to accurately depict the complex structural features of images. Therefore, this application utilizes the strong generalization ability of visual language models and introduces a multimodal cue learning mechanism. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization. Specifically: a frequency domain dynamic cue module is designed to integrate deep frequency domain features of the image into text prompts, reducing reliance on specific product prompts; a cross-attention guided anomaly map generation module is constructed to dynamically update text features using fine-grained image features, achieving more accurate image-text alignment and generating anomaly maps; furthermore, a dual-branch pooling attention module is introduced to fuse multi-layered contextual information, enhancing the ability to discriminate different anomaly patterns. The zero-shot anomaly detection method based on multimodal cue learning provided in this application can be widely applied in fields such as industrial quality inspection and medical image analysis, achieving efficient, accurate and universal zero-shot anomaly detection.

[0044] Please see Figure 3 , Figure 3 This is a flowchart illustrating a zero-shot anomaly detection method based on multimodal cue learning provided in an embodiment of this application, as shown below. Figure 3 As shown in the figure, the overall implementation process of a zero-shot anomaly detection method based on multimodal cue learning provided in this application includes three stages: anomaly detection network construction, anomaly detection network training, and anomaly detection network application. The specific implementation content is as follows: The first phase involves building an anomaly detection network; Step 1, Construct the frequency domain dynamic prompt module: To enhance the model's ability to perceive specific semantic information in images, embodiments of this application introduce frequency domain dynamic features during the prompt generation process. For example... Figure 2 As shown, the specific implementation details of the frequency domain dynamic prompt module are as follows: First, image I is input into the anomaly detection network, and Fourier transform (FFT) is used to transform the input image I to the frequency domain. This Fourier transform process can be represented as: ,in, The frequency domain representation of the image is represented by FFT, which means performing a Fourier transform on the image; then, the frequency domain representation is... Decomposition yields the amplitude spectrum A and phase spectrum P. In the frequency domain features, the amplitude and phase spectra are two complementary types of information. The amplitude spectrum primarily carries the global structural information of the image, while the phase spectrum contains local detail information. Specifically, the frequency domain representation of the image... The process of decomposing to obtain the amplitude spectrum A and the phase spectrum P can be represented by the following formula: ; ; Furthermore, in this embodiment, a convolutional module is used to spatially model the frequency domain features (here, the frequency domain features are the amplitude spectrum A and the phase spectrum P) to obtain the amplitude spectrum features. and phase spectrum characteristics The convolutional module (ConvB) consists of convolutional layers, normalization layers, and activation layers. The processing of the convolutional module can be represented by the following formula: ; ; Regarding the construction of text prompts, the template format used in this application embodiment is "a photo of a [state][ ][ ]…[ ]”, where “a photo of a” is fixed text, and [state] represents words describing the state of the target (such as good or damaged). ][ ]…[ ] is a set of learnable embedding vectors used to encode category-specific semantic information and is initialized with a standard normal distribution. In this embodiment, i is 2.

[0045] Subsequently, the frequency domain features (here, the frequency domain features are amplitude spectrum features) are... and phase spectrum characteristics The fused category representation is obtained by adding the learnable vector to the fused vector, where: , The above fusion categories are represented as follows and Replacing it with the prompt template, the updated text prompt is "a photo of a [state][ ] [ ]”; Finally, the text prompt reads "a photo of a [state][ ] [ The text is input into the CLIP model's text encoder to obtain the corresponding text embedding vector (also known as text feature). .

[0046] Step 2, Construct a cross-attention-guided anomaly graph generation module: Since step 1 introduces frequency domain features into the text prompt, aligning the text embedding with fine-grained image features becomes complex. Therefore, this application proposes a cross-attention-guided anomaly map generation module that uses a cross-attention-based guidance mechanism to dynamically update text features and generate anomaly maps.

[0047] Specifically, such as Figure 2 As shown, the anomaly detection network includes an image encoder. The image encoder used in this embodiment is the visual encoder of the pre-trained visual language model CLIP. This image encoder consists of a 24-layer Transformer structure. In this embodiment, the feature outputs of layers 6, 12, 18, and 24 are extracted and labeled as the first stage, second stage, third stage, and fourth stage, respectively. The image feature outputs obtained in the first, second, third, and fourth stages are respectively denoted as... , , , Furthermore, this image encoder performs global processing on the input image and outputs a category representation feature. .

[0048] like Figure 2 As shown, for text features Image features at each stage (i=1, 2, 3, 4), firstly, the text features are processed through a one-dimensional convolutional layer. Convert to query ,in: ,in, This is the weight matrix of the convolutional layer; simultaneously, the image features of each stage are... Converted into bonds using one-dimensional convolution respectively Sum ,in: , ,in, and This represents the weight matrix of the corresponding convolutional layer. Based on the cross-attention mechanism in the cross-attention-guided anomaly graph generation module, updated text features can be obtained. The process of cross-attention mechanism can be expressed by the following formula: ; Where C represents image features The channel dimension, In this context, T represents the transpose calculation. This represents the activation function.

[0049] Subsequently, a staged anomaly map is generated by calculating image-text similarity, which represents the image features of each stage. With updated text features Calculate the similarity to obtain the initial anomaly map corresponding to each stage. The process of generating the stage anomaly map can be represented by the following formula: ; in, This indicates that bilinear interpolation is used for upsampling. The T in the upper right corner represents the transpose symbol. The "T" in the bottom right corner indicates a text tag. The anomaly map generated in the i-th stage is represented by the anomaly map of the target image that is finally obtained by averaging the anomaly maps of the four stages, thereby achieving accurate localization of anomaly regions under multi-level information fusion.

[0050] Step 3, construct the dual-branch pooling attention module: To further aggregate fine-grained image features and improve anomaly detection capabilities, this application proposes a dual-branch pooling attention module. The core objective of this module is to integrate the fused local image features... Category representation of image encoder output These are combined to calculate image-level anomaly scores.

[0051] Specifically, such as Figure 2 As shown, image features at different stages are first... Image features from different stages are mapped to a joint feature space through linear layers. The features are concatenated along the channel dimension to obtain the fused feature representation. To effectively capture contextual information and enhance the expressive power of channel features, this application employs a parallel dual-branch pooling strategy. Global average pooling (GAP) calculates the average value within the window, preserving feature distribution patterns and structural information, which is beneficial for the model to understand the global semantics of the image. Global max pooling (GMP) selects the maximum value within the window, highlighting local salient features. Specifically, for the fused feature representation... The calculation process for dual-branch pooling is as follows: ; ; Subsequently, the features obtained after global average pooling (GAP) are... Features obtained after Global Max Pooling (GMP) The data are fed into a multilayer perceptron (MLP) and generated separately. and The corresponding channel weights are represented and summed element by element. and These two sets of channel weights are fused to obtain the final channel weights. The processing procedure of a multilayer perceptron (MLP) can be represented by the following formula: ; in, This represents the sigmoid activation function.

[0052] Next, channel weights are used. Multi-layer fusion features Perform element-wise multiplication and combine with multi-layer fusion features Add them together to obtain the reweighted image features: ; in, This indicates element-wise multiplication.

[0053] Subsequently, the reweighted image features Perform global average pooling to obtain the fused local context representation. and represent it with the category of the image encoder output. Adding them together yields the global image embedding vector: Among them, the reweighted image features The purpose of global average pooling is to reweight the image features. The dimensions are adjusted to match the category representation output by the image encoder. Consistent, facilitating category representation with the image encoder output. Add them together.

[0054] Next, the global image embedding vector is... Text features Perform normalization, and then calculate the normalized result. and The cosine similarity between the two is used to obtain the image-level anomaly score S. The calculation process of the image-level anomaly score S is as follows: ; ; ; in, and Indicates to proceed Normalization, the first In this context, T represents the transpose symbol.

[0055] The second phase involves training the anomaly detection network. To verify the detection performance of the anomaly detection network constructed in this application embodiment, the model was trained on the VisA dataset and its generalization ability was evaluated on other datasets. When evaluating on the VisA dataset itself, the MVTec-AD dataset was used as the training set to maintain a zero-shot detection experimental setup. During training, the ViT-L / 14-336 CLIP model was used as both the image encoder and text encoder, with both parameters kept fixed. The input image size was set to 518×518, the optimizer was Adam, and the learning rate was set to... The training process involved 20 iterations. All experiments were conducted on a single NVIDIA A100 (80G) GPU, resulting in a fully trained anomaly detection network.

[0056] The third stage involves the application of the anomaly detection network. It should be noted that the parameter characteristics used in this stage follow the parameter representations used in the first stage (constructing the anomaly detection network) (e.g., the image to be detected is represented by parameter I). In practical applications, different parameters can be used to represent the data, and this embodiment does not impose any restrictions on this. The technical implementation process and principles of this application stage are similar to the implementation processes and principles of each module in the first stage. This embodiment only provides a brief illustrative description of the overall process of the anomaly detection network application; detailed descriptions of each step can be found in the relevant content of the aforementioned embodiments.

[0057] In practical implementation, the main process of anomaly detection network application is as follows: input the image to be detected I into the trained anomaly detection network, firstly utilizing the extracted visual features... With learned text features Calculate cosine similarity to generate a predicted anomaly map; then, based on global image features... Text features The cosine similarity between the two is used to calculate the image-level anomaly score, thereby achieving accurate localization of abnormal regions and overall anomaly determination.

[0058] It should be noted that this embodiment is only a brief illustrative description of the overall process of the zero-shot anomaly detection method based on multimodal cue learning. Detailed descriptions of each step can be found in the relevant content of the foregoing embodiments, and will not be repeated here. It is understood that the present invention does not limit this.

[0059] This application embodiment inputs the image to be detected into a trained anomaly detection network. The anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cueing module, a cross-attention guided anomaly map generation module, and a two-branch pooling attention module. The frequency domain dynamic cueing module models the global frequency domain features of the image to be detected to obtain target spectral features, and fuses these target spectral features with a multimodal cueing learning template to obtain target text cueing. The target text cueing is input into the text encoder to obtain initial text features. The cross-attention guided anomaly map generation module dynamically updates the initial text features to obtain target text features, and performs cosine similarity calculation between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The two-branch pooling attention module performs two-branch pooling processing on the staged image features to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features, and the cosine similarity calculation between the global image features and the initial text features yields an image-level anomaly score. This application embodiment integrates deep frequency domain features of images into text prompts through a frequency domain dynamic prompt module, reducing the model's reliance on specific product prompts. A cross-attention-guided anomaly map generation module dynamically updates text features using fine-grained image features, achieving more accurate image-text alignment and generating accurate anomaly maps. A dual-branch pooling attention module fuses multi-layered contextual information to obtain local image features, thereby calculating image-level anomaly scores and enhancing the ability to distinguish different anomaly patterns. In summary, this application embodiment leverages the strong generalization ability of visual language models, introducing a multimodal prompt learning mechanism combining a frequency domain dynamic prompt module, a cross-attention-guided anomaly map generation module, and a dual-branch pooling attention module. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization.

[0060] In summary, the zero-shot anomaly detection method based on multimodal cue learning provided in this application aims to improve the adaptability of the visual language model CLIP in zero-shot anomaly detection tasks by adaptively perceiving global and local contextual information. This method, trained on an auxiliary dataset, achieves effective localization of anomaly regions of unseen categories and exhibits good generalization performance even with significant domain differences. The zero-shot anomaly detection method based on multimodal cue learning provided in this application mainly includes three core components: a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module. Specifically, given an input image, the system first models the global frequency domain features of the image using a frequency domain dynamic prompting module and integrates them into a general prompting template, thereby enhancing the semantic expressiveness of the text prompts. Then, a cross-attention guided anomaly map generation module is introduced to dynamically guide and refine the text features obtained from the frequency domain dynamic prompting module, achieving effective alignment with the image's fine-grained spatial structure and semantic information, and further generating anomaly maps. This enables precise localization of anomaly regions, effectively improving the accuracy and robustness of anomaly localization. Finally, a dual-branch pooling attention module fuses the local image features from the four stages output by the image encoder, models channel dependencies, strengthens the focus on key anomaly features, and further fuses the obtained local features with the category embedding features extracted by the image encoder. These are then used to calculate anomaly scores with the text embeddings, thereby achieving accurate anomaly detection.

[0061] The key point of the zero-shot anomaly detection method based on multimodal cue learning provided in this application embodiment is: (1) By introducing image frequency domain features into text prompts through the frequency domain dynamic prompt module, adaptive perception of different detection categories can be achieved; (2) The cross-attention-guided anomaly graph generation module dynamically extracts text features based on fine-grained image features, which is beneficial for accurately locating anomaly regions; (3) Enhance the ability to identify anomalies by fusing multi-level contextual information through a dual-branch pooling attention module.

[0062] Please see Figure 4 This application also provides a zero-shot anomaly detection device 400 based on multimodal cue learning, which can implement the above-described method. The device includes the following modules: The image input module 401 is used to input the image to be detected into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic prompting module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; The frequency domain dynamic prompting construction module 402 is used to model the global frequency domain features of the image to be detected through the frequency domain dynamic prompting module to obtain the target spectral features, and to fuse the target spectral features with the multimodal prompting learning template to obtain the target text prompt; The text prompt encoding module 403 is used to input the target text prompt into the text encoder to obtain initial text features; The image anomaly generation module 404 is used to dynamically update the initial text features through the cross-attention guidance anomaly generation module to obtain target text features, and to perform cosine similarity calculation on the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The dual-branch pooling processing module 405 is used to perform dual-branch pooling processing on the staged image features through the dual-branch pooling attention module to obtain local image features. The image-level anomaly score calculation module 406 is used to add the local image features to the category representation features generated by the image encoder to obtain global image features, and to calculate the cosine similarity between the global image features and the initial text features to obtain the image-level anomaly score.

[0063] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0064] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0065] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0066] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the methods described in the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0067] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0068] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0069] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0070] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0071] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] The zero-shot anomaly detection method, apparatus, electronic device, storage medium, and program product based on multimodal cue learning provided in this application embodiment involve inputting the image to be detected into a trained anomaly detection network. The anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a two-branch pooling attention module. The frequency domain dynamic cue module models the global frequency domain features of the image to be detected to obtain target spectral features, and fuses these target spectral features with a multimodal cue learning template to obtain target text cue. The target text cue is input into the text encoder to obtain initial text features. The cross-attention guided anomaly map generation module dynamically updates the initial text features to obtain target text features, and performs cosine similarity calculation between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map. The two-branch pooling attention module performs two-branch pooling processing on the staged image features to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features, and the cosine similarity calculation between the global image features and the initial text features yields an image-level anomaly score. This application embodiment integrates deep frequency domain features of images into text prompts through a frequency domain dynamic prompt module, reducing the model's reliance on specific product prompts. A cross-attention-guided anomaly map generation module dynamically updates text features using fine-grained image features, achieving more accurate image-text alignment and generating accurate anomaly maps. A dual-branch pooling attention module fuses multi-layered contextual information to obtain local image features, thereby calculating image-level anomaly scores and enhancing the ability to distinguish different anomaly patterns. In summary, this application embodiment leverages the strong generalization ability of visual language models, introducing a multimodal prompt learning mechanism combining a frequency domain dynamic prompt module, a cross-attention-guided anomaly map generation module, and a dual-branch pooling attention module. By jointly modeling local and global contextual dependencies, it effectively improves the accuracy and robustness of anomaly localization.

[0073] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0074] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0075] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0076] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0077] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0078] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0080] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0082] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0083] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A zero-shot anomaly detection method based on multimodal cue learning, characterized in that, The method includes the following steps: The image to be detected is input into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic cue module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; The frequency domain dynamic prompting module models the global frequency domain features of the image to be detected to obtain target spectral features, and then fuses the target spectral features with the multimodal prompting learning template to obtain target text prompts. The target text prompt is input into the text encoder to obtain the initial text features; The cross-attention guided anomaly graph generation module dynamically updates the initial text features to obtain target text features, and calculates the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly graph. The staged image features are processed by bi-branch pooling attention module to obtain local image features. The local image features are added to the category representation features generated by the image encoder to obtain global image features. The cosine similarity between the global image features and the initial text features is calculated to obtain the image-level anomaly score. The step of performing bi-branch pooling processing on the staged image features through the bi-branch pooling attention module to obtain local image features includes: The dual-branch pooling attention module uses a linear layer to map several image features contained in the staged image features to the joint feature space, and then performs channel-dimensional concatenation in the joint feature space to obtain multi-layer fusion features. The multi-layer fusion features are subjected to global average pooling through the dual-branch pooling attention module to obtain global average pooling features of the image. The multi-layer fusion features are subjected to global max pooling through the dual-branch pooling attention module to obtain global max pooling features of the image. The dual-branch pooling attention module inputs the global average pooling features and the global max pooling features of the image into the multilayer perceptron to generate target channel weights. The dual-branch pooling attention module multiplies the target channel weights and the multi-layer fusion features element-wise, and adds the result of the element-wise multiplication to the multi-layer fusion features to obtain weighted image features. The local image features are obtained by performing global average pooling on the weighted image features through the dual-branch pooling attention module.

2. The method according to claim 1, characterized in that, The step of modeling the global frequency domain features of the image to be detected through the frequency domain dynamic prompting module to obtain the target spectral features includes: The frequency domain dynamic prompting module performs Fourier transform processing on the image to be detected to obtain an initial frequency domain representation. The frequency domain dynamic prompting module performs feature decomposition on the initial frequency domain representation to obtain initial spectral features; wherein, the initial spectral features include amplitude spectrum and phase spectrum; The frequency domain dynamic prompting module uses a convolution module to spatially model the initial spectral features to obtain the target spectral features; wherein, the target spectral features include amplitude spectrum features and phase spectrum features.

3. The method according to claim 1, characterized in that, The multimodal cue learning template includes learnable embedding vectors. The step of fusing the target spectral features with the multimodal cue learning template to obtain the target text cue includes: The frequency domain dynamic prompting module adds the target spectral features to the learnable embedding vector to obtain a fused category representation. The frequency domain dynamic prompting module replaces the original content in the learnable embedding vector with the fused category representation to obtain the target text prompt.

4. The method according to claim 1, characterized in that, The step of dynamically updating the initial text features through the cross-attention-guided anomaly graph generation module to obtain target text features includes: The initial text features are converted into text query vectors through the convolutional layers in the cross-attention guided anomaly graph generation module. The convolutional layer in the cross-attention guided anomaly graph generation module converts several image features contained in the staged image features into corresponding keys and values. The cross-attention-guided anomaly graph generation module, based on the cross-attention mechanism, dynamically updates the initial text features according to the text query vector and the keys and values ​​corresponding to several image features contained in the staged image features, to obtain the target text features.

5. The method according to claim 1, characterized in that, The step of calculating the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly map includes: The cross-attention guided anomaly graph generation module performs cosine similarity calculation on the target text features and several image features contained in the staged image features to obtain initial image anomaly graphs corresponding to several image features. The cross-attention guided anomaly map generation module calculates the average of several initial image anomaly maps to obtain the target image anomaly map.

6. A zero-shot anomaly detection device based on multimodal cue learning, characterized in that, The device includes the following modules: The image input module is used to input the image to be detected into the trained anomaly detection network; wherein, the anomaly detection network includes an image encoder, a text encoder, a frequency domain dynamic prompting module, a cross-attention guided anomaly map generation module, and a dual-branch pooling attention module; The frequency domain dynamic prompting construction module is used to model the global frequency domain features of the image to be detected through the frequency domain dynamic prompting module to obtain the target spectral features, and to fuse the target spectral features with the multimodal prompting learning template to obtain the target text prompt; A text prompt encoding module is used to input the target text prompt into the text encoder to obtain initial text features; The image anomaly generation module is used to dynamically update the initial text features by guiding the anomaly generation module through the cross attention to obtain target text features, and to calculate the cosine similarity between the target text features and the staged image features generated by the image encoder to generate a target image anomaly. A dual-branch pooling processing module is used to perform dual-branch pooling processing on the staged image features through the dual-branch pooling attention module to obtain local image features; The image-level anomaly score calculation module is used to add the local image features to the category representation features generated by the image encoder to obtain global image features, and to calculate the cosine similarity between the global image features and the initial text features to obtain the image-level anomaly score. Specifically, the dual-branch pooling processing module is used for: The dual-branch pooling attention module uses a linear layer to map several image features contained in the staged image features to the joint feature space, and then performs channel-dimensional concatenation in the joint feature space to obtain multi-layer fusion features. The multi-layer fusion features are subjected to global average pooling through the dual-branch pooling attention module to obtain global average pooling features of the image. The multi-layer fusion features are subjected to global max pooling through the dual-branch pooling attention module to obtain global max pooling features of the image. The dual-branch pooling attention module inputs the global average pooling features and the global max pooling features of the image into the multilayer perceptron to generate target channel weights. The dual-branch pooling attention module multiplies the target channel weights and the multi-layer fusion features element-wise, and adds the result of the element-wise multiplication to the multi-layer fusion features to obtain weighted image features. The local image features are obtained by performing global average pooling on the weighted image features through the dual-branch pooling attention module.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images