Zero-shot anomaly detection method, device and electronic equipment

Abstract

The application provides a zero-shot anomaly detection method, device and electronic equipment, and relates to the technical field of image anomaly detection. The method comprises the following steps: acquiring a to-be-detected image, and extracting a visual feature by using a CLIP model; performing visual enhancement on the visual feature to obtain an enhanced visual feature; injecting the enhanced visual feature into a learnable text prompt template to generate an adaptive text prompt; injecting the adaptive text prompt into a text encoder for coding to obtain a text embedding representation; mapping the adaptive text prompt to a visual space to obtain a visual prompt, and inputting the visual prompt into a local visual feature to obtain a scale visual feature; and calculating an anomaly score and an anomaly positioning map of a preset scale based on the text embedding representation and the scale visual feature respectively, and fusing the anomaly score and the anomaly positioning map to obtain a detection result. The zero-shot anomaly detection method, device and electronic equipment provided by the application effectively improve the generalization ability of the entire detection process, thereby meeting the actual application requirements.

Description

Technical Field

[0001] This invention relates to the technical field of image anomaly detection, and in particular to a zero-sample anomaly detection method, apparatus, and electronic device. Background Technology

[0002] Automated anomaly detection has become a key task for quality control and improving production efficiency in today's society. Traditional anomaly detection methods mainly rely on supervised and semi-supervised learning models. These methods are often limited by the difficulty in obtaining or the insufficient amount of data in practical applications, thus restricting their application in certain limited scenarios. To overcome these limitations, zero-shot anomaly detection methods have been proposed. Zero-shot anomaly detection methods do not require a large amount of domain-specific data. Conversely, such methods are required to have good generalization performance. They are required to learn general anomaly patterns on a common anomaly dataset and then transfer them to a specific domain. Although such methods have significant advantages in not requiring a large amount of domain-specific labeled data, they still face challenges at present, such as weak spatial representation capabilities and poor cross-scale capabilities, which limit the universality and accuracy of such methods in applications. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a zero-sample anomaly detection method, apparatus and electronic device to alleviate the above-mentioned technical problems.

[0004] In a first aspect, embodiments of the present invention provide a zero-sample anomaly detection method, the method comprising: acquiring an image to be detected; extracting visual features of the image to be detected using a pre-trained CLIP model visual encoder, wherein the visual features include: global visual features and local visual features at multiple preset scales; visually enhancing the visual features to obtain enhanced visual features suitable for the preset scales; injecting the enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts containing multiple preset scales; injecting the adaptive text prompts into the CLIP model text encoder, and using the text encoder to process the adaptive text prompts... The text prompts are encoded, and text embedding representations at each preset scale are output. A preset adapter model is used to map the adaptive text prompts to a visual space to obtain visual prompts. These visual prompts are then input into the local visual features of the image to be detected at each preset scale to obtain scale visual features at each preset scale. Based on the text embedding representations and the scale visual features, corresponding anomaly scores and anomaly localization maps are calculated in the preset scale space. The anomaly scores and anomaly localization maps at all scales are fused to output the detection results for the image to be detected, wherein the detection results include global anomaly scores and global anomaly localization maps.

[0005] In conjunction with the first aspect, the present invention provides a first possible implementation of the first aspect, wherein the step of visually enhancing the visual features to obtain enhanced visual features suitable for a preset scale includes: mapping the global visual features to a generalized scale space using a pre-built perceptron model to obtain a first enhanced feature in the scale space; further enhancing the first enhanced feature according to a pre-configured cross-attention mechanism to obtain a second enhanced feature; and superimposing the second enhanced feature with the first enhanced feature to obtain the enhanced visual feature.

[0006] In conjunction with the first possible implementation of the first aspect, this embodiment of the invention provides a second possible implementation of the first aspect, wherein the step of further enhancing the first enhanced feature according to a pre-configured cross-attention mechanism to obtain the second enhanced feature includes: sequentially concatenating the first enhanced feature in the scale space with the local visual features in multiple preset scales to obtain a concatenated feature; using the first enhanced feature as a query vector, and the concatenated feature as a key matrix and a value matrix, further enhancing the first enhanced feature using a pre-configured cross-attention algorithm to obtain the second enhanced feature.

[0007] In conjunction with the first aspect, this embodiment of the invention provides a third possible implementation of the first aspect, wherein the steps of calculating the corresponding anomaly score and anomaly localization map in a preset scale space based on the text embedding representation and the scale visual features respectively include: calculating the cosine similarity between the text embedding representation and the scale visual features at each preset scale to obtain anomaly localization maps at each preset scale, wherein the anomaly localization maps satisfy preset constraints; sorting the local visual features at each preset scale according to the anomaly degree of each anomaly localization map, and selecting a preset number of local visual features with the highest anomaly degree; performing clustering processing on the preset number of local visual features with the highest anomaly degree to obtain anomaly region cluster centers; using the global visual features of the image to be detected to enhance the anomaly region cluster centers to obtain enhanced anomaly region cluster centers; calculating the cosine similarity between the enhanced anomaly region cluster centers and the text embedding representation at each preset scale to obtain the anomaly score at the preset scale.

[0008] In conjunction with the first aspect, this embodiment of the invention provides a fourth possible implementation of the first aspect, wherein the step of fusing the anomaly scores and anomaly localization maps at all scales to output the detection result corresponding to the image to be detected includes: aggregating the anomaly scores at each preset scale to obtain a global anomaly score; and aggregating the anomaly localization maps at each preset scale to obtain a global anomaly localization map; and generating a detection result containing the global anomaly score and the global anomaly localization map.

[0009] In conjunction with the first possible implementation of the first aspect, this embodiment of the invention provides a fifth possible implementation of the first aspect, wherein the method further includes: injecting the adaptive text prompt into a pre-configured learnable text prompt template to construct a learnable text prompt; wherein the learnable text prompt template includes a normal prompt template and an abnormal prompt template, and the adaptive text prompt is injected bit-by-bit into the learnable prompt portions of the normal prompt template and the abnormal prompt template to constitute a normal prompt and an abnormal prompt; constructing fine-tuning prompt features based on the enhanced visual features according to pre-configured parameters; wherein the fine-tuning prompt features include: normal prompt features, abnormal prompt features, and adaptive prompt features; adding the normal prompt features bit-by-bit to the learnable portion of the normal prompt, and adding the abnormal prompt features bit-by-bit to the learnable portion of the abnormal prompt to obtain learnable parameters; and using the adaptive prompt features to fine-tune the prompts of the text encoder based on the learnable parameters.

[0010] In conjunction with the fifth possible implementation of the first aspect, this embodiment of the invention provides a sixth possible implementation of the first aspect, wherein the steps of injecting the adaptive text prompt into the text encoder of the CLIP model, encoding the adaptive text prompt through the text encoder, and outputting the text embedding representation at each preset scale include: injecting the adaptive text prompt into the text encoder after prompt fine-tuning, encoding the adaptive text prompt through the fine-tuned text encoder, and outputting the text embedding representation at each preset scale.

[0011] In conjunction with the first aspect, the present invention provides a seventh possible implementation of the first aspect, wherein the above method further includes: injecting the visual cues into the visual encoder and performing cues tuning on the visual encoder.

[0012] Secondly, embodiments of the present invention also provide a zero-sample anomaly detection device, the device comprising: an extraction module for acquiring an image to be detected and extracting visual features of the image to be detected using a pre-trained CLIP model visual encoder, wherein the visual features include: global visual features and local visual features at multiple preset scales; an enhancement module for visually enhancing the visual features to obtain enhanced visual features suitable for the preset scales; a prompting module for injecting the enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts containing multiple preset scales; and a feature module for injecting the adaptive text prompts into the CLIP model text encoder, and using the text encoding... The device encodes the adaptive text prompt and outputs a text embedding representation at each preset scale; and maps the adaptive text prompt to a visual space using a preset adapter model to obtain a visual prompt, and inputs the visual prompt into the local visual features at each preset scale of the image to be detected to obtain scale visual features at each preset scale: a calculation module is used to calculate the corresponding anomaly score and anomaly localization map in the preset scale space based on the text embedding representation and the scale visual features respectively; a detection module is used to perform fusion processing on the anomaly score and the anomaly localization map at all scales, and output the detection result corresponding to the image to be detected, wherein the detection result includes a global anomaly score and a global anomaly localization map.

[0013] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] The zero-shot anomaly detection method, apparatus, and electronic device provided in this invention can acquire an image to be detected, extract visual features of the image using a pre-trained CLIP model visual encoder, enhance the visual features to obtain enhanced visual features suitable for a preset scale, inject the enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts containing multiple preset scales, inject the adaptive text prompts into the CLIP model text encoder, encode the adaptive text prompts through the text encoder, and output the text embedding representation for each preset scale, and map the adaptive text prompts to the visual space using a preset adapter model to obtain a visual representation. The system provides visual cues and inputs them into the local visual features of the image to be detected at each preset scale, resulting in scale visual features at each preset scale. Based on text embedding representation and scale visual features, the corresponding anomaly scores and anomaly localization maps are calculated in the preset scale space. The anomaly scores and anomaly localization maps at all scales are fused to output the detection results for the image to be detected. Furthermore, visual enhancement processing is applied to the visual features during the detection process, which can effectively improve the detection accuracy. At the same time, the adaptive text cues at multiple preset scales constructed using these enhanced visual features can improve the feature representation ability at different scales and viewpoints, thereby effectively improving the generalization ability of the entire detection process and meeting the needs of practical applications.

[0016] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 A flowchart of a zero-sample anomaly detection method provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating a visual enhancement process provided in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the structure of a zero-sample anomaly detection device provided in an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Currently, deep learning-based product quality inspection methods have become a key means of ensuring production efficiency and product qualification rates in industrial production processes. Existing anomaly detection technologies mainly rely on supervised or semi-supervised learning methods, which typically require collecting anomaly samples from specific domains for model training. However, in certain application scenarios, the difficulty in obtaining anomaly samples and the limited number of such samples restrict the applicability and generalization capabilities of deep learning methods.

[0025] To address the aforementioned challenges, zero-shot anomaly detection methods have been extensively studied. Compared to traditional supervised or semi-supervised anomaly detection methods, zero-shot anomaly detection methods do not rely on anomaly samples specific to a particular scenario. These methods first learn anomaly patterns from a large amount of data in a general domain and then transfer them to specific domain detection tasks. Therefore, this method is required to possess good generalization ability, enabling it to accurately identify anomalies in different application environments.

[0026] However, existing zero-shot anomaly detection methods still face certain technical bottlenecks. On the one hand, current methods mostly use a single text cue space to represent various anomaly patterns. A single cue is insufficient to comprehensively cover anomaly features at different semantic levels and scales. For example, surface scratches and structural deformations differ significantly in semantic understanding and feature granularity. A single cue cannot effectively represent these different types of anomalies simultaneously, which affects detection accuracy. On the other hand, existing methods use the same text cue for calculation at all scales of visual feature levels when generating anomaly localization maps. They ignore the differences in visual features at different scales, which can easily introduce irrelevant noise, thereby reducing the accuracy of anomaly localization.

[0027] Based on this, the zero-sample anomaly detection method, apparatus, and electronic device provided by the embodiments of the present invention can effectively alleviate the above-mentioned technical problems and can effectively adapt to complex and ever-changing specific fields, meeting the needs of practical applications.

[0028] To facilitate understanding of this embodiment, a zero-sample anomaly detection method disclosed in this embodiment of the invention will first be described in detail.

[0029] In one possible implementation, embodiments of the present invention provide a zero-sample anomaly detection method, which can be applied to industrial, medical, and other fields to detect anomalies in images. Specifically, such as... Figure 1 The flowchart shown illustrates a zero-sample anomaly detection method, which includes the following steps:

[0030] Step S102: Obtain the image to be detected and extract the visual features of the image to be detected using the visual encoder of the pre-trained CLIP model.

[0031] The visual features in this embodiment of the invention include: global visual features and local visual features at multiple preset scales;

[0032] Specifically, the CLIP (Contrastive Language-Image Pre-training) model is a multimodal pre-trained model that typically includes a visual encoder and a text encoder. In this embodiment of the invention, the CLIP model is used as a zero-shot anomaly detection model to extract visual features from the image to be detected.

[0033] Furthermore, the image to be detected in the embodiments of the present invention can be an image acquired using imaging equipment in industrial or medical fields, or it can be an image acquired manually in real time. The specific application shall prevail, and the embodiments of the present invention do not impose any restrictions on this.

[0034] Step S104: Visually enhance the visual features to obtain enhanced visual features suitable for a preset scale;

[0035] Step S106: Inject enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts with multiple preset scales;

[0036] In practical use, the process of generating adaptive text prompts in this step can be implemented using a hierarchical multimodal prompt optimization approach. Furthermore, the learnable text prompt template in this embodiment is typically an object-independent learnable text prompt template. Therefore, the process of generating adaptive text prompts with multiple preset scales in step S106 is actually a process of constructing text prompts at different scales that are object-independent; typically, it is denoted as... Where i represents different preset scales and n is the length of the text prompt. This Pi is also called a multi-scale prompt, where different scales focus on features at different spatial resolutions.

[0037] Step S108: Inject the adaptive text prompt into the text encoder of the CLIP model, encode the adaptive text prompt using the text encoder, and output the text embedding representation at each preset scale; and,

[0038] Step S110: Using a preset adapter model, the adaptive text prompt is mapped to the visual space to obtain the visual prompt. The visual prompt is then input into the local visual features of each preset scale of the image to be detected to obtain the scale visual features of each preset scale.

[0039] In step S110, the mapping relationship between the adaptive text prompt and the visual space can be expressed as follows:

[0040]

[0041] Among them, Adapter i The mapping relationship representing the adapter is a learnable linear transformation layer. The visual cue obtained from the mapping can be represented as... Should It can be injected into the corresponding stage of the visual encoder of the CLIP model to fine-tune the visual features.

[0042] Step S112: Calculate the corresponding anomaly score and anomaly localization map in the preset scale space based on text embedding representation and scale visual features, respectively.

[0043] Step S114: Perform fusion processing on all the anomaly scores and anomaly localization maps at the preset scales, and output the detection results corresponding to the image to be detected.

[0044] In this embodiment of the invention, the detection results include a global anomaly score and a global anomaly localization map. In practical use, a score threshold can be preset; when the score exceeds this threshold, the image is considered to have an anomaly. Therefore, the global anomaly score characterizes the degree of anomaly in the image being detected; the higher the global anomaly score, the higher the degree of anomaly in the image. In the global anomaly localization map, every pixel has an anomaly score; the higher the anomaly score of a pixel, the more likely that pixel is to have an anomaly. In other words, the global anomaly localization map marks out areas with anomalies.

[0045] In practical use, the global visual features mentioned above can capture the overall structure and contextual information of the image, providing high-level semantic guidance for visual understanding. Local visual features, on the other hand, focus on fine-grained patterns such as texture and edges. In this embodiment of the invention, in order to effectively capture anomalous features, the visual enhancement in step S104 includes two enhancement processes. Specifically, Figure 2 A flowchart illustrating a visual enhancement process is shown, such as... Figure 2 As shown, it includes the following steps:

[0046] Step S202: Use a pre-built perceptron model to map global visual features to a generalized scale space to obtain the first enhanced features in this scale space.

[0047] In a specific implementation, the global visual features extracted in step S102 above are represented as T. CLS Local visual features at multiple preset scales are represented as Where i represents different preset scales;

[0048] Furthermore, the perceptron model in this embodiment of the invention can be a perceptron model composed of two linear layers, which will integrate the aforementioned global visual features T. CLS Mapping to the generalized scale space yields the first enhanced feature in that scale space; this first enhanced feature, also known as the preliminary enhanced scale feature, can be represented as T. scale .

[0049] Step S204: The first enhanced feature is further enhanced according to the pre-configured cross-attention mechanism to obtain the second enhanced feature;

[0050] Specifically, the first enhanced feature in the scale space is sequentially concatenated with local visual features in multiple preset scales to obtain the concatenated feature; then, using the first enhanced feature as the query vector and the concatenated feature as the key matrix and value matrix, the first enhanced feature is further enhanced using a pre-configured cross-attention algorithm to obtain the second enhanced feature.

[0051] Among them, the splicing feature can be represented as Its formula is expressed as follows:

[0052]

[0053] Furthermore, after using the cross-attention algorithm to further enhance the first enhanced feature to obtain the second enhanced feature, the following step S206 is executed.

[0054] Step S206: The second enhancement feature is superimposed on the first enhancement feature to obtain the enhanced visual feature.

[0055] In this process, step S206 is equivalent to using the residual channel and the initial enhanced scale feature T. scale Adding them together again can improve the fitting efficiency. Specifically, the formula for enhancing the expression of visual features is as follows:

[0056]

[0057] Among them, T i This represents the enhanced visual features; the part corresponding to the softmax() function is the second enhanced feature mentioned above, W. Q W K and W V The weights for the cross-attention algorithm are the query vector, key matrix, and value matrix, respectively; d k The feature dimension can be obtained from the input features.

[0058] In practical use, this embodiment of the invention also includes an optimization process for the CLIP model, namely, optimization of the text encoder in the CLIP model and optimization of the visual encoder in the CLIP model. The optimization process for the text encoder includes the following steps:

[0059] (1) Inject adaptive text prompts into a pre-configured learnable text prompt template to construct learnable text prompts; wherein, the learnable text prompt template includes a normal prompt template and an abnormal prompt template, and the adaptive text prompts are injected bit by bit into the learnable prompt parts of the normal prompt template and the abnormal prompt template to form normal prompts and abnormal prompts.

[0060] Where Tn represents the normal message template and Ta represents the abnormal message template, it can be represented as: T n =[V1][V2]…[Vn][STATE][OBJECT];

[0061] Ta=[W1][W2]…[Wn][STATE][OBJECT];

[0062] Where Vi represents a normal learnable hint, and Wi represents an abnormal learnable hint.

[0063] The above adaptive text prompt It can be injected bitwise into the above normal and abnormal prompt templates to construct learnable text prompts containing both normal and abnormal prompts. Specifically, this is done by injecting the adaptive text prompt P... i The learnable cue vectors at the corresponding positions, namely [V1][V2]…[Vn] and [W1][W2]…[Wn], are added bit by bit.

[0064] (2) Construct fine-tuned cue features based on enhanced visual features according to pre-configured parameters;

[0065] The fine-tuning prompt features include: normal prompt features, abnormal prompt features, and adaptation prompt features; their specific expression formulas are as follows:

[0066]

[0067] Among them, W normal W anomaly and W adapter B is a learnable weight parameter. normal B anomaly and B adapter The bias parameters are learnable. The weight parameters and bias parameters here are pre-configured parameters, and the weights of the normal prompt features, abnormal prompt features and adaptive prompt features mentioned above are not shared.

[0068] (3) Add the normal prompt features to the learnable part of the normal prompt bit by bit, and add the abnormal prompt features to the learnable part of the abnormal prompt bit by bit to obtain the learnable parameters;

[0069] (4) Based on learnable parameters, the text encoder is fine-tuned using adaptive cue features. Specifically, the above... This is a normal prompt feature. This is an abnormality indication feature. To adapt to the prompt features, in (3) above, it is to... and Compared with the normal prompt T n and abnormal prompt T a The learnable parts are summed bitwise to obtain the learning parameters. In (4) above, the learning parameters are obtained by using... The text encoder is fine-tuned based on learned parameters, and the text encoder is optimized. The expression for this optimization strategy is as follows:

[0070]

[0071] [T j+1 ,θ (j+1) ] = layer j ([Tj,θ j ]),j>J

[0072] Among them, layer j This represents the j-th Transformer layer. For the original input features of the j-th layer, θ j,i Let θ represent the learnable parameters of the j-th layer in the i-th preset scale cue space, where J is the preset threshold parameter. jThis indicates that there can be multiple learnable parameters in the j-th layer, θ. j,i For θ j The i-th one in Indicates a prompt feature, usually as follows: and The result is obtained by adding the two together.

[0073] Furthermore, the text encoder, after being optimized using the aforementioned optimization strategy, can be used to encode adaptive text prompts and output text embedding representations at each preset scale. That is, in step S108, the adaptive text prompts can be injected into the finely tuned text encoder, which then encodes the adaptive text prompts and outputs text embedding representations at each preset scale. In this embodiment of the invention, the text embedding representation at each preset scale is used... Characterize it.

[0074] Furthermore, the visual cues obtained in step S110 can be injected into the visual encoder to optimize it. That is, in this embodiment, a dual-stream cue optimization strategy is implemented. This strategy aims to achieve an effective mapping from the text cue space to the visual cue space, thereby completing the cascade optimization of the visual encoder and the text encoder. This mechanism not only maintains semantic consistency but also significantly enhances the visual encoder's sensitivity to and expressive power regarding anomalous features.

[0075] Specifically, when optimizing the visual encoder, the process is based on the mapping relationship described above, which maps adaptive text prompts to the visual space.

[0076] in, For the text encoder to learnable hints, and The learnable cue for the visual encoder obtained from the mapping, where, The real space, representing d t Let d be the dimension of this real number space, which is also the dimension of the text prompt. v Dimensions for visual cues.

[0077] The visual cues generated above The algorithm is integrated into different stages of the visual encoder using a cue-based optimization approach to enhance the encoder's ability to extract visual features relevant to abnormal tasks. Specifically, for the j-th layer at the i-th preset scale, the integration method is as follows:

[0078]

[0079] layer j This represents the j-th Transformer layer. For the original input features of the j-th layer, For each layer of learnable visual tokens, θ can be pre-set. j,i The cue vector is used as the input vector. j represents the layer number of the visual encoder, and i represents the scale number of the input to that layer. The fine-tuning process is controlled by hyperparameters: when j ≤ J, the cue is replaced; when j > J layers, only forward propagation is performed, and the cue is not replaced, thus preserving the deep pre-trained knowledge.

[0080] Furthermore, after optimizing the visual encoder as described above, the optimized visual encoder can be used to fine-tune the scale visual features. During the fine-tuning process, the visual features obtained by the fine-tuned visual encoder are added to and fused with the original scale visual features to obtain the fine-tuned scale visual features. The formula can be expressed as:

[0081]

[0082] in, E represents the scaled visual features after fine-tuning. img ΔE represents the scale visual characteristics before fine-tuning. img This represents the visual features obtained by the fine-tuned visual encoder.

[0083] Thus, in step S112 above, the anomaly score and anomaly localization map can be calculated using the fine-tuned scale visual features.

[0084] Through the aforementioned dual-stream cue optimization strategy, explicit guidance of the visual encoder by textual cues is achieved, enabling the visual encoder to possess the scale awareness and semantic alignment capabilities required for anomaly detection tasks. However, information from a single scale may still suffer from insufficient local representation or inadequate global structure awareness. Therefore, in this embodiment of the invention, a multi-view cue ensemble learning method is further employed to effectively fuse and learn information from different scale perspectives, enabling the CLIP model to possess the ability to perceive specific scales. Specifically, the multi-view cue ensemble learning method is used in the processes of calculating anomaly scores and anomaly localization maps in steps S112 and S114 above. Specifically, in this embodiment of the invention, the calculation of anomaly scores and anomaly localization maps includes the following processes:

[0085] (1) Calculate the cosine similarity between the text embedding representation and the scale visual features at each preset scale to obtain the anomaly localization map at each preset scale;

[0086] Among them, the anomaly location map meets the preset constraints;

[0087] Specifically, in this embodiment of the invention, the text embedding representation at each preset scale is used... The text is then represented. This process involves calculating the aforementioned text embedding representation. Scale visual features The cosine similarity is used to obtain the anomaly localization map, as shown in the following formula:

[0088]

[0089] Where M i The anomaly localization map is a pre-defined anomaly localization map at a predetermined scale i. Furthermore, the anomaly localization map obtained in this process satisfies pre-defined constraints through a loss function L. focal and L dice The constraints are combined, and the specific constraints are as follows:

[0090]

[0091] in,

[0092]

[0093] Where α is a hyperparameter that is set manually. and Let M represent the anomaly localization map and the normal localization map at the i-th preset scale, respectively. The normal localization map and the anomaly localization map are complementary, and M... y For the real label, γ is a hyperparameter that is set manually.

[0094] (2) Sort the local visual features at each preset scale according to the degree of anomaly of each anomaly localization map, and select the preset number of local visual features with the highest degree of anomaly.

[0095] (3) Cluster the local visual features with the highest degree of abnormality to obtain the cluster centers of the abnormal regions;

[0096] Specifically, the processes (2) and (3) above can obtain the cluster center of the j-th abnormal region at the i-th preset scale. Specifically, it can be based on the anomaly localization map M i The degree of abnormality in local visual features The top 5N local visual features with the highest degree of anomaly are sorted, and then K-means clustering is used to obtain N cluster centers for the anomaly regions. Where N is a pre-set hyperparameter.

[0097] (4) The cluster centers of abnormal regions are enhanced by using the global visual features of the image to be detected, and the enhanced cluster centers of abnormal regions are obtained.

[0098] In this process, the global visual features provide corresponding global information for each abnormal region cluster center, and the enhancement process here is actually to combine the global visual features with the abnormal region cluster centers obtained in (3) above. The specific formula is expressed as follows:

[0099]

[0100] in, For the enhanced abnormal region cluster centers, α is a hyperparameter that adjusts the degree of fusion between global information and abnormal region cluster centers. α is usually set to 0.5 by default, or can be set according to actual usage. This embodiment of the invention does not impose any restrictions on this.

[0101] (5) Calculate the cosine similarity between the enhanced abnormal region cluster center and the text embedding representation at each preset scale to obtain the abnormal score at the preset scale.

[0102] Specifically, in this process, the enhanced regional anomaly cluster centers are... Text embedding under preset scale Calculate the cosine similarity, which is the anomaly score S at the preset scale. i A higher anomaly score indicates a greater degree of anomaly. Subsequently, based on a global loss function, the prediction results are aggregated across all m preset scales, calculated using the following formula:

[0103]

[0104] L focal (S i ,S y )=-αS y (1-S i ) γ logS i -(1-α)(1-S y )S i γlog(1-S i )

[0105] Where α is a manually set hyperparameter, m represents the number of preset scales, and S i S represents the anomaly score at scale i. y ∈{0,1} represents the real labels, and γ is a hyperparameter that is set manually.

[0106] Furthermore, when performing fusion processing on the anomaly scores and anomaly localization maps obtained at the preset scales in the above process, the anomaly scores at each preset scale can be aggregated to obtain a global anomaly score; and the anomaly localization maps at each preset scale can be aggregated to obtain a global anomaly localization map; thus generating a detection result containing the global anomaly score and the global anomaly localization map.

[0107] Specifically, the final anomaly score S and anomaly location M can be obtained by averaging the anomaly location maps and anomaly scores of all preset scales. The final anomaly score S is the global anomaly score mentioned above in the embodiments of the present invention, and the anomaly location M is the location of the global anomaly location map mentioned above.

[0108] In summary, the zero-shot anomaly detection method provided in this invention employs hierarchical multimodal cue tuning to construct multi-scale adaptive text cues, thereby effectively enhancing the CLIP model's ability to represent anomaly patterns across scales and its anomaly localization capabilities. Simultaneously, a dual-stream cue tuning strategy is combined to jointly fine-tune the text encoder and visual encoder, enabling the visual encoder to effectively capture visual representations relevant to the anomaly detection task. Furthermore, a multi-view cue combination learning mechanism can be introduced during training to further improve the CLIP model's ability to perceive anomaly features from different scale perspectives. Compared to existing technologies, the zero-shot anomaly detection method provided in this invention has significant advantages in improving detection accuracy and model generalization ability, and can more effectively adapt to complex and ever-changing specific domains, meeting practical application needs.

[0109] Furthermore, embodiments of the present invention also provide a zero-sample anomaly detection device, such as... Figure 3 The diagram shows a structural schematic of a zero-sample anomaly detection device, which includes:

[0110] Extraction module 30 is used to acquire the image to be detected and extract the visual features of the image to be detected using the visual encoder of the pre-trained CLIP model. The visual features include: global visual features and local visual features at multiple preset scales.

[0111] Enhancement module 31 is used to visually enhance the visual features to obtain enhanced visual features suitable for a preset scale;

[0112] The prompt module 32 is used to inject the enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts containing multiple preset scales;

[0113] Feature module 33 is used to inject the adaptive text prompt into the text encoder of the CLIP model, encode the adaptive text prompt through the text encoder, and output the text embedding representation at each preset scale; and to map the adaptive text prompt to the visual space using a preset adapter model to obtain a visual prompt, and input the visual prompt into the local visual features at each preset scale of the image to be detected to obtain the scale visual features at each preset scale:

[0114] The calculation module 34 is used to calculate the corresponding anomaly score and anomaly localization map in a preset scale space based on the text embedding representation and the scale visual features, respectively.

[0115] The detection module 35 is used to perform fusion processing on the anomaly scores and anomaly localization maps at all scales, and output the detection results corresponding to the image to be detected, wherein the detection results include global anomaly scores and global anomaly localization maps.

[0116] The zero-sample anomaly detection device provided in this embodiment of the invention has the same technical features as the zero-sample anomaly detection method provided in the above embodiments, so it can also solve the same technical problems and achieve the same technical effects.

[0117] Furthermore, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0118] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0119] Furthermore, embodiments of the present invention also provide a schematic diagram of the structure of an electronic device, such as... Figure 4 The diagram shows the structure of the electronic device, which includes a processor 41 and a memory 40. The memory 40 stores computer-executable instructions that can be executed by the processor 41, and the processor 41 executes the computer-executable instructions to implement the above-described method.

[0120] exist Figure 4 In the illustrated embodiment, the electronic device further includes a bus 42 and a communication interface 43, wherein the processor 41, the communication interface 43, and the memory 40 are connected via the bus 42.

[0121] The memory 40 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 43 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 42 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 42 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0122] Processor 41 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 41 or by software instructions. Processor 41 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor 41 reads the information in the memory and uses its hardware to complete the aforementioned method.

[0123] The computer program product of the zero-sample anomaly detection method, apparatus, and electronic device provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0125] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0126] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0127] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0128] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A zero-sample anomaly detection method, characterized in that, The method includes: The image to be detected is acquired, and the visual features of the image to be detected are extracted using the visual encoder of the pre-trained CLIP model. The visual features include global visual features and local visual features at multiple preset scales. The visual features are visually enhanced to obtain enhanced visual features suitable for a preset scale; The enhanced visual features are injected into a pre-configured learnable text prompt template to generate adaptive text prompts with multiple preset scales; The adaptive text prompt is injected into the text encoder of the CLIP model, the text encoder encodes the adaptive text prompt, and outputs the text embedding representation at each preset scale; and a preset adapter model is used to map the adaptive text prompt to the visual space to obtain a visual prompt, and the visual prompt is input into the local visual features at each preset scale of the image to be detected to obtain the scale visual features at each preset scale. Based on the text embedding representation and the scale visual features, the corresponding anomaly score and anomaly localization map are calculated in the preset scale space, respectively. The anomaly scores and anomaly localization maps of all preset scales are fused to output the detection result corresponding to the image to be detected, wherein the detection result includes a global anomaly score and a global anomaly localization map; The step of visually enhancing the visual features to obtain enhanced visual features suitable for a preset scale includes: The global visual features are mapped to a generalized scale space using a pre-built perceptron model to obtain the first enhanced features in that scale space. The first enhanced feature is further enhanced according to a pre-configured cross-attention mechanism to obtain the second enhanced feature; The second enhancement feature is superimposed on the first enhancement feature to obtain the enhanced visual feature; The step of further enhancing the first enhanced feature according to a pre-configured cross-attention mechanism to obtain the second enhanced feature includes: The first enhanced feature in this scale space is sequentially spliced ​​with the local visual features in multiple preset scales to obtain the spliced ​​feature; Using the first enhanced feature as the query vector, and the concatenated features as the key matrix and value matrix, the first enhanced feature is further enhanced using a pre-configured cross-attention algorithm to obtain the second enhanced feature.

2. The method according to claim 1, characterized in that, The steps of calculating the corresponding anomaly scores and anomaly localization maps in a preset scale space based on the text embedding representation and the scale visual features include: The cosine similarity between the text embedding representation and the visual features at each preset scale is calculated to obtain an anomaly localization map at each preset scale, wherein the anomaly localization map satisfies preset constraints. The local visual features at each preset scale are sorted according to the degree of anomaly in each anomaly localization map, and a preset number of local visual features with the highest degree of anomaly are selected. Clustering is performed on a preset number of local visual features with the highest degree of anomaly to obtain cluster centers for the abnormal regions; The cluster centers of the abnormal regions are enhanced by using the global visual features of the image to be detected, resulting in enhanced cluster centers of the abnormal regions. Calculate the cosine similarity between the enhanced abnormal region cluster center and the text embedding representation at each preset scale to obtain the abnormality score at the preset scale.

3. The method according to claim 1, characterized in that, The step of fusing the anomaly scores and anomaly localization maps at all scales to output the detection result corresponding to the image to be detected includes: The anomaly scores at each preset scale are aggregated to obtain a global anomaly score; and the anomaly location maps at each preset scale are aggregated to obtain a global anomaly location map. Generate detection results that include the global anomaly score and the global anomaly localization map.

4. The method according to claim 1, characterized in that, The method further includes: The adaptive text prompts are injected into a pre-configured learnable text prompt template to construct learnable text prompts; wherein, the learnable text prompt template includes a normal prompt template and an abnormal prompt template, and the adaptive text prompts are injected bit by bit into the learnable prompt portions of the normal prompt template and the abnormal prompt template to form normal prompts and abnormal prompts; Fine-tuned prompt features are constructed based on the enhanced visual features according to pre-configured parameters; wherein, the fine-tuned prompt features include: normal prompt features, abnormal prompt features, and adaptive prompt features; The normal prompt features are added bitwise to the learnable portion of the normal prompt, and the abnormal prompt features are added bitwise to the learnable portion of the abnormal prompt to obtain learnable parameters; Based on the learnable parameters, the text encoder is fine-tuned using adaptive cue features.

5. The method according to claim 4, characterized in that, The steps of injecting the adaptive text prompt into the text encoder of the CLIP model, encoding the adaptive text prompt through the text encoder, and outputting the text embedding representation at each preset scale include: The adaptive text prompt is injected into the text encoder after prompt fine-tuning. The adaptive text prompt is encoded by the fine-tuned text encoder, and the text embedding representation under each preset scale is output.

6. The method according to claim 1, characterized in that, The method further includes: The visual cues are injected into the visual encoder, and the visual encoder is then tuned based on the cues.

7. A zero-sample anomaly detection device, characterized in that, The device includes: The extraction module is used to acquire the image to be detected and extract the visual features of the image to be detected using the visual encoder of the pre-trained CLIP model. The visual features include: global visual features and local visual features at multiple preset scales. An enhancement module is used to visually enhance the visual features to obtain enhanced visual features suitable for a preset scale; The prompt module is used to inject the enhanced visual features into a pre-configured learnable text prompt template to generate adaptive text prompts containing multiple preset scales. The feature module is used to inject the adaptive text prompt into the text encoder of the CLIP model, encode the adaptive text prompt through the text encoder, and output the text embedding representation at each preset scale; and to map the adaptive text prompt to the visual space using a preset adapter model to obtain a visual prompt, and input the visual prompt into the local visual features at each preset scale of the image to be detected to obtain the scale visual features at each preset scale: The calculation module is used to calculate the corresponding anomaly score and anomaly localization map in a preset scale space based on the text embedding representation and the scale visual features, respectively. The detection module is used to fuse the anomaly scores and anomaly localization maps of all preset scales and output the detection results corresponding to the image to be detected, wherein the detection results include global anomaly scores and global anomaly localization maps. The step of visually enhancing the visual features to obtain enhanced visual features suitable for a preset scale includes: The global visual features are mapped to a generalized scale space using a pre-built perceptron model to obtain the first enhanced features in that scale space. The first enhanced feature is further enhanced according to a pre-configured cross-attention mechanism to obtain the second enhanced feature; The second enhancement feature is superimposed on the first enhancement feature to obtain the enhanced visual feature; The step of further enhancing the first enhanced feature according to a pre-configured cross-attention mechanism to obtain the second enhanced feature includes: The first enhanced feature in this scale space is sequentially spliced ​​with the local visual features in multiple preset scales to obtain the spliced ​​feature; Using the first enhanced feature as the query vector, and the concatenated features as the key matrix and value matrix, the first enhanced feature is further enhanced using a pre-configured cross-attention algorithm to obtain the second enhanced feature.

8. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described in any one of claims 1-6.

Images