Abnormality detection method and device

Abstract

The invention discloses an anomaly detection method and equipment. The method comprises the following steps: acquiring image data of a to-be-detected object; inputting the image data into an image encoder to extract corresponding global features and local features; inputting the global feature into a partial derivative sensing network module to obtain a global normal bias vector and a global abnormal bias vector; the local features are input to a partial derivative sensing network module after being subjected to space flattening, so that a local normal offset vector and a local abnormal offset vector are obtained; determining a normal feature vector based on a preset normal text prompt, the global normal bias vector, the local normal bias vector and a text encoder; determining an abnormal feature vector based on a preset abnormal text prompt, the local abnormal offset vector, the global normal offset vector and a text encoder; and determining the comprehensive anomaly classification of the object to be detected based on the global feature, the normal feature vector and the anomaly feature vector. The generalization ability of the anomaly detection model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of image anomaly detection, and in particular to an anomaly detection method and device. Background Art

[0002] In the industrial production process, anomaly detection (defect detection) is a key task to ensure product quality, improve production efficiency and reduce economic losses.

[0003] Traditional industrial anomaly detection methods primarily rely on rule-based systems or supervised deep learning models. Their effectiveness relies on extensive labeled data and requires retraining for different scenarios. This results in insufficient diversity and generalization of anomaly detection models for practical applications. Even with limited labeled data, traditional methods struggle to cover the complex and diverse anomaly types found in real-world scenarios (such as unknown defects and dynamic environmental disturbances). This leads to significant performance degradation when encountering unseen anomalies or migrating across scenarios. For example, a defect detection model designed for one product category cannot be directly applied to other categories, requiring repeated labeling and training, severely hindering the advancement of industrial intelligence. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an anomaly detection method and device to solve the problem of insufficient generalization ability of anomaly detection models in the prior art.

[0005] In order to achieve the above objectives, the present application provides, in a first aspect, a method for detecting anomalies, comprising: Acquiring image data of the object to be detected; Input the image data into the image encoder to extract the corresponding global features and local features; Input the global features into the partial derivative perception network module to obtain the global normal bias vector and the global abnormal bias vector; The local features are spatially flattened and then input into the partial derivative perception network module to obtain the local normal bias vector and the local abnormal bias vector; determining a normal feature vector based on a preset normal text prompt, a global normal bias vector, a local normal bias vector, and a text encoder; Determining an abnormal feature vector based on a preset abnormal text prompt, a local abnormal bias vector, a global normal bias vector, and a text encoder; A comprehensive abnormality classification of the object to be detected is determined based on the global features, normal feature vectors and abnormal feature vectors.

[0006] In an embodiment of the present application, the partial derivative perception network module includes multiple multi-layer perceptron modules with the same structure; each multi-layer perceptron module includes a linear layer, an activation layer and a linear transformation layer connected in sequence; the linear layer is used to receive global features or local features, and realize dimensionality reduction projection through a first parameter matrix and a first preset bias term; the activation layer is used to apply a nonlinear activation function to the features after dimensionality reduction; the linear transformation layer is used to map the activated hidden layer features to the target space through a second parameter matrix and a second preset bias term, and generate a normal bias vector with the same dimension as the preset normal text prompt, and an abnormal bias vector with the same dimension as the preset abnormal text prompt.

[0007] In an embodiment of the present application, the step of determining a normal feature vector based on a preset normal text prompt, a global normal bias vector, a local normal bias vector and a text encoder includes: determining a normal enhancement prompt vector based on the preset normal text prompt, the global normal bias vector and the local normal bias vector; and inputting the normal enhancement vector into the text encoder to determine a normal feature vector.

[0008] In an embodiment of the present application, the step of determining an abnormal feature vector based on a preset abnormal text prompt, a local abnormal bias vector, a global abnormal bias vector and a text encoder includes: determining an abnormal enhancement prompt vector based on the preset abnormal text prompt, the local abnormal bias vector and the global abnormal bias vector; and inputting the abnormal enhancement vector into the text encoder to determine the abnormal feature vector.

[0009] In the embodiment of the present application, the step of determining the comprehensive abnormality classification of the object to be detected based on the global features, the normal feature vector and the abnormal feature vector includes: According to formula (1), the global feature and the normal feature vector determine the corresponding first cosine similarity: (1) in, is the first cosine similarity; is a global feature; is the normal eigenvector; According to formula (2), the global features and the abnormal feature vectors determine the corresponding second cosine similarity: (2) in, is the second cosine similarity; is a global feature; is the abnormal feature vector; According to formula (3), formula (4), the first cosine similarity and the second cosine similarity, the probability of comprehensive abnormal classification is determined: (3) (4) in, is the normal probability in the comprehensive abnormal classification; is the abnormal probability in the comprehensive abnormal classification; is the first cosine similarity; is the second cosine similarity; Represents the temperature hyperparameter, which is used to control the sharpness of the probability distribution.

[0010] In an embodiment of the present application, the method further includes: determining a normal similarity map between local features and normal feature vectors; determining an abnormal similarity map between local features and abnormal feature vectors; performing upsampling operations on the abnormal similarity map and the normal similarity map respectively to obtain an abnormal location map of the object to be detected; and locating the abnormal part of the object to be detected according to the abnormal location map.

[0011] In the embodiment of the present application, the normal similarity graph and the abnormal similarity graph are determined according to formula (5) and formula (6): (5) (6) in, It is the height of local features; is the width of the local feature; are the indices of the height and width directions of the local features respectively; It is the feature vector obtained at the specified position corresponding to the index on the local feature; is the normal eigenvector; is the abnormal feature vector; is the normal similarity map at the specified position corresponding to the index on the local feature; is the abnormal similarity graph of the specified position corresponding to the index on the local feature; is the normal similarity map of local features; is the abnormal similarity graph of local features; And / or, determine the anomaly location map according to formula (7): (7) in, It is an abnormal location map; and They represent upsampling operations on abnormal similarity maps and normal similarity maps of local features respectively.

[0012] The second aspect of the present application provides an anomaly detection method, which includes: obtaining image data of an object to be detected; inputting the image data into an image encoder to extract corresponding global features; inputting the global features into a global processing unit of a biased perception network module to obtain a global normal bias vector and a global abnormal bias vector; determining a global normal feature vector based on a preset normal text prompt, a global normal bias vector and a text encoder; determining a global abnormal feature vector based on a preset abnormal text prompt, a global abnormal bias vector and a text encoder; and determining an abnormal classification of the object to be detected based on the global features, the global normal feature vector and the global abnormal feature vector.

[0013] A third aspect of the present application provides an abnormality detection method, which includes: obtaining image data of an object to be detected; inputting the image data into an image encoder to extract corresponding local features; inputting the local features into a local processing unit of a biased perception network module to obtain a local normal bias vector and a local abnormal bias vector; determining a local normal feature vector based on a preset normal text prompt, a local normal bias vector and a text encoder; determining a local abnormal feature vector based on a preset abnormal text prompt, a local abnormal bias vector and a text encoder; determining an abnormal position of the object to be detected based on the local features, the local normal feature vector and the local abnormal feature vector; and / or, inputting the image data into an image encoder to further extract corresponding global features; determining the abnormal classification of the object to be detected based on the global features, the local normal feature vector and the local abnormal feature vector.

[0014] A fourth aspect of the present application provides an anomaly detection device, comprising: a memory configured to store instructions; and a processor configured to call instructions from the memory and implement the above method when executing the instructions.

[0015] A fifth aspect of the present application provides a storage medium, on which instructions are stored for enabling a machine to execute the above method.

[0016] The above technical solution first acquires image data of the object to be detected. The image data is input into an image encoder to extract corresponding global and local features. The global features are then input into a partial derivative perceptual network module to obtain a global normal bias vector and a global abnormal bias vector. The local features are spatially flattened and then input into the partial derivative perceptual network module to obtain local normal bias vectors and local abnormal bias vectors. Specifically, the partial derivative perceptual network module converts the four bias vectors (visual embeddings) from the image encoder to the textual prompt into input instance-specific biases. A normal feature vector is then determined based on a preset normal text prompt, a global normal bias vector, a local normal bias vector, and the text encoder. An abnormal feature vector is then determined based on a preset abnormal text prompt, a local abnormal bias vector, a global normal bias vector, and the text encoder. Finally, a comprehensive abnormal classification of the object to be detected is determined based on the global features, the normal feature vector, and the abnormal feature vector. These specific biases, combined with the previously fixed textual prompt, enrich cross-modal interactions, enabling the anomaly detection model to better adapt to new data distributions and categories, significantly improving its generalization ability in detecting anomalies.

[0017] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present application but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings: Figure 1 A flowchart of an anomaly detection method according to an embodiment of the present application is schematically shown; Figure 2 A schematic diagram of a visual bias extraction network design based on a multi-layer perceptron according to an embodiment of the present application is shown; Figure 3 A schematic diagram of zero-shot classification based on CLIP according to an embodiment of the present application is shown; Figure 4 The schematic diagram of the visual prior cue learning framework based on the multi-layer perceptron is shown; Figure 5 The following schematically shows a flow chart of an anomaly detection method according to an embodiment of the present application; Figure 6 A flowchart of an anomaly detection method according to an embodiment of the present application is schematically shown; Figure 7 A flowchart of an anomaly detection method according to an embodiment of the present application is schematically shown; Figure 8Schematically shows a visual comparison diagram of anomaly detection according to an embodiment of the present application and the AnomalyCLIP method; Figure 9 Schematically shows a visualization diagram of attention weights corresponding to two semantics according to an embodiment of the present application; Figure 10 Schematically shows a multi-level feature ablation experiment result diagram according to an embodiment of the present application; Figure 11 The figure schematically shows a structural diagram of an abnormality detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0020] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the various components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0021] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0022] Figure 1 The following schematically shows a flow chart of an abnormality detection method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides an anomaly detection method, which may include the following steps.

[0023] Step 101: Acquire image data of the object to be detected.

[0024] Step 102: Input the image data into an image encoder to extract corresponding global features and local features.

[0025] In an embodiment of the present application, the object to be detected may be a physical object to be detected in an industrial scene (such as a battery cluster, a circuit board, a mechanical component), which needs to be monitored for deviations from a normal state. Image data may refer to a two-dimensional visual information carrier collected by an optical imaging device, which is used to characterize the apparent state of the object to be detected. The image encoder is an image encoder in CLIP (a pre-trained neural network model for matching images and texts), a feature extraction module based on a deep neural network, which is used to map the original image into a high-dimensional semantic feature space. Global features refer to abstract feature vectors (such as size, color distribution) that characterize the overall state of the object to be detected, reflecting macroscopic consistency. Local features refer to spatial feature tensors that characterize the detailed structure of the object to be detected, retaining pixel-level spatial relationships for microscopic defect capture.

[0026] Step 103: Input the global features into the partial derivative perception network module to obtain a global normal bias vector and a global abnormal bias vector.

[0027] Step 104: Flatten the local features in space and input them into the partial derivative perception network module to obtain a local normal bias vector and a local abnormal bias vector.

[0028] In an embodiment of the present application, the biased perception network module may include multiple multi-layer perceptron modules, which may refer to learnable nonlinear transformation modules for adapting visual features to text semantic space. The global / local processing unit refers to a branch module in the multi-layer perceptron that processes global features and local features respectively, and outputs a bias vector to modulate text semantics. Spatial flattening refers to the operation of converting local features from a two-dimensional image structure to a one-dimensional sequence form, which provides vectorization convenience for subsequent similarity calculation and positioning processing.

[0029] Step 105: Determine a normal feature vector based on a preset normal text prompt, a global normal bias vector, a local normal bias vector, and a text encoder.

[0030] Step 106: Determine an abnormal feature vector based on the preset abnormal text prompt, the local abnormal bias vector, the global normal bias vector, and the text encoder.

[0031] In an embodiment of the present application, the normal / abnormal feature vector may refer to a semantic description vector that integrates a visual bias vector and a preset text prompt, representing the quantitative feature benchmarks of "normal" and "abnormal", respectively.

[0032] Step 107: Determine a comprehensive abnormality classification of the object to be detected based on the global features, the normal feature vector, and the abnormal feature vector.

[0033] In the embodiment of the present application, comprehensive anomaly classification refers to a probabilistic decision result based on the similarity comparison between global features and text features, and outputs the confidence level of whether the object to be detected belongs to the normal or abnormal category.

[0034] Through the above technical solution, global and local dual-scale features are first extracted, enabling the anomaly detection model to simultaneously learn macroscopic structural regularities and microscopic detail patterns, avoiding overfitting to features at specific training data scales and laying the foundation for improved generalization. Furthermore, the pre-set text prompt semantics are dynamically modulated through visual bias vectors (global normal bias vector, global abnormal bias vector, local normal bias vector, and local abnormal bias vector), adapting fixed prompts (such as "normal") to the current visual context, eliminating the semantic gap between traditional text prompts and unknown scenes. Finally, cross-modal similarity decisions are made based on global features and dynamically generated feature vectors (normal feature vector and abnormal feature vector), freeing anomaly determination from the decision boundary of the closed training set and transforming it into an open-domain semantic matching problem. Even when faced with defect types or environmental interference not covered by the training data, the model can still accurately classify the detected object using global features and normal / abnormal feature vectors. This breaks through the traditional model's reliance on data distribution and achieves zero-shot generalization transfer for industrial multi-morphic anomaly scenarios, thereby improving the model's generalization capability.

[0035] Figure 2 Schematically shows a visual bias extraction network design diagram based on a multi-layer perceptron according to an embodiment of the present application; Figure 2 As shown, in an embodiment of the present application, the partial derivative perception network module includes a plurality of multi-layer perceptron modules with the same structure; each multi-layer perceptron module includes a linear layer, an activation layer and a linear transformation layer connected in sequence; the linear layer is used to receive global features or local features, and realize dimensionality reduction projection through a first parameter matrix and a first preset bias term; the activation layer is used to apply a nonlinear activation function to the features after dimensionality reduction; the linear transformation layer is used to map the activated hidden layer features to the target space through a second parameter matrix and a second preset bias term, and generate a normal bias vector with the same dimension as the preset normal text prompt, and an abnormal bias vector with the same dimension as the preset abnormal text prompt.

[0036] In an embodiment of the present application, the biased perception network module (BiasNet, Bias-Guided Perception Network) can be used to extract visual biases of global features and local features. Specifically, BiasNet can convert the four visual embeddings (global normal bias vector, global abnormal bias vector, local normal bias vector and local abnormal bias vector) from the image encoder to the text prompt as input instance-specific biases. These instance-level biases fuse the originally fixed prompts to enrich cross-modal interactions to better adapt to data distributions and categories that the model has not seen, greatly improving the generalization ability of the model to detect anomalies, thereby improving the discriminability and accuracy of anomaly detection. Each multi-layer perceptron module adopts a two-layer fully connected structure: the first layer performs feature dimension transformation ( ), through the first parameter matrix and the first preset bias term Implement dimensionality reduction projection; then apply ReLU nonlinear activation function To enhance the model's expressive power; the last layer passes the second parameter matrix and the second preset bias term Map the hidden layer features to the target space and generate a bias vector aligned with the text prompt dimension. represents the channel dimension of the original text hint embedding. In particular, for local features Due to the spatial structure characteristics of , a tensor flattening operation needs to be performed before inputting BiasNet: This process not only preserves the spatial correlation of the feature map, but also meets the dimensionality requirements of the fully connected layer for the input form. The mathematical expression of each BiasNet module can be formalized as: in Represents various prior information extracted from image features, which can be used to enhance the original text prompts. According to different inputs and different uses, it can be subdivided into Four biases. represents the input image features, This design allows the model to adaptively extract complementary prior knowledge from visual features of different granularities during training, thereby enhancing textual cues through vector superposition.

[0037] In an embodiment of the present application, the step of determining a normal feature vector based on a preset normal text prompt, a global normal bias vector, a local normal bias vector and a text encoder may include: determining a normal enhancement prompt vector based on the preset normal text prompt, the global normal bias vector and the local normal bias vector; and inputting the normal enhancement vector into the text encoder to determine a normal feature vector.

[0038] In the embodiments of this application, the preset normal text prompt can fall under the category of "semantic prior" or "prompt engineering." It serves as a bridge in multimodal models, transforming "human-understandable semantic information" into "model-processable vector space information." Preset text prompts enable the model to have a preliminary and adjustable understanding of the concepts of "normal" and "abnormal." A global normal bias vector refers to a set of vectors representing the overall features of the object being detected (usually represented as the concatenation or pooling of features from one or more layers) obtained after an image encoder (such as a convolutional neural network or visual transformer) processes the entire image. Based on these "overall features," a lightweight network module (such as a fully connected layer or MLP) generates a "bias indicator of the current image's normal state," i.e., a "global normal bias vector." A local normal bias vector refers to the "local feature map" (typically H×W×C (height H, width W, number of channels C) that the image encoder retains alongside the "global features" output. At each pixel position in the feature map (corresponding to a certain spatial scale of the original image), a shared linear transformation or convolution operation generates a feature bias representing the normal state of that local region. The global normal bias vector tells the model what the overall image should look like (normal), while the local normal bias vector tells the model what a specific region in the image should look like when normal.

[0039] In an embodiment of the present application, the normal enhancement prompt vector may refer to first mapping the "preset normal text prompt" into the initial "text embedding vector" in the dictionary (vocabulary) or pre-trained word embedding layer of the text encoder; then performing certain fusion operations (such as vector weighted addition, concatenation and then MLP, or element-by-element weighting, etc.) with the "global normal bias vector" and "local normal bias vector" calculated on the image side to obtain a new vector, which is the "normal enhancement prompt vector". The normal feature vector may refer to the output vector finally obtained after taking the above-mentioned "normal enhancement prompt vector" as input and passing it through the text encoder (usually several layers of Transformer or multiple layers of fully connected layers). It can be regarded as "a semantic prototype corresponding to the normal state of the current image in the multimodal embedding space".

[0040] In an embodiment of the present application, the step of determining an abnormal feature vector based on a preset abnormal text prompt, a local abnormal bias vector, a global abnormal bias vector and a text encoder may include: determining an abnormal enhancement prompt vector based on the preset abnormal text prompt, the local abnormal bias vector and the global abnormal bias vector; and inputting the abnormal enhancement vector into the text encoder to determine the abnormal feature vector.

[0041] In the embodiment of the present application, the preset abnormal text prompt, local abnormal bias vector, and global abnormal bias vector correspond to the above-mentioned preset normal text prompt, local normal bias vector, and global normal bias vector. Their respective meanings can be inferred, and their functions are almost the same. Therefore, they will not be repeated here.

[0042] In an embodiment of the present application, the step of determining the comprehensive abnormal classification of the object to be detected based on the global features, the normal feature vectors, and the abnormal feature vectors may include: determining the corresponding first cosine similarity according to formula (1), the global features, and the normal feature vectors: (1) in, is the first cosine similarity; is a global feature; is the normal eigenvector; According to formula (2), the global features and the abnormal feature vectors determine the corresponding second cosine similarity: (2) in, is the second cosine similarity; is a global feature; is the abnormal feature vector; According to formula (3), formula (4), the first cosine similarity and the second cosine similarity, the probability of comprehensive abnormal classification is determined: (3) (4) in, is the normal probability in the comprehensive abnormal classification; is the abnormal probability in the comprehensive abnormal classification; is the first cosine similarity; is the second cosine similarity; Represents the temperature hyperparameter, which is used to control the sharpness of the probability distribution.

[0043] In an embodiment of the present application, the method may further include: determining a normal similarity map between local features and normal feature vectors; determining an abnormal similarity map between local features and abnormal feature vectors; performing upsampling operations on the abnormal similarity map and the normal similarity map respectively to obtain an abnormal location map of the object to be detected; and locating the abnormal part of the object to be detected according to the abnormal location map.

[0044] In the embodiments of the present application, the abnormal similarity map and the normal similarity map may refer to a "normalized similarity heat map at the local region level." In multimodal detection, it is an application form of "feature-prototype matching," used to characterize the "degree of match between each spatial location and the abnormal concept / normal concept in the embedding space." The upsampling operation may refer to the fact that the resolution H×W of the local feature map is generally much smaller than that of the original image (for example, if the original image is 512×512, the feature map may only be 32×32 or 16×16), so the resulting similarity map is also low-resolution. In order to map the similarity distribution back to the original image space, spatial interpolation of the similarity map is required. After obtaining an "anomaly localization map" with the same resolution as the original image, a threshold is usually set on the map. Pixels above the threshold are considered "suspected defect pixels." Connected component analysis can then be performed to obtain a series of bounding boxes or contours of the defective areas. Finally, it can be marked on the original image using a rectangular frame, segmentation mask or hot zone coverage.

[0045] In the embodiment of the present application, the normal similarity graph and the abnormal similarity graph are determined according to formula (5) and formula (6): (5) (6) in, It is the height of local features; is the width of the local feature; are the indices of the height and width directions of the local features respectively; It is the feature vector obtained at the specified position corresponding to the index on the local feature; is the normal eigenvector; is the abnormal feature vector; is the normal similarity map at the specified position corresponding to the index on the local feature; is the abnormal similarity graph of the specified position corresponding to the index on the local feature; is the normal similarity map of local features; is the abnormal similarity graph of local features; And / or, determine the anomaly location map according to formula (7): (7) in, It is an abnormal location map; and Indicates that the abnormal similarity map and the normal similarity map of the local features are up-sampled.

[0046] The following is an example: Figure 3 FIG. 4 shows a schematic diagram of zero-sample classification based on CLIP according to an embodiment of the present application; FIG. Figure 3 As shown, CLIP can handle many zero-shot tasks without additional training, significantly reducing the time and data costs of task customization. Here, we use image classification as an example to illustrate the CLIP-based zero-shot task workflow. First, we construct a text input using a text prompt template and the name of each category. The text prompt here is also called a prompt. Its general form is as follows:

[0047] The "[class]" can be replaced with the name of each category in the classification task (plane, dog, cat, bird). Assume there are a total of categories need to be classified, then a A text prompt can be obtained through the CLIP text encoder text feature vector . Input the image into CLIP's image encoder to obtain the corresponding image feature vector , then we can calculate the similarity between the image and each text label, and then use the softmax function for normalization to get the classification probability of the image belonging to each category, as shown in the following formula:

[0048] in Is an adjustable temperature parameter used to control the smoothness of softmax. It represents the classification probability of the image output by CLIP belonging to each category. Taking the maximum value completes the classification of the input image.

[0049] Figure 4 The schematic diagram shows the visual prior cue learning framework based on the multi-layer perceptron; Figure 4 As shown, in one embodiment, the given width and height are Input image , first pass the parameter-frozen CLIP image encoder Extract global features and local features, where global features Refers to the image encoder The vector representation corresponding to the class label in the output contains global information and is often used for classification tasks; while the local features The image is divided into multiple image blocks and then input into the image encoder. The output result retains the information of the local space of the image and is often used for pixel-level tasks such as segmentation. The process of obtaining the two features can be expressed as follows:

[0050] In the formula represents the spatial resolution of the local feature map, is the channel dimension of the two features.

[0051] In order to fully exploit the multi-granularity visual prior, four sets of multi-layer perceptron modules with independent parameters are constructed, collectively referred to as the partial derivative perception network module BiasNet, which is represented by the symbol , processing global features and local features respectively:

[0052] in It is a spatial flattening operation used to match the input shape of BiasNet. Represents the bias vector representing normal and abnormal semantics learned from global features Represents the bias vector representing normal and abnormal semantics learned from local features, is the embedding dimension of the text prompt. The process of enhancing the text prompt with the learned bias vector can be expressed as follows:

[0053]

[0054] In the formula For the initial text prompt, are the enhanced prompt vectors, which are fed into the text encoder of CLIP You can get the semantic embedding:

[0055]

[0056] in They are text feature vectors representing normal and abnormal semantics respectively. The results of abnormal classification can be obtained by calculating the similarity with different image features.

[0057] For the abnormal classification task, we need to get a binary probability distribution to indicate whether the model judges the input image to be normal or abnormal. This binary probability distribution is obtained by calculating the cosine similarity between the global feature and the two text embeddings and normalizing them. Represent the probability of classifying the image as abnormal and normal, respectively, and the abnormal classification probability That is, the anomaly score can be calculated using the following formula:

[0058] Normal classification probability It can be calculated using the following formula:

[0059] in Represents the temperature hyperparameter, which is used to control the sharpness of the probability distribution and is usually taken directly This enables optimal performance when using CLIP primitive parameters. Represent the global eigenvectors and two text features The cosine similarity is calculated as follows:

[0060]

[0061] For the anomaly location task, we need to get an anomaly location map in the end. , used to indicate which areas in the image have abnormalities. Usually by local features The cosine similarity is calculated by spatial position and then upsampled to obtain the local features. The similarity graph of the text features is represented as They can be calculated by the following formula:

[0062] in Represents the index in the height and width directions respectively, Indicates the feature vector obtained at the specified position on the local feature according to the index, and the similarity graph is obtained Afterwards, the spatial resolution of the similarity map is restored to the same size as the input image using an upsampling algorithm, and then the The final abnormality location map can be obtained by , the specific calculation process is as follows:

[0063] in Indicates the upsampling operation, which usually uses a bilinear interpolation algorithm with local continuity preservation properties. In the process of calculating the anomaly location map, not only the local features are considered and abnormal semantic features The similarity of local features is also used and normal semantic features The complementary operation of the similarity graph of is used to obtain another piece of abnormal location information, and the two are fused through the mean fusion strategy to obtain the abnormal location map , which can effectively alleviate the false positive problem that may arise from single-modal similarity calculation.

[0064] During the training of the anomaly detection model: It should be noted that the above anomaly detection model can be Figure 2 A visual bias extraction network design based on a multi-layer perceptron is shown and Figure 4 The visual prior cue learning framework diagram based on multi-layer perceptron is constructed as shown.

[0065] The parameters of both encoders in the CLIP model remain frozen. The optimization targets include the existing textual prompts and the multiple visual bias extraction networks designed in this section. The goal is to enable the visual bias extraction networks to accurately extract useful information from the two image features, thereby further improving the performance of anomaly classification and localization. Because the anomaly detection task consists of two subtasks, two loss functions are used during training to constrain the optimization direction of anomaly classification and anomaly localization, respectively.

[0066] For the anomaly classification task, the classification result of the model is The binary cross entropy loss function is used for constraint, and its calculation formula is as follows:

[0067] in, Indicates the number of samples in each batch; Indicates the The true label of the image, the abnormal image corresponds to , the normal image corresponds to In the actual calculation process, the true classification label can be obtained by taking the maximum value of the abnormal location label; and Respectively represent the model The abnormal and normal probability output of the input image satisfies The design of this loss function can optimize the anomaly classification task and make the overall performance of anomaly detection better.

[0068] For the anomaly localization task, it can be regarded as a semantic segmentation task. After calculating the anomaly localization map, it and the corresponding label are represented as This application uses the classic DiceLoss loss function to constrain the network optimization process, and its calculation formula is as follows:

[0069] in, represents the number of images in a batch, and W represent the height and width of the input image respectively, M represents the The abnormal location map corresponding to the image is located in The value of the position, Y represents the The value at position (ii) in the true label corresponding to the image. To ensure that the denominator is not zero, a smoothing variable 1 is added to both the numerator and denominator.

[0070] Finally, the loss function used for training is and A combination of:

[0071] The design of this loss function can take into account the optimization of both anomaly classification and anomaly localization, making the overall performance of anomaly detection better.

[0072] Figure 5 The following schematically shows a flow chart of an abnormality detection method according to an embodiment of the present application. Figure 5 As shown, an embodiment of the present application provides an anomaly detection method, which may include the following steps.

[0073] Step 501: Acquire image data of the object to be detected.

[0074] Step 502: Input the image data into an image encoder to extract corresponding global features.

[0075] Step 503: Input the global features into the partial derivative perception network module to obtain a global normal bias vector and a global abnormal bias vector.

[0076] Step 504: Determine a global normal feature vector based on a preset normal text prompt, a global normal bias vector, and a text encoder.

[0077] Step 505: Determine a global abnormal feature vector based on the preset abnormal text prompt, the global abnormal bias vector and the text encoder.

[0078] Step 506: Determine the abnormality classification of the object to be detected based on the global features, the global normal feature vector, and the global abnormal feature vector.

[0079] Through the above technical solution, the global visual features extracted by the image encoder are aligned with the preset "normal / abnormal" text prompts in the text encoder space to generate "global normal feature vectors" and "global abnormal feature vectors" with conditional semantic guidance. The similarity is then calculated with the original global features and classified. This allows for accurate judgment of whether the entire image has abnormalities without relying on large-scale pixel-level annotation. This not only improves the model's robustness to diverse appearance changes, but also has good interpretability and scalability.

[0080] Figure 6 The following schematically shows a flow chart of an abnormality detection method according to an embodiment of the present application. Figure 6 As shown, an embodiment of the present application provides an anomaly detection method, which may include the following steps.

[0081] Step 601: Acquire image data of the object to be detected.

[0082] Step 602: Input the image data into an image encoder to extract corresponding local features.

[0083] Step 603: Input the local features into the local processing unit of the partial derivative perception network module to obtain a local normal bias vector and a local abnormal bias vector.

[0084] Step 604: Determine a local normal feature vector based on a preset normal text prompt, a local normal bias vector, and a text encoder.

[0085] Step 605: Determine a local abnormality feature vector based on the preset abnormal text prompt, the local abnormality bias vector and the text encoder.

[0086] Step 606: Determine the abnormal position of the object to be detected based on the local features, the local normal feature vector, and the local abnormal feature vector.

[0087] By acquiring image data of the object to be detected and extracting local features, a biased perception network (BiasNet) module is introduced to generate local normal bias vectors and local abnormal bias vectors, respectively, thereby constructing semantic guidance information that is highly relevant to the current image context. Combined with preset text prompts, these bias vectors dynamically enhance the text semantics, making the local normal and abnormal feature vectors generated by the text encoder more targeted and discriminative. These feature vectors are then matched with spatially flattened local image features for position-level similarity, and an abnormality similarity map is further generated to accurately locate abnormal regions in the image. This scheme not only achieves anomaly localization in high-resolution images but also introduces dynamic adaptation between image and text semantics through a bias modeling mechanism. This effectively improves the model's semantic transfer and generalization capabilities across different image types and anomaly patterns, enabling it to maintain stable performance in anomaly detection tasks across scenarios and categories.

[0088] Figure 7 The following schematically shows a flow chart of an abnormality detection method according to an embodiment of the present application. Figure 7 As shown, an embodiment of the present application provides an anomaly detection method, which may include the following steps.

[0089] Step 701: Acquire image data of the object to be detected.

[0090] Step 702: Input the image data into an image encoder to extract corresponding local features.

[0091] Step 703: Input the local features into the local processing unit of the partial derivative perception network module to obtain a local normal bias vector and a local abnormal bias vector.

[0092] Step 704: Determine a local normal feature vector based on a preset normal text prompt, a local normal bias vector, and a text encoder.

[0093] Step 705: Determine a local abnormality feature vector based on the preset abnormal text prompt, the local abnormality bias vector and the text encoder.

[0094] Step 706: Input the image data into an image encoder to further extract corresponding global features; determine the abnormality classification of the object to be detected based on the global features, the local normal feature vectors and the local abnormal feature vectors.

[0095] By acquiring image data of the object to be detected, local and global features are extracted separately. The multi-layer perceptron module in the biased perception network (BiasNet) is then used to generate local normal and abnormal bias vectors. This guides the preset normal and abnormal text prompts to generate local normal and abnormal feature vectors that better align with the image semantics. Subsequently, the global features of the image are fused with these local semantic features to perform abnormality classification. This approach introduces a local bias mechanism, enabling text features to adapt to the richer and more diverse local patterns in the image. At the same time, it leverages global features to enhance the overall consistency of abnormality judgment. This significantly improves the model's adaptability to different detection objects and abnormal scenarios, as well as its ability to generalize across samples and domains, while maintaining discrimination accuracy.

[0096] Figure 8 Schematically shows a visual comparison diagram of anomaly detection according to an embodiment of the present application and the AnomalyCLIP method; Figure 9 Schematically shows a visualization diagram of attention weights corresponding to two semantics according to an embodiment of the present application; Figure 10 The diagram schematically shows the result of a multi-level feature ablation experiment according to an embodiment of the present application. In the prior art, although WinCLIP can preliminarily capture the semantic features of abnormal areas, its positioning results have a significant spatial diffusion phenomenon, which is specifically manifested in that the distribution range of abnormal scores is too wide and the peak response values ​​are generally low. This fuzzy positioning characteristic reflects that the method has insufficient semantic generalization ability in learning the normal / abnormal feature discrimination boundary. In comparison, the VAND method significantly improves the spatial concentration of its abnormal positioning heat map by introducing a visual-language collaborative attention mechanism, but there is still a problem that the score of the abnormal area is not high enough. The AnomalyCLIP method demonstrates sub-pixel positioning capability by constructing object-independent prompt words, and has a higher response to abnormal areas. However, this method shows a lot of noise in the abnormal positioning heat map, exposing the limitations of fixed text prompt templates in cross-category adaptability. The visual prior cue learning framework based on the multi-layer perceptron module proposed in this application extracts semantic information through visual features, enabling the model to more accurately describe abnormal / normal phenomena in each image. It not only accurately locates the area where the anomaly occurs and is highly responsive to anomalies, but also reduces the noise areas introduced by fixed text cues compared to AnomalyCLIP. This visual comparison result fully verifies the effectiveness of the above technical solution in zero-shot industrial anomaly detection tasks, and the generalization ability of the anomaly detection model based on this framework is significantly improved compared to existing technologies.

[0097] To verify the effectiveness of the above technical solution, we can conduct ablation experiments to verify the effectiveness of using global and local features to extract multi-level visual bias information and jointly use it to enhance text prompts. Using AnomalyCLIP as a benchmark, we then use BiasNet to extract visual biases for global features and local features, respectively. We design the following three sets of experimental schemes: a. Ours (G): Use global visual bias alone to enhance textual cues for anomaly detection; b. Ours (L): Using local visual bias alone to enhance textual cues for anomaly detection; c. Ours (G+L): We simultaneously exploit two visual biases to enhance textual cues for anomaly detection. The average value of each solution in groups a, b, and c on six datasets (including MVTec-AD, VisA, MPDD, BTAD, SDD, and DTD-Synthetic (DTD)) is taken as a measure of anomaly detection performance.

[0098] like Figure 8 、 Figure 9 and Figure 10 As shown in the figure, when global features are used alone for visual bias learning, compared to the AnomalyCLIP baseline method, although the image-level detection indicators AUROC and AP improve by 0.5 percentage points and 0.9 percentage points, respectively, the pixel-level indicators AUROC and AUPRO decrease by 0.4 percentage points and 1.0 percentage points. This shows that global features are difficult to achieve optimal anomaly detection results. The reason is that global features mainly carry image-level classification information, and the visual bias extracted from them is difficult to assist text prompts in accurately locating local anomaly areas, and may even introduce interference signals. When local features are used alone, the image-level AUROC and AP improve by 1.5 percentage points and 2.5 percentage points, respectively, compared to the baseline method, while the pixel-level AUROC and AUPRO both improve by 0.3 percentage points. This result significantly demonstrates the key role of local visual features in improving anomaly detection performance. The fine-grained information they contain can effectively enhance the text prompt's ability to represent anomaly areas. The solution that combines both features for visual bias learning achieves the best performance in terms of overall indicators, which fully verifies the effectiveness of the multi-layer perceptron module proposed in this application. By collaboratively integrating visual features of different granularities, we not only maintain global semantic consistency but also enhance the local anomaly representation capability, thereby achieving a significant improvement in overall detection performance.

[0099] Figure 11 Schematically shows a structural diagram of an abnormality detection device according to an embodiment of the present application; Figure 11 As shown, an embodiment of the present application further provides an abnormality detection device, the device comprising: Memory 1110 configured to store instructions; and The processor 1120 is configured to call instructions from the memory and implement the above method when executing the instructions.

[0100] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above method.

[0101] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0106] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0107] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0108] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0109] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for detecting anomalies, characterized in that: The method comprises: Acquiring image data of the object to be detected; Inputting the image data into an image encoder to extract corresponding global features and local features; Inputting the global features into a partial derivative perception network module to obtain a global normal bias vector and a global abnormal bias vector; Inputting the local features into the partial derivative perception network module after spatial flattening to obtain a local normal bias vector and a local abnormal bias vector; Determining a normal feature vector based on a preset normal text prompt, the global normal bias vector, the local normal bias vector, and a text encoder; Determining an abnormal feature vector based on a preset abnormal text prompt, the local abnormal bias vector, the global normal bias vector, and the text encoder; A comprehensive abnormality classification of the object to be detected is determined based on the global feature, the normal feature vector, and the abnormal feature vector.

2. The method according to claim 1, characterized in that The partial derivative perception network module includes multiple multi-layer perceptron modules with the same structure; each multi-layer perceptron module includes a linear layer, an activation layer and a linear transformation layer connected in sequence; the linear layer is used to receive the global features or the local features, and realize dimensionality reduction projection through a first parameter matrix and a first preset bias term; the activation layer is used to apply a nonlinear activation function to the features after dimensionality reduction; the linear transformation layer is used to map the activated hidden layer features to the target space through a second parameter matrix and a second preset bias term, and generate a normal bias vector with the same dimension as the preset normal text prompt, and an abnormal bias vector with the same dimension as the preset abnormal text prompt.

3. The method according to claim 1, characterized in that The step of determining a normal feature vector based on a preset normal text prompt, the global normal bias vector, the local normal bias vector, and a text encoder comprises: Determining a normal enhancement prompt vector according to the preset normal text prompt, the global normal offset vector and the local normal offset vector; The normal enhancement vector is input to the text encoder to determine the normal feature vector.

4. The method according to claim 1, wherein The step of determining an abnormal feature vector based on a preset abnormal text prompt, the local abnormal bias vector, the global abnormal bias vector, and the text encoder includes: Determining an abnormality enhancement prompt vector according to the preset abnormality text prompt, the local abnormality bias vector and the global abnormality bias vector; The anomaly enhancement vector is input to the text encoder to determine the anomaly feature vector.

5. The method according to claim 1, wherein The step of determining the comprehensive abnormality classification of the object to be detected based on the global feature, the normal feature vector and the abnormal feature vector comprises: According to formula (1), the global feature and the normal feature vector determine the corresponding first cosine similarity: （1） in, is the first cosine similarity; is the global feature; is the normal eigenvector; According to formula (2), the global feature and the abnormal feature vector, the corresponding second cosine similarity is determined: （2） in, is the second cosine similarity; is the global feature; is the abnormal feature vector; The probability of the comprehensive abnormal classification is determined according to formula (3), formula (4), the first cosine similarity and the second cosine similarity: （3） （4） in, is the normal probability in the comprehensive abnormal classification; is the abnormality probability in the comprehensive abnormality classification; is the first cosine similarity; is the second cosine similarity; Represents the temperature hyperparameter, which is used to control the sharpness of the probability distribution.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: determining a normal similarity graph between the local feature and the normal feature vector; determining an abnormal similarity graph between the local feature and the abnormal feature vector; performing upsampling operations on the abnormal similarity map and the normal similarity map respectively to obtain an abnormality location map of the object to be detected; The abnormality of the object to be detected is located according to the abnormality location map.

7. The method according to claim 6, characterized in that The method further comprises: According to formula (5) and formula (6), the normal similarity graph and the abnormal similarity graph are determined: （5） （6） in, is the height of the local feature; is the width of the local feature; are the indices of the height and width directions of the local feature respectively; is a feature vector obtained at a specified position corresponding to the index on the local feature; is the normal eigenvector; is the abnormal feature vector; is a normal similarity map at a designated position corresponding to the index on the local feature; is an abnormal similarity graph of a designated position corresponding to the index on the local feature; is the normal similarity map of the local features; is an abnormal similarity graph of the local features; And / or, determine the abnormality location map according to formula (7): （7） in, A map for locating the abnormality; and They respectively represent upsampling operations on the abnormal similarity map and the normal similarity map of the local features.

8. An anomaly detection method, characterized in that: The method comprises: Acquiring image data of the object to be detected; Inputting the image data into an image encoder to extract corresponding global features; Inputting the global features into a partial derivative perception network module to obtain a global normal bias vector and a global abnormal bias vector; Determining a global normal feature vector based on a preset normal text prompt, the global normal bias vector, and a text encoder; Determining a global abnormality feature vector based on a preset abnormal text prompt, the global abnormality bias vector and the text encoder; An abnormality classification of the object to be detected is determined based on the global feature, the global normal feature vector, and the global abnormal feature vector.

9. An anomaly detection method, characterized in that: The method comprises: Acquiring image data of the object to be detected; Inputting the image data into an image encoder to extract corresponding local features; Inputting the local features into a partial derivative perception network module to obtain a local normal bias vector and a local abnormal bias vector; Determining a local normal feature vector based on a preset normal text prompt, the local normal bias vector, and a text encoder; Determining a local abnormality feature vector based on a preset abnormal text prompt, the local abnormality bias vector and the text encoder; Determining the abnormal position of the object to be detected based on the local feature, the local normal feature vector and the local abnormal feature vector; And / or, the image data is input into an image encoder to further extract corresponding global features; and the abnormal classification of the object to be detected is determined based on the global features, the local normal feature vectors, and the local abnormal feature vectors.

10. An abnormality detection device, characterized in that: include: a memory configured to store instructions; as well as A processor is configured to call the instructions from the memory and implement the method according to any one of claims 1 to 9 when executing the instructions.

Images