A method and apparatus for few-sample anomaly detection based on visual-language joint learning

Abstract

This invention provides a few-sample anomaly detection method and apparatus based on visual-language joint learning, applied in the field of image detection technology. The method includes: generating anomaly prompt word templates based on preset anomaly type descriptions, the anomaly prompt word templates being used to represent potential anomaly types through text descriptions; constructing anomaly text prompts based on the anomaly prompt word templates and learnable context markers; inputting normal text prompts and anomaly text prompts, as well as multi-level image features of the query image, into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model; performing visual-guided anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and fusing the text-level anomaly score and the visual-level anomaly score to obtain the anomaly detection score of the query image. This invention can improve the accuracy of image anomaly detection under few-sample settings.

Description

Technical Field

[0001] This invention relates to the field of image detection technology, and in particular to a method and apparatus for detecting small-sample anomalies based on visual-language joint learning. Background Technology

[0002] Anomaly detection is a core component in ensuring product quality and improving production efficiency. In modern operating or production systems, due to complex process parameters and numerous influencing factors, even minor anomalies can lead to product defects. Timely and accurate identification of anomalies and defects in closed, open, and industrial environments, such as power systems, rail transportation, and manufacturing, not only helps reduce defect rates but also optimizes processes and improves overall efficiency.

[0003] In recent years, deep learning technology has made significant progress in the field of image recognition, driving the development of anomaly detection algorithms. However, most deep learning-based methods rely on large amounts of labeled data for training, which often makes it difficult to obtain sufficient training samples in real-world scenarios, especially with new products or new types of defects.

[0004] This shows that image detection methods in related technologies suffer from low anomaly detection accuracy due to the scarcity of training data. Summary of the Invention

[0005] This invention provides a few-sample anomaly detection method and apparatus based on visual-language joint learning, which addresses the shortcomings of low anomaly detection accuracy in existing image detection methods. It achieves improved image anomaly detection accuracy under few-sample settings by fusing text and image information through visual-language joint learning.

[0006] This invention provides a few-sample anomaly detection method based on visual-language joint learning, comprising the following steps: Generating anomaly prompt word templates based on preset anomaly type descriptions, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; constructing anomaly text prompts based on the anomaly prompt word templates and learnable context markers; inputting normal text prompts and the anomaly text prompts, along with multi-level image features of the query image, into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model; performing visual-guided anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and fusing the text-level anomaly score and the visual-level anomaly score to obtain the anomaly detection score of the query image.

[0007] According to a few-sample anomaly detection method based on visual-language joint learning provided by the present invention, the anomaly text prompt includes a preset text anomaly prompt and a learnable text anomaly prompt; the construction of the anomaly text prompt based on the anomaly prompt word template and the learnable context marker includes: concatenating the learnable context marker with a predefined anomaly description to obtain the preset text anomaly prompt, which is used to indicate a text prompt for a known anomaly type; and concatenating the learnable context marker with a learnable anomaly suffix marker to obtain a learnable text anomaly prompt, which is used to indicate a text prompt for an unknown anomaly type by adjusting the learnable anomaly suffix marker.

[0008] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided. The method involves inputting normal text prompts, the anomalous text prompts, and multi-level image features of a query image into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model. This includes: determining the similarity between the multi-level image features of the query image and the normal text prompts to obtain a normal similarity value; determining the similarity between the multi-level image features of the query image and the anomalous text prompts to obtain an anomaly similarity value; and generating a text-level anomaly score based on the difference between the normal similarity value and the anomaly similarity value.

[0009] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided. The method further includes: acquiring a training image dataset, wherein the training image dataset includes normal samples and anomaly samples; extracting multi-level image features based on the training image dataset; extracting text features through a text encoder based on normal text prompts and the anomaly text prompts to generate normal text prototypes and anomaly text prototypes; performing contrastive learning optimization based on the multi-level image features and the text prototypes to obtain an optimized similarity relationship; performing triple loss optimization based on the optimized similarity relationship to obtain an enhanced discrimination boundary; and dynamically adjusting the learnable parameters using an optimizer based on the enhanced discrimination boundary to obtain a trained prompt-guided anomaly detection model.

[0010] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided. The method involves performing comparative learning optimization based on the multi-level image features and the text prototype to obtain an optimized similarity relationship. This includes: maximizing the similarity between the multi-level image features of the normal sample and the normal text prototype, and minimizing the similarity between the normal sample and the abnormal text prototype; maximizing the similarity between the multi-level image features of the abnormal sample and the corresponding abnormal text prototype, and minimizing the similarity between the abnormal sample and the normal text prototype and other abnormal text prototypes. The method further involves performing triple loss optimization based on the optimized similarity relationship to obtain an enhanced discrimination boundary. This includes: forcing the similarity between the multi-level image features of the normal sample and the normal text prototype to be higher than the similarity with the abnormal text prototype; and forcing the similarity between the multi-level image features of the abnormal sample and the corresponding abnormal text prototype to be higher than the similarity with the normal text prototype.

[0011] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided. The method involves performing visually guided anomaly detection and generating a visual-level anomaly score based on multi-level image features and an image feature library of the query image. The method includes: calculating local anomaly scores by comparing the query image features with a feature memory, wherein the feature memory comparison uses the K-nearest neighbor algorithm to calculate the Euclidean distance between the query image features and the corresponding level features in the image feature library; calculating the Euclidean distance between the multi-level image features of the query image and the corresponding level features in the image feature library using the K-nearest neighbor algorithm to obtain local anomaly scores; calculating global anomaly scores by image-level residual learning based on the query image features and the image feature library; and generating a visual-level anomaly score by dynamically weighted fusion based on the local anomaly scores and the global anomaly scores, wherein the dynamic weighted fusion adaptively allocates local and global weights through a gating network.

[0012] This invention also provides a few-sample anomaly detection device based on visual-language joint learning, comprising the following modules: a generation module, used to generate anomaly prompt word templates based on preset anomaly type descriptions, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; a construction module, used to construct anomaly text prompts based on the anomaly prompt word templates and learnable context markers; a prompt guidance module, used to input normal text prompts and the anomaly text prompts, as well as multi-level image features of the query image, into a prompt guidance anomaly detection model to obtain a text-level anomaly score output by the prompt guidance anomaly detection model; a visual guidance module, used to perform visual guidance anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and a score fusion module, used to perform score fusion based on the text anomaly score and the visual anomaly score to obtain an anomaly detection score for the query image.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the few-sample anomaly detection method based on visual-language joint learning as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the few-sample anomaly detection method based on visual-language joint learning as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a few-sample anomaly detection method based on visual-language joint learning as described above.

[0016] This invention provides a few-sample anomaly detection method and apparatus based on visual-language joint learning. It generates anomaly prompt word templates based on preset anomaly type descriptions, enabling potential anomaly types to be represented through textual descriptions. It constructs anomaly text prompts based on the anomaly prompt word templates and learnable contextual tags, enhancing the adaptability and task relevance of text representation and improving the ability to capture few-sample anomaly types. Normal and anomalous text prompts, along with multi-level image features of the query image, are input into a prompt-guided anomaly detection model. By integrating textual and visual information, a text-level anomaly score is generated, effectively utilizing textual prompt guidance to identify anomaly patterns in the image. Visually guided anomaly detection is performed based on the multi-level image features of the query image and an image feature library. Visual-level anomaly scores are generated through comparative learning, strengthening the perception of anomalies in visual details and hierarchical features. Finally, the text-level and visual-level anomaly scores are fused, integrating visual and linguistic cues, to ultimately output the anomaly detection score of the query image, thereby improving the comprehensiveness, accuracy, and robustness of anomaly detection in few-sample scenarios. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the few-sample anomaly detection method based on joint visual-language learning provided by the present invention.

[0019] Figure 2 This is a structural diagram of the few-sample anomaly detection method based on visual-language joint learning provided by the present invention.

[0020] Figure 3 This is a schematic diagram of a module of the small sample anomaly detection device based on visual-language joint learning provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Achieving efficient anomaly detection under few-shot or even zero-shot conditions has become a key research focus. Against this backdrop, Vision-Language Models (VLMs), with their powerful semantic understanding and cross-modal alignment capabilities, offer a new approach to addressing the data scarcity problem under few-shot conditions.

[0024] Most existing methods only utilize normal samples for modeling, lacking explicit learning of anomaly semantics, resulting in the ability to detect anomalies but difficulty in achieving fine-grained classification. The method proposed in this invention fully leverages the limited number of anomaly samples available in industrial scenarios, utilizing the repeatability of anomaly patterns to enhance the model's discriminative ability. This not only achieves accurate detection but also supports fine-grained classification of anomaly types, significantly improving the model's practicality and generalization performance.

[0025] This invention proposes a few-sample anomaly detection method based on visual-language joint learning. It solves the problem of image anomaly detection in the case of few samples by fusing multiple information, which can effectively improve the detection accuracy.

[0026] Figure 1 This is a flowchart illustrating the few-sample anomaly detection method based on joint visual-language learning provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 101: Generate an exception prompt word template based on the preset exception type description, wherein the exception prompt word template is used to represent the potential exception type through text description.

[0027] In this embodiment of the invention, potential anomaly types in the target detection category are systematically analyzed by joint domain experts, such as surface scratches, size deviations, and color anomalies, and scenario modeling is carried out in combination with typical cases in actual production.

[0028] For each anomaly type, a detailed text description is provided, thereby constructing a corresponding cue word template. The anomaly cue word template is used to represent the potential anomaly type through textual description. For example, for the carpet category, the anomaly cue word template may include specific descriptions such as "with holes," "with stains," or "with metal contamination." This process ensures that the textual representation of anomaly types is domain-relevant and scalable, laying the foundation for subsequent visual-language joint learning.

[0029] Step 102: Construct an abnormal text prompt based on the abnormal prompt word template and learnable context markers.

[0030] In this embodiment of the invention, the text prompt construction includes the generation of normal text prompts and abnormal text prompts.

[0031] Normal text prompts are formed by concatenating a learnable context marker and a target object name. For example, a normal text prompt is formed by combining a learnable context marker of length EN (such as [N1][N2][N3]) with a target object name (such as [cable]).

[0032] Anomaly text prompts are divided into manual anomaly prompts and learnable anomaly prompts: manual anomaly prompts use the description in the anomaly prompt word template to concatenate the normal prompt prefix with the anomaly suffix (such as "with color spots"); learnable anomaly prompts dynamically capture unknown anomaly features by combining randomly initialized learnable anomaly suffix tags with the target object name.

[0033] This step enhances the adaptability of the text representation through learnable parameters, enabling the model to flexibly handle known and unknown anomaly types in small sample scenarios.

[0034] Step 103: Input the normal text prompts and abnormal text prompts, as well as the multi-level image features of the query image, into the prompt-guided anomaly detection model to obtain the text-level anomaly score output by the prompt-guided anomaly detection model.

[0035] The multi-level image features of the query image are extracted through a visual encoder, including low-level texture features and high-level semantic features.

[0036] The prompt-guided anomaly detection model optimizes the text prompt embedding space through a contrastive learning mechanism: maximizing the similarity between normal image features and normal text prompts, and minimizing their similarity with abnormal text prompts; simultaneously, maximizing the similarity between abnormal image features and abnormal text prompts, and minimizing their similarity with normal text prompts.

[0037] During training, the Triplet loss function is used to enhance the separation of positive and negative samples, ensuring that the model can clearly distinguish between normal and abnormal patterns in the embedding space.

[0038] Finally, by calculating the similarity difference between image features and text prompts, a text-level anomaly score is output, which reflects the assessment of the degree of anomaly of the query image under text guidance.

[0039] Step 104: Based on the multi-level image features and image feature library of the query image, perform visually guided anomaly detection and generate a visual-level anomaly score.

[0040] Visually guided anomaly detection includes local anomaly score calculation and global anomaly assessment.

[0041] The K-nearest neighbor algorithm is used to compare the local features of the query image with pre-built normal feature libraries and abnormal feature libraries, calculate the average Euclidean distance, and generate patch-level anomaly scores to capture minor defects or local texture anomalies.

[0042] By using an adapter to align the global features of the query image with the feature library, and calculating the overall image deviation through residual learning, an image-level anomaly score is generated, which is suitable for large-scale anomaly detection.

[0043] A dynamic weighting strategy is employed to fuse local and global scores, generating pixel-level anomaly heatmaps. The Focal loss function is used to address class imbalance and enhance the model's sensitivity to subtle anomalies. Visual-level anomaly scores integrate multi-level visual information, ensuring comprehensive detection.

[0044] Step 105: The scores are fused based on the text-level anomaly score and the visual-level anomaly score to obtain the anomaly detection score of the query image.

[0045] The fusion process uses a harmonic average formula to weight and integrate the global classification score of the image and the maximum value of the local heatmap.

[0046] Specifically, text-level anomaly scores come from the prompt guidance branch, while visual-level anomaly scores come from the visual guidance branch. A reverse weighting mechanism ensures that low scores in either branch will significantly reduce the fusion result, thereby avoiding misjudgment of a single modality.

[0047] For example, if the visual branch score is high but the text branch score is low, the fusion score will decrease, prompting the model to further verify its reliability. This score fusion strategy enhances the model's sensitivity to anomalies, ultimately outputting anomaly detection scores for the query image, achieving high-precision and robust anomaly detection under small sample conditions.

[0048] This invention utilizes semantic information guided by text prompts and features extracted by a visual encoder for joint modeling. By designing a loss function to optimize model parameters, and finally combining a dual-branch detection strategy of text prompts and visual guidance during the testing phase, the accuracy and robustness of anomaly detection are improved. Specifically, it includes the following steps.

[0049] Step 1: Design of Anomaly Prompt Terms. Describe the anomalies that may occur in actual production for each detection category using text, and construct corresponding manual prompt term templates.

[0050] Step 2: Image Feature Extraction and KNN Model Construction. The model is used to extract features from the images in the training data, obtaining feature representations at different levels, and feature libraries are constructed accordingly.

[0051] Step 3: Text Prompt Construction. By randomly initializing the generated context vector, and utilizing the token embedding mechanism of the CLIP model, normal prompts are concatenated with predefined exception descriptions and learned exception context to generate hybrid text prompts.

[0052] Step 4: Cue-Guided Anomaly Detection (PAD). Through cue word learning, the similarity between normal image features and normal cue words is maximized while the similarity with abnormal cue words is minimized; simultaneously, the similarity between abnormal image features and abnormal cue words is maximized while the similarity with normal cue words is minimized, generating a text-level anomaly score.

[0053] Step 5: Visual Guided Anomaly Detection (VAD). A patch-level anomaly score based on image features is obtained by comparing the query image with features from the normal and abnormal sample databases using a memory database. This is combined with an image-level anomaly score obtained through image-level residual learning.

[0054] Step 6: Text and Image Score Fusion. Anomaly scores from the text branch and the image branch are fused using a harmonic average, and the ability to perceive anomalies in the text is provided by introducing explicit anomaly boundaries to control the boundary between normal and abnormal cue features.

[0055] Through the embodiments of this invention, anomaly prompt word templates are generated based on preset anomaly type descriptions, enabling potential anomaly types to be represented through textual descriptions. Anomaly text prompts are constructed based on the anomaly prompt word templates and learnable contextual tags, enhancing the adaptability and task relevance of text representation and improving the ability to capture anomaly types in small samples. Normal text prompts, anomaly text prompts, and multi-level image features of the query image are input into a prompt-guided anomaly detection model. By integrating textual and visual information, a text-level anomaly score is generated, effectively utilizing textual prompt guidance to identify anomaly patterns in the image. Visual-guided anomaly detection is performed based on the multi-level image features of the query image and an image feature library. A visual-level anomaly score is generated through comparative learning, strengthening the perception of anomalies in visual details and hierarchical features. Finally, the text-level and visual-level anomaly scores are fused, integrating visual and linguistic cues, to ultimately output the anomaly detection score of the query image, thereby improving the comprehensiveness, accuracy, and robustness of anomaly detection in small sample scenarios.

[0056] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided, wherein the anomaly text prompts include preset text anomaly prompts and learnable text anomaly prompts. Based on anomaly alert word templates and learnable contextual tags, anomaly text alerts are constructed, including: The preset text exception prompt is obtained by concatenating the learnable context marker with the predefined exception description. The preset text exception prompt is used to indicate the text prompt for known exception types. Learnable text exception prompts are obtained by concatenating learnable context tags and learnable exception suffix tags. Learnable text exception prompts are used to indicate text prompts for unknown exception types by adjusting the learnable exception suffix tags.

[0057] In some embodiments, a systematic analysis of potential anomaly types (such as surface scratches, size deviations, color anomalies, etc.) in the target detection category is conducted by collaborating domain experts, and scenario modeling is performed in conjunction with typical cases in actual production.

[0058] For each exception type, a detailed text description must be provided for the exception text prompt. For example, for the carpet category, the exception could be constructed as: 'carpet': ['{} with hole', '{} with color stain', '{} with metal contamination', '{} with thread residue', '{} with thread', '{} with cut'].

[0059] The current anomaly detection task is defined as having M+1 classes: one normal class, M-1 known anomaly classes whose prompts can be partially defined manually, and the remaining unknown anomaly classes that cannot be defined are grouped into one class. The construction of text prompts is a crucial step in the anomaly detection system, aiming to guide the model to identify normal and abnormal states through natural language descriptions.

[0060] Construct normal text prompts in the following ways Guide the model to align with normal visual features: in, It is a length of Learnable contextual tags (such as [N1][N2][N3]) are used to enhance semantic representation. [obj.] is the name of the target object (e.g., the name of a query image, such as [cable]). During training, the embedding vectors of normal text prompts gradually align with normal visual features through contrastive learning, forming a stable semantic representation.

[0061] M-1 types of exceptions are constructed by using manual exception prompts to describe the expected M-1 possible exception scenarios in text. Preset text exception prompts are built using the following method. Guide the model to align abnormal visual features: Here, [with][color][stain] are suffixes extracted from the anomaly labels in the dataset (such as "with color spots" or "with cracks"). Directly incorporating domain knowledge makes the model sensitive to known anomaly types. By semantically connecting learnable normal suggestion prefixes with various known anomalies, M-1 text suggestions sensitive to different anomaly types can be obtained, achieving fine-grained classification.

[0062] Pre-defined text-based anomaly warnings rely on domain knowledge and anomaly cases from real-world production scenarios. For example, for surface defects, a specific description (such as "scratches exist on the surface") can be generated using pre-defined text-based anomaly warnings. Furthermore, to enable the model to learn other undescribed anomalies for M-type anomalies, learnable text-based anomaly warnings are constructed using the following method. Guide the model to align abnormal visual features in It is a learnable anomaly suffix label of length EA, with initial values ​​randomly initialized using a normal distribution. The suffix parameters are dynamically adjusted through training to capture unknown anomaly features.

[0063] The preset text anomaly prompt is generated by concatenating learnable context markers with predefined anomaly descriptions. In practice, firstly, based on a systematic analysis of target detection categories by domain experts, known anomaly types (such as surface scratches, dimensional deviations, color anomalies, etc.) are identified, and detailed anomaly description text is generated accordingly. The learnable context markers consist of a set of trainable parameterized markers, such as sequences of length EN [N1][N2][N3], whose initial values ​​are generated randomly and dynamically optimized during training. The concatenation format of the preset text anomaly prompt is as follows: the learnable context markers, the target object name, and the predefined anomaly description suffix (such as "with color spots" or "with cracks") are sequentially concatenated to form a complete text prompt.

[0064] For example, for cable objects, preset text anomaly prompts such as "[N1][N2][N3][Cable][with color spots]" can be constructed. These preset text anomaly prompts directly incorporate domain knowledge, enabling the model to have a clear semantic awareness of known anomaly types. During training, the model aligns with the corresponding visual features of anomalies through comparative learning, enhancing the accuracy of identifying known anomalies.

[0065] Learnable text anomaly hints are generated by concatenating learnable context markers with learnable anomaly suffix markers. The learnable anomaly suffix markers consist of a set of configurable, trainable markers (e.g., [A1][A2]…[AEA]), whose initial values ​​are randomly initialized using a normal distribution and adaptively adjusted based on the anomaly data distribution during model training. The concatenation format of the learnable text anomaly hints is as follows: the learnable context markers, the target object name, and the learnable anomaly suffix markers are sequentially concatenated to form a dynamic text hint, such as "[N1][N2][N3][cable][A1][A2]". This design allows the model to capture the semantic features of unknown anomaly types without relying on prior anomaly descriptions, simply by optimizing the learnable parameters. During training, the learnable text anomaly hints gradually learn to represent undefined anomaly patterns in the training data by maximizing the similarity to anomaly visual features, thereby expanding the model's generalization ability to unknown anomalies.

[0066] Through the embodiments of the present invention, explicit modeling of known anomaly types and adaptive learning of unknown anomaly types are realized. Combined with normal text prompts, a multi-granular text guidance mechanism is constructed, which effectively improves the semantic perception ability and robustness of anomaly detection under small sample conditions.

[0067] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided, which inputs normal text prompts, abnormal text prompts, and multi-level image features of the query image into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model, including: Determine the similarity between the multi-level image features of the query image and the normal text prompt, and obtain the normal similarity value; Determine the similarity between the multi-level image features of the query image and the abnormal text prompts to obtain the abnormal similarity value; Based on the difference between normal and abnormal similarity values, a text-level anomaly score is generated.

[0068] In this embodiment of the invention, after the prompt-guided anomaly detection model completes training, it enters the inference phase. The model first extracts multi-level image features from the query image using a visual encoder. These multi-level image features encompass a comprehensive visual representation, ranging from low-level texture details to high-level semantic information. The text encoder maps normal text prompts and abnormal text prompts into text feature vectors, where normal text prompts are composed of learnable context markers and the target object name, and abnormal text prompts include preset text anomaly prompts and learnable text anomaly prompts.

[0069] When determining the similarity between the multi-level image features of the query image and the normal text prompt, the model uses the cosine similarity measurement method. Specifically, the global feature vector in the multi-level image features is multiplied by the feature vector of the normal text prompt and then normalized to calculate the normal similarity value. This value reflects the degree of semantic alignment between the visual content of the query image and the normal text description; a higher normal similarity value indicates that the image is more likely to belong to the normal category.

[0070] When determining the similarity between multi-level image features of the query image and anomalous text prompts, the model calculates the cosine similarity between the image features and each type of anomalous text prompt (including preset anomalous prompts and learnable anomalous prompts), and takes the maximum similarity between anomalous classes as the anomalous similarity value. This ensures that the model can capture the semantic association between the query image and the most relevant anomaly type; a higher anomalous similarity value indicates a greater likelihood that the image presents anomalous features.

[0071] When generating text-level anomaly scores based on the difference between normal and abnormal similarity values, the model employs a difference comparison strategy. The abnormal similarity value is subtracted from the normal similarity value, and the resulting difference is processed by a sign function to obtain the text-level anomaly score. If the difference is positive, it indicates that abnormal similarity is dominant, and a positive score is output to suggest the possibility of an anomaly; if the difference is negative, it indicates that normal similarity is dominant, and a negative score is output to suggest a normal state. Thus, by fully utilizing the semantic guidance of textual prompts and comparing the correlation strength between positive and negative text prototypes and visual features, the model achieves sensitive discrimination of small sample anomalies.

[0072] Through the embodiments of the present invention, by integrating multi-level image features and multi-type text prompts in interactive calculation, the text-level anomaly score has both semantic interpretability and the ability to perceive visual details.

[0073] According to the present invention, a few-shot anomaly detection method based on visual-language joint learning is provided, the method further includes: Obtain the training image dataset, which includes normal samples and abnormal samples; Based on the training image dataset, multi-level image features are extracted; Based on normal and abnormal text prompts, text features are extracted through a text encoder to generate normal and abnormal text prototypes. Based on multi-level image features and text prototypes, comparative learning optimization is performed to obtain the optimized similarity relationship. Based on the optimized similarity relationship, a triple loss optimization is performed to obtain the enhanced discrimination boundary; Based on the enhanced discrimination boundary, the learnable parameters are dynamically adjusted using an optimizer to obtain the trained cue-guided anomaly detection model.

[0074] In this embodiment of the invention, after the text prompt is constructed, the model further optimizes the prompt embedding space through comparative learning.

[0075] The CLIP model extracts normal and abnormal text features using a text encoder. Then, by calculating the average feature vector and L2 normalization, a normal text feature anchor and M abnormal text feature anchors for different abnormal categories are constructed. Through contrastive learning and Triplet Loss dual constraint mechanisms, the cue embedding space of positive and negative samples is optimized simultaneously.

[0076] The core objectives of the model are: to maximize the similarity between normal image features and normal text prompts; to minimize the similarity between normal image features and abnormal text prompts; to maximize the similarity between abnormal image features and abnormal text prompts; and to minimize the similarity between abnormal image features and normal text prompts.

[0077] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning is provided. Based on multi-level image features and text prototypes, contrastive learning optimization is performed to obtain an optimized similarity relationship, including: Maximize the similarity between multi-level image features of normal samples and normal text prototypes, and minimize the similarity between normal samples and abnormal text prototypes: Maximize the similarity between the multi-level image features of the anomalous sample and the corresponding anomalous text prototype, and minimize the similarity between the anomalous sample and the normal text prototype and other anomalous text prototypes. Based on the optimized similarity relationship, a triple loss optimization is performed to obtain the enhanced discrimination boundary, including: The multi-level image features of forced normal samples have a higher similarity to normal text prototypes than to abnormal text prototypes. The multi-level image features of forced anomalous samples show a higher similarity to the corresponding anomalous text prototypes than to the normal text prototypes.

[0078] In this few-shot learning framework, this invention defines a batch as containing at least 2 normal samples, 2 similar abnormal samples, and 1 dissimilar abnormal sample; using Identify the currently selected anomaly category. The specific loss function design is as follows: Normal sample contrast learning: Maximizing multi-level image features of normal samples Compared to normal text prototypes The similarity is minimized to the abnormal text prototype. Similarity: in For temperature parameters, Represents cosine similarity. This represents the i-th normal sample.

[0079] Anomaly Comparative Learning: Maximizing Multi-Level Image Features of Single-Class Anomalies Its corresponding exception text prototype The similarity is minimized with normal and other abnormal text prototypes. Similarity: in This represents the p-th abnormal sample. Indicate the category of the p-th abnormal sample: Total comparison loss: Through a contrastive learning mechanism, the multi-level image features of normal samples are forced to be close to normal text anchors in the embedding space, while being far away from abnormal text anchors; at the same time, the multi-level image features of abnormal samples must be close to abnormal text anchors, while being far away from normal text anchors. This two-way constraint ensures clear separation of positive and negative samples in the embedding space.

[0080] Normal Sample Triplet Loss: Forced multi-level image features for normal samples Compared to normal text prototypes The similarity is higher than that with the abnormal text prototype. Similarity: in For the margin.

[0081] Triplet Loss for Outliers: Enforcing Multi-Level Image Features for Outliers Similar abnormal text prototypes The similarity is higher than that with the normal text prototype. Similarity: The definition is the same as above, total Triplet Loss: Building upon contrastive learning, Triplet Loss is further introduced. By setting an interval parameter, the similarity between normal samples and normal anchors is forced to be higher than their similarity with abnormal anchors, and vice versa. This design enhances the model's ability to distinguish anomalies (the discrimination boundary is the difference between normal and abnormal anchors).

[0082] During training, the model uses the AdamW optimizer to dynamically adjust the embedding parameters of learnable cues (NP, LAP). During inference, the model generates anomaly scores for text branches by calculating the similarity difference between image features and normal / abnormal cues.

[0083] Through the embodiments of the present invention, by fusing multi-level image features and text prototypes, and by using contrastive learning and triple loss to optimize similarity relationships and discrimination boundaries, the generalization ability and accuracy of anomaly detection under small sample conditions are effectively improved; at the same time, a learnable prompting mechanism is introduced to dynamically adjust model parameters and enhance the ability to distinguish between normal and abnormal semantics.

[0084] According to the present invention, a few-sample anomaly detection method based on visual-language joint learning performs visual-guided anomaly detection based on multi-level image features of a query image and an image feature library, generating a visual-level anomaly score, including: Based on the multi-level image features and image feature library of the query image, the local anomaly score is calculated by comparing the feature memory. The feature memory comparison uses the K-nearest neighbor algorithm to calculate the Euclidean distance between the query image features and the corresponding level features in the image feature library. The K-nearest neighbors algorithm is used to calculate the Euclidean distance between the multi-level image features of the query image and the corresponding level features in the image feature database to obtain the local anomaly score. Based on the multi-level image features and image feature library of the query image, the global anomaly score is calculated through image-level residual learning; Visual-level anomaly scores are generated by dynamically weighted fusion based on local and global anomaly scores. The dynamic weighted fusion adaptively allocates local and global weights through a gating network.

[0085] In this embodiment of the invention, image features are extracted using the image encoding module of the CLIP model. Multi-level image features v (low-level texture + high-level semantics) of the image samples are extracted respectively, and the normalized features are... The input image is then stored as an independent image feature library. After encoding, the input image outputs multi-level features: low-level features can capture local details such as edges and textures, while mid- to high-level features have richer semantic information.

[0086] The extracted features are L2 normalized to eliminate scale differences. The formula is as follows: The normalized features are stored in an image feature database, storing the normalization results for different levels of features. A KNN model is built using NearestNeighbors, and Euclidean distance is used to calculate the multi-level image features of the query image during testing. Corresponding hierarchical features in the image feature library Similarity. The Euclidean distance formula is: in, Representing feature dimension, This indicates the multi-level image features of the query image at the 1st level. Each dimension of feature components, This indicates that the corresponding hierarchical feature in the image feature library is at the [number]th [level]. Each dimension of feature components.

[0087] At the local scale, visually guided anomaly detection uses the K-Nearest Neighbors (KNN) algorithm to compare the local features of the query image with the mean Euclidean distance of the normal feature library and the anomaly feature library. The system generates patch-level anomaly scores. Specifically, after flattening the multi-level local features of the input image, KNN searches are performed with a predefined feature library. By calculating the average Euclidean distance between the patch features of the test image and the nearest neighbor normal features, the model obtains local anomaly scores at multiple levels. This process can effectively capture subtle defects (such as scratches and blemishes) or local texture anomalies.

[0088] At a global scale, visually guided anomaly detection utilizes an adapter to align the global token features of the query image with both normal and anomaly feature libraries. By calculating the residuals between the two libraries separately and inputting them into a fully connected layer, the model generates an image-level anomaly score (i.e., a global anomaly score). This score reflects the degree of deviation between the overall image and normal samples, and is suitable for detecting large-scale anomalies (such as distortion and blurring).

[0089] By dynamically fusing local KNN distance scores (i.e., local anomaly scores) with global residual scores (i.e., global anomaly scores), the model generates the final visual anomaly heatmap. Specifically, the scores are first concatenated and averaged, then input into a gating network to generate dynamic weights; subsequently, they are reshaped into a grid shape (HxW) through weighted averages to form a pixel-level anomaly heatmap. This method balances local and global information through adaptive weight allocation, avoiding the subjective bias of fixed weights, and achieving high-precision anomaly localization and evaluation.

[0090] Focal Loss mitigates the class imbalance between normal and abnormal samples in anomaly detection by dynamically adjusting loss weights, enabling the model to focus more on difficult-to-classify anomalous regions. In a visually guided anomaly detection framework, it combines multi-level anomaly heatmaps to reduce the loss contribution of easily classified normal regions, enhancing the model's sensitivity to minor anomalies and thus improving detection accuracy and robustness.

[0091] In some embodiments, an anomaly detection score for the query image is obtained by fusing scores based on text-level anomaly scores and visual-level anomaly scores, through the following steps.

[0092] The anomaly scores of the text branch and the image branch are fused using a harmonic mean formula to enhance sensitivity to anomalies.

[0093] In some embodiments, the global anomaly score (img_score) and local anomaly score (i.e., the maximum value of the local heatmap, map_score) of the query image are used to obtain the visual anomaly score (fused_score) using the following formula: The fusion process is performed. This formula uses a reverse weighting mechanism to ensure that a low score in any branch (e.g., no anomaly detected in the local heatmap) significantly reduces the fusion result, thus avoiding misjudgments from a single modality. For example, if the overall image score is high but the local heatmap score is low, the fused score will decrease significantly, indicating that the model needs further verification of the anomaly's reliability, effectively improving robustness to minor anomalies.

[0094] refer to Figure 2 , Figure 2 This is a structural diagram of the few-sample anomaly detection method based on visual-language joint learning provided by the present invention.

[0095] First, anomaly warning words are designed, and text warning words are constructed accordingly. The generated text warnings are converted into features by a text encoder. Simultaneously, visual features are extracted from the input image using an image encoder. Then, two core detection branches are established: one is warning-guided anomaly detection, which integrates text-encoded features and image-encoded features for cross-modal alignment and comparison; the other is visually guided anomaly detection, which compares image-encoded features with a memory bank storing a large number of prototype features to identify visually abnormal patterns. Finally, the detection results from both branches are fed into a scoring fusion module, which integrates text-level and visual-level anomaly signals to output the final anomaly detection score, thus achieving comprehensive and sensitive anomaly recognition even with small sample sizes.

[0096] The following describes the few-shot anomaly detection device based on visual language joint learning provided by the present invention. The few-shot anomaly detection device based on visual language joint learning described below can be referred to in correspondence with the few-shot anomaly detection method based on visual language joint learning described above.

[0097] refer to Figure 3 , Figure 3 This is a schematic diagram of a module of the small sample anomaly detection device based on visual-language joint learning provided by the present invention.

[0098] The generation module 301 is used to generate an exception prompt word template based on a preset exception type description, wherein the exception prompt word template is used to represent the potential exception type through text description; Module 302 is used to construct an exception text prompt based on the exception prompt word template and learnable context markers; The prompting module 303 is used to input normal text prompts, abnormal text prompts, and multi-level image features of the query image into the prompting anomaly detection model to obtain the text-level anomaly score output by the prompting anomaly detection model. The visual guidance module 304 is used to perform visual guidance anomaly detection based on the multi-level image features and image feature library of the query image, and generate a visual-level anomaly score. The scoring fusion module 305 is used to fuse scores based on text anomaly scores and visual anomaly scores to obtain an anomaly detection score for the query image.

[0099] Specifically, the small sample anomaly detection device based on visual language joint learning provided by the present invention can implement all the method steps implemented in the above embodiments of the small sample anomaly detection method based on visual language joint learning, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0100] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a few-sample anomaly detection method based on visual-language joint learning. This method includes: generating anomaly prompt word templates based on preset anomaly type descriptions, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; constructing anomaly text prompts based on the anomaly prompt word templates and learnable context markers; inputting normal text prompts and anomaly text prompts, as well as multi-level image features of the query image, into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model; performing visual-guided anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and fusing the text-level anomaly score and the visual-level anomaly score to obtain an anomaly detection score for the query image.

[0101] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the few-sample anomaly detection method based on visual-language joint learning provided by the above methods. The method includes: generating anomaly prompt word templates based on preset anomaly type descriptions, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; constructing anomaly text prompts based on the anomaly prompt word templates and learnable context markers; inputting normal text prompts and anomaly text prompts, as well as multi-level image features of the query image, into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model; performing visual-guided anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and fusing the scores based on the text-level anomaly score and the visual-level anomaly score to obtain an anomaly detection score for the query image.

[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the few-sample anomaly detection method based on visual-language joint learning provided by the above methods. This method includes: generating anomaly prompt word templates based on preset anomaly type descriptions, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; constructing anomaly text prompts based on the anomaly prompt word templates and learnable context markers; inputting normal text prompts and anomaly text prompts, as well as multi-level image features of the query image, into a prompt-guided anomaly detection model to obtain a text-level anomaly score output by the prompt-guided anomaly detection model; performing visual-guided anomaly detection based on the multi-level image features of the query image and an image feature library to generate a visual-level anomaly score; and fusing the text-level anomaly score and the visual-level anomaly score to obtain an anomaly detection score for the query image.

[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A few-sample anomaly detection method based on visual-language joint learning, characterized in that, include: Based on a preset anomaly type description, anomaly prompt word templates are generated, wherein the anomaly prompt word templates are used to represent potential anomaly types through text descriptions; Based on the aforementioned anomaly prompt word template and learnable context markers, anomaly text prompts are constructed; The normal text prompts and the abnormal text prompts, as well as the multi-level image features of the query image, are input into the prompt guidance anomaly detection model to obtain the text-level anomaly score output by the prompt guidance anomaly detection model. Based on the multi-level image features and image feature library of the query image, visual-guided anomaly detection is performed to generate a visual-level anomaly score. An anomaly detection score for the query image is obtained by fusing the text-level anomaly score and the visual-level anomaly score.

2. The few-sample anomaly detection method based on visual-language joint learning according to claim 1, characterized in that, The abnormal text prompts include preset text abnormal prompts and learnable text abnormal prompts; The construction of abnormal text prompts based on the abnormal prompt word template and learnable context markers includes: A preset text exception prompt is obtained by concatenating learnable context markers with predefined exception descriptions. The preset text exception prompt is used to indicate a text prompt for a known exception type. The learnable text exception prompt is obtained by concatenating the learnable context marker and the learnable exception suffix marker. The learnable text exception prompt is used to indicate the text prompt for unknown exception types by adjusting the learnable exception suffix marker.

3. The few-sample anomaly detection method based on visual-language joint learning according to claim 1, characterized in that, The step of inputting normal text prompts, abnormal text prompts, and multi-level image features of the query image into the prompt guidance anomaly detection model to obtain the text-level anomaly score output by the prompt guidance anomaly detection model includes: Determine the similarity between the multi-level image features of the query image and the normal text prompt, and obtain the normal similarity value; Determine the similarity between the multi-level image features of the query image and the abnormal text prompt, and obtain the abnormal similarity value; Based on the difference between the normal similarity value and the abnormal similarity value, a text-level anomaly score is generated.

4. The few-sample anomaly detection method based on visual-language joint learning according to claim 1, characterized in that, The method further includes: Obtain a training image dataset, wherein the training image dataset includes normal samples and abnormal samples; Based on the training image dataset, multi-level image features are extracted; Based on the normal text prompts and the abnormal text prompts, text features are extracted by a text encoder to generate normal text prototypes and abnormal text prototypes; Based on the multi-level image features and the text prototype, comparative learning optimization is performed to obtain the optimized similarity relationship; Based on the optimized similarity relationship, triple loss optimization is performed to obtain the enhanced discrimination boundary; Based on the enhanced discrimination boundary, the learnable parameters are dynamically adjusted using an optimizer to obtain the trained cue-guided anomaly detection model.

5. The few-shot anomaly detection method based on visual-language joint learning according to claim 4, characterized in that, The process of performing comparative learning optimization based on the multi-level image features and the text prototype to obtain the optimized similarity relationship includes: Maximize the similarity between the multi-level image features of the normal sample and the normal text prototype, and minimize the similarity between the normal sample and the abnormal text prototype: Maximize the similarity between the multi-level image features of the anomalous sample and the corresponding anomalous text prototype, and minimize the similarity between the anomalous sample and the normal text prototype and other anomalous text prototypes. The enhanced discrimination boundary is obtained by performing triple loss optimization based on the optimized similarity relationship, including: The similarity between the multi-level image features of the normal sample and the normal text prototype is forced to be higher than the similarity with the abnormal text prototype. The similarity between the multi-level image features of the anomalous sample and the corresponding anomalous text prototype is forced to be higher than the similarity with the normal text prototype.

6. The few-sample anomaly detection method based on visual-language joint learning according to claim 1, characterized in that, The step of performing visually guided anomaly detection based on the multi-level image features and image feature library of the query image, and generating a visual-level anomaly score, includes: Based on the multi-level image features of the query image and the image feature library, a local anomaly score is calculated by comparing the feature memory library. The feature memory library comparison uses the K-nearest neighbor algorithm to calculate the Euclidean distance between the query image features and the corresponding level features in the image feature library. The K-nearest neighbor algorithm is used to calculate the Euclidean distance between the multi-level image features of the query image and the corresponding level features in the image feature database to obtain the local anomaly score. Based on the multi-level image features of the query image and the image feature library, a global anomaly score is calculated through image-level residual learning; Based on the local anomaly score and the global anomaly score, a visual-level anomaly score is generated through dynamic weighted fusion, wherein the dynamic weighted fusion adaptively allocates local weights and global weights through a gating network.

7. A few-sample anomaly detection device based on visual-language joint learning, characterized in that, include: The generation module is used to generate an exception prompt word template based on a preset exception type description, wherein the exception prompt word template is used to represent the potential exception type through text description; A construction module is used to construct abnormal text prompts based on the abnormal prompt word template and learnable context markers; The prompting and guidance module is used to input normal text prompts and abnormal text prompts, as well as multi-level image features of the query image, into the prompting and guidance anomaly detection model to obtain the text-level anomaly score output by the prompting and guidance anomaly detection model. The visual guidance module is used to perform visual guidance anomaly detection based on the multi-level image features and image feature library of the query image, and generate a visual-level anomaly score. The scoring fusion module is used to fuse the scores based on the text anomaly score and the visual anomaly score to obtain the anomaly detection score of the query image.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the few-sample anomaly detection method based on visual-language joint learning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the few-sample anomaly detection method based on visual-language joint learning as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the few-sample anomaly detection method based on visual-language joint learning as described in any one of claims 1 to 6.

Images