Model training method and device, equipment and readable storage medium

Abstract

This application provides a model training method, apparatus, device, and readable storage medium. The method includes: inputting training data into a first model in a target model to extract visual features, obtaining first visual features of image samples in the training data; based on the first visual features, using a second model in the target model to generate text features aligned with the first visual features, obtaining two target text prompt features of the image samples in the training data; based on the first visual features and the two target text prompt features of the first image samples in the training data, using a third model in the target model to generate fence features; based on the fence features and the two target text prompt features, constructing a first loss of the second model, the first loss being used to update the two target text prompt features; and, after the second model has been trained, training the first model based on the fence features and the two target text prompt features.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, and readable storage medium. Background Technology

[0002] The core objective of industrial visual anomaly detection is to build an anomaly detection model that can accurately identify abnormal (defective) products, given only normal (defect-free) product samples available for learning. This technology is crucial for ensuring the automation of industrial production lines and improving product quality.

[0003] In related technologies, a large-scale visual-language model (VLM) can be used to construct anomaly detection models. By performing contrastive learning on a massive dataset containing hundreds of millions of image-text pairs, it successfully maps the outputs of the visual encoder and text encoder into a unified multimodal semantic space. This allows for the direct use of natural language to define the concepts of normal and abnormal products in images. For example, text prompts can be constructed, such as "a photo of a perfectly good bottle" and "a photo of a defective bottle." After aligning the visual features with the text prompt features, the similarity between the image's visual features and these two text prompt features can be compared to determine whether the product in the image is abnormal.

[0004] However, anomaly detection models have an inherent contradiction between adaptability and discriminability. Optimized models are usually the product of compromise between the two. Furthermore, current text prompts are usually fixed manual prompts. Using a set of general and fixed prompts to measure all situations will inevitably lead to coarse and vague decision boundaries between normal and abnormal regions in the feature space. This directly results in high false positive and false negative rates when facing atypical normal samples and subtle defects, making the accuracy of the trained anomaly detection model relatively low. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, and readable storage medium, which can solve the technical problem that the accuracy of anomaly detection models trained in related technologies is relatively low.

[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0007] The training data is input into the first model of the target model to extract visual features, thereby obtaining the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image.

[0008] Based on the first visual feature, text features aligned with the first visual feature are generated using the second model in the target model, thereby obtaining two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal.

[0009] Based on the first visual features of the first image sample in the training data and the two target text prompt features, a fence feature is generated using the third model in the target model. The first image sample is an image sample with normal objects, and the fence feature is used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects.

[0010] Based on the fence features and the two target text prompt features, a first loss of the second model is constructed, and the first loss is used to update the two target text prompt features.

[0011] Once the second model has been trained, the first model is trained based on the fence features and the two target text prompt features. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

[0012] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0013] The visual feature extraction module is used to input training data into the first model of the target model to extract visual features and obtain the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image.

[0014] The feature alignment module is used to generate text features aligned with the first visual feature based on the first visual feature and using the second model in the target model, thereby obtaining two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal.

[0015] The generation module is used to generate fence features based on the first visual features of the first image sample in the training data and the two target text prompt features, using the third model in the target model. The first image sample is an image sample with normal objects, and the fence features are used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects.

[0016] A construction module is used to construct a first loss for the second model based on the fence features and the two target text prompt features, wherein the first loss is used to update the two target text prompt features;

[0017] The training module is used to train the first model based on the fence features and the two target text prompt features after the second model has been trained. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

[0018] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the model training method as described in the first aspect.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the model training method as described in the first aspect.

[0020] Fifthly, embodiments of this application provide a computer program product, including computer instructions that, when executed by a processor, implement the steps of the model training method as described in the first aspect.

[0021] In this embodiment, visual features are extracted by inputting training data into a first model in the target model to obtain first visual features of image samples in the training data. The target model is used for anomaly detection of objects in the image. Based on the first visual features, text features aligned with the first visual features are generated using a second model in the target model to obtain two target text prompt features of the image samples in the training data. The two target text prompt features include: a first text prompt feature indicating that objects in the image samples are normal and a second text prompt feature indicating that objects in the image samples are abnormal. Based on the first visual features of the first image samples in the training data and the two target text prompt features, the target model is used... The third model generates fence features, where the first image sample is a normal image sample, and the fence features are used to indicate the decision boundary between the first image sample and the second image sample, where the second image sample is an abnormal image sample. Based on the fence features and the two target text prompt features, a first loss of the second model is constructed, and the first loss is used to update the two target text prompt features. When the second model has been trained, the first model is trained based on the fence features and the two target text prompt features, where the two target text prompt features are used to combine the second visual features of the first image sample extracted by the first model when training is complete to detect anomalies in the image. In this way, a decoupled two-stage training framework can be used to completely separate the two conflicting optimization tasks of semantic concept learning for text prompts and visual feature boundary sculpting. In the first stage, the focus is on training the second model to learn a set of highly discriminative and adaptive semantic concepts using the fence features output by the third model, resulting in two target text prompt features. In the second stage, the first model is trained using this set of solidified high-quality concepts, namely the fence features and the two target text prompt features, to actively and explicitly sculpt the decision boundaries of the visual feature space, ultimately forming a clear, sharp, and robust anomaly detection model, improving the accuracy of anomaly detection. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a model training method provided in an embodiment of this application;

[0024] Figure 2This is the architecture diagram of the target model;

[0025] Figure 3 This is a schematic diagram of the DCF module;

[0026] Figure 4 This is a schematic diagram of the adversarial attack in the third model;

[0027] Figure 5 This is a diagram illustrating the comparison of decision boundaries;

[0028] Figure 6 This is a schematic diagram illustrating the principle of model training;

[0029] Figure 7 This is a structural diagram of a model training device provided in an embodiment of this application;

[0030] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0032] Industrial visual anomaly detection aims to build a model that can accurately identify abnormal (defective) products, given only normal (defect-free) product samples for learning. This technology is crucial for ensuring the automation of industrial production lines and improving product quality. In publicly known technologies, two main technological paradigms exist in this field:

[0033] 1. Reconstruction-based technical solutions.

[0034] The core assumption of this approach is that a model trained only on normal samples will produce significant errors when attempting to reconstruct an anomalous sample. A typical implementation uses a deep autoencoder network, consisting of an encoder and a decoder. During training, it is fed a large number of normal images (i.e., images of normal objects), with the goal of making the decoder's output image as consistent as possible with the input images. During the detection phase, the image to be detected is input into the model; if the difference between the reconstructed image and the original image (i.e., the reconstruction error) exceeds a preset threshold, it is considered an anomalous. In recent years, diffusion models have also been applied to this type of task, identifying anomalous images by learning the process of recovering normal images from noise.

[0035] 2. Feature Embedding-based technical solution.

[0036] Such schemes aim to map the features of normal samples to a compact, well-defined region (or "manifold") in the feature space. They typically employ deep neural networks pre-trained on large image datasets (such as ImageNet) as powerful feature extractors. A representative and well-known technique is to extract features from local image patches of all training normal images, constructing a large, representative "normal feature memory." During detection, the features of each image patch in the image to be detected are compared with all features in the memory, calculating its distance to its nearest neighbor. If the feature distance of an image patch is too far, the region is marked as an anomaly.

[0037] While both of these traditional approaches are effective in specific scenarios, they share a fundamental limitation: they follow a one-model-per-class paradigm. This means that whenever a new product category appears on the production line, a large number of normal samples for that category must be collected again, and a completely new, dedicated detection model must be trained from scratch. In the context of modern industrial environments with diverse products and rapid iterations, this results in extremely high development costs and lengthy deployment cycles, lacking the necessary flexibility and scalability.

[0038] In recent years, in order to overcome the limitations of traditional methods, large-scale visual language models, represented by Contrastive Language-Image Pre-Training (CLIP) models, have brought revolutionary changes to the field of anomaly detection.

[0039] The core technology of the CLIP model lies in its ability to successfully map the outputs of the visual encoder and the text encoder to a unified multimodal semantic space through contrastive learning on a massive dataset containing hundreds of millions of image-text pairs. In this space, the features of an image of a "dog" are spatially very close to the text features of its corresponding text "a dog".

[0040] This characteristic gives CLIP powerful zero- / few-shot generalization capabilities. For anomaly detection tasks, a large number of image samples are no longer required; instead, the concepts of "normal" and "abnormal" can be directly defined using natural language. For example, text prompts such as "a picture of a perfectly intact bottle" and "a picture of a flawed bottle" can be constructed, and the visual features of the image to be detected can be compared with the features of these two text prompts to determine whether it is abnormal. This paradigm of using language supervision to guide visual tasks is the foundation of the technical solution constructed in the embodiments of this application.

[0041] Among related technologies, a series of small-sample anomaly detection schemes have emerged based on VLM.

[0042] Option 1: A solution based on fixed manual prompts. For example, the implementation process of this technical solution is as follows:

[0043] 1. Model foundation: The pre-trained CLIP model, including its visual encoder and text encoder, is used directly, and the model parameters are completely frozen during the detection process.

[0044] 2. Prompt Construction: For specific detection tasks, a fixed set of text prompts is designed manually. For example, for detecting defects in "bottles", multiple sentences describing normal and abnormal states would be designed, such as "a perfect bottle photo", "a bottle photo with scratches", and "a bottle photo with stains".

[0045] 3. Detection Process: The image to be detected is divided into multiple overlapping image patches using a sliding window approach. Each image patch is independently fed into the visual encoder of the CLIP model to extract its visual features.

[0046] 4. Anomaly Scoring: For each extracted image patch feature, the cosine similarity is calculated with the text features of all pre-designed "normal" and "abnormal" prompts. By comparing the average similarity score of the image patch with the "normal" and "abnormal" concept sets, it is determined whether the image patch is abnormal. Finally, the anomaly scores of all image patches are integrated into an anomaly heatmap.

[0047] The core of solutions based on fixed manual prompts lies in the static and manual nature of their text prompts; the prompts are predefined and unchanging. Such static prompts cannot capture normal appearance variations within the same product category, such as the metallic luster under different lighting conditions, nor can they precisely describe the unique attributes of different product categories. For example, "bent transistor pins" and "broken medicine bottle caps" are distinctly different anomalies. Therefore, using a universal, fixed set of prompts to measure all situations inevitably leads to coarse and vague decision boundaries in the feature space, directly resulting in high false positive and false negative rates when facing atypical normal samples and subtle defects.

[0048] Option 2: A solution based on static prompting learning. For example, its technical implementation is as follows:

[0049] 1. Automated learning of prompts: This approach no longer relies on manually written sentences, but instead learns the concepts of "normal" and "abnormal" as two continuous, optimizable vectors (i.e., "soft prompts").

[0050] 2. Training Process: During the training phase, the model receives a small number of normal samples (e.g., 1-shot or 2-shot). The image features of these samples are extracted and interacted with the learning "normal" and "abnormal" cue vectors in the CLIP semantic space. The training objective is to maximize the similarity between the image features of these normal samples and the "normal" cue vectors, while minimizing the similarity with the "abnormal" cue vectors.

[0051] 3. Detection Process: After training, the two cue vectors for "normal" and "abnormal" are fixed. During detection, the features of the image to be detected are compared with these two learned cue vectors to calculate the anomalous score.

[0052] While static cue learning-based schemes achieve automated learning of text cues, the learned cue vectors remain static and globally shared. This static approach fundamentally fails to address the core problem: it learns only an average concept of "normal," failing to capture the significant semantic differences between the normal texture of "wood" and the normal luster of "metal." Therefore, although the resulting decision boundary may be adaptively learned, for multi-class tasks, this boundary is inevitably a compromised and imprecise hyperplane, unable to closely fit the normal feature distribution of each class. This also leads to performance bottlenecks in complex multi-class scenarios.

[0053] Option 3: In pursuit of higher performance, a more complex structure is introduced based on the standard CLIP model, namely a scheme based on heavyweight fusion modules. For example, its technical implementation is as follows:

[0054] 1. Introducing a heavyweight module: The core of this approach lies in integrating an additional, pre-trained heavyweight module, such as Q-Former, which originates from the BLIP-2 model. Q-Former is a complex Transformer structure that is inserted between the visual encoder and the text encoder, acting as a "visual-language information mediator".

[0055] 2. Working Mechanism: First, from the image features output by the visual encoder, the most relevant and refined visual summary information is extracted from the text through a series of learnable query vectors. Then, this summary information is further interacted with and integrated with the text prompts at a deeper and more refined level.

[0056] 3. Objective: To better bridge the modal gap between visual and textual representations through this powerful mediation module, in order to obtain more discriminative feature representations.

[0057] The technical approach involves increasing the complexity of the model structure to improve performance, with the performance gain primarily stemming from the powerful information processing capabilities of the introduced heavyweight Q-Former module. This approach significantly increases the number of model parameters and computational complexity, making it unsuitable for deployment on resource-constrained industrial edge devices. Secondly, while it optimizes the feature fusion method, it doesn't fundamentally solve the problem of ambiguous decision boundaries. It still passively relies on the model learning the boundaries during optimization, without providing an explicit, proactive mechanism to forge and sharpen the decision boundaries. Therefore, despite its good performance, its technical philosophy remains focused on how to better represent features, rather than how to proactively construct a perfect decision boundary.

[0058] It is known that the relevant technologies have the following technical shortcomings when solving the problem of detecting anomalies in a small number of samples and multiple categories:

[0059] 1. The decision-making boundary is vague and passively formed, leading to misjudgment of key samples.

[0060] Whether using fixed manual prompts or learned static global prompts, the essence is to attempt to measure all situations with a single semantic standard. This approach cannot adapt to subtle differences between different categories and instances. Therefore, the decision boundaries of "normal" and "abnormal" learned in the feature space are inevitably fuzzy, broad, and poorly defined. For schemes that introduce complex modules, although their feature representation capabilities are stronger, they still rely on the model to passively and implicitly form decision boundaries driven by training data, failing to fundamentally solve the problem of boundary fuzziness.

[0061] This blurred boundary causes the model to perform poorly when faced with two key types of samples: atypical normal samples, such as normal products whose appearance differs from the training samples due to changes in lighting or angle, and slightly abnormal samples, such as defective products with minor scratches or slight color differences. The former are easily falsely reported as abnormal, while the latter are easily falsely reported as normal, which is unacceptable in industrial quality inspection scenarios that require high precision.

[0062] 2. There is an inherent contradiction between the adaptability and discriminativeness of the model, and related technologies are difficult to effectively balance.

[0063] An ideal anomaly detection model needs to simultaneously satisfy two contradictory requirements: adaptability, meaning the distribution area of ​​normal features should be large enough to encompass all normal variations; and discriminability, meaning the boundary between normal and anomalous regions must be clear and sharp. Related technical solutions struggle to reconcile this contradiction during the optimization process. For example, to adapt to multiple categories, the statically learned cues are inevitably a compromise, an average concept, sacrificing discriminability for specific categories.

[0064] In other words, in related technologies, the integrated, end-to-end training method causes the two objectives of adaptability and discriminability to hinder each other during the optimization process. The final model is often a compromise between the two, and cannot achieve the best in both dimensions at the same time, thus limiting the upper limit of detection performance.

[0065] 3. There is a lack of technical means to proactively and explicitly shape the decision-making boundaries.

[0066] The commonality among related technologies lies in the fact that they all provide data (images and text prompts) to the model and rely on optimization algorithms such as gradient descent to find a suitable decision boundary. The whole process is indirect and implicit.

[0067] This boundary learning method might work when there is sufficient data, but in scenarios with extremely sparse samples, the results exhibit significant uncertainty and instability. Due to the lack of effective supervision from samples in the boundary region, the quality of the boundaries learned by the model cannot be guaranteed.

[0068] Based on this, the embodiments of this application provide a new model training method, which aims to solve the above-mentioned technical problems. It can balance adaptability and discriminability through decoupling optimization and introduce an active and explicit boundary shaping mechanism, thereby constructing a clear, accurate and robust decision boundary under conditions of few samples, fundamentally improving the accuracy and reliability of multi-class anomaly detection.

[0069] The following provides a detailed description of the model training method provided in the embodiments of this application.

[0070] See Figure 1 , Figure 1 This is a flowchart of a model training method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0071] Step 101: Input the training data into the first model of the target model to extract visual features and obtain the first visual features of the image samples in the training data. The target model is used to detect anomalies of objects in the image.

[0072] Step 102: Based on the first visual feature, use the second model in the target model to generate text features aligned with the first visual feature, and obtain two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal.

[0073] Step 103: Based on the first visual features of the first image sample in the training data and the two target text prompt features, generate fence features using the third model in the target model. The first image sample is an image sample with normal objects. The fence features are used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects.

[0074] Step 104: Based on the fence features and the two target text prompt features, construct the first loss of the second model, which is used to update the two target text prompt features;

[0075] Step 105: After the second model training is completed, the first model is trained based on the fence features and the two target text prompt features. The two target text prompt features are used to combine the second visual features of the first image sample extracted by the first model after training to perform anomaly detection of objects in the image.

[0076] This application primarily relates to the fields of artificial intelligence and computer vision, specifically to scenarios such as industrial automation, intelligent manufacturing quality control, and machine learning model training. In particular, it relates to a technique for high-precision anomaly detection of objects, such as products, across multiple image categories under conditions of extremely limited training samples. Here, an image category represents an object category; for example, an image of a nail refers to a nail product, and an image of a screw refers to a screw product.

[0077] In step 101, the first model can be a ViT (Vision Transformer) based visual encoder. This visual encoder, as the backbone network of the target model, can extract visual features from the input image, thereby obtaining the first visual features of the image samples in the training data. In some embodiments, such as Figure 2 In the architecture diagram of the target model shown, the first model 201, i.e., the ViT-based visual encoder, can include multiple layers (VIT Layer), which can extract multi-level feature maps. At the output layer of the visual encoder, the global visual features of the image can be output, such as... Figure 2 As shown in the CLS Token, the first visual feature of the image sample is the global visual feature. Simultaneously, it can maintain a Normal Visual Memory to store the second visual features of the first image sample during the training phase. The second visual features can include the global visual features and / or local visual features of the first image sample.

[0078] It should be noted that the training data may include a first image sample and a second image sample. The first image sample refers to an image sample of a normal object, and the second image sample refers to an image sample of an abnormal object. Furthermore, the training data may include image samples from multiple image categories, enabling the target model to perform anomaly detection for multiple product categories.

[0079] In step 102, the target model may further include a second model, such as Figure 2 As shown, the second model 202 may include a Dynamic Concept Fusion (DCF) module for adaptively generating text features aligned with the first visual features. The second model may also include a Text Encoder for encoding the text features to obtain two target text prompt features from the image samples in the training data. These two target text prompt features include a first text prompt feature and a second text prompt feature. The first text prompt feature is a positive text prompt, indicating that the product in the image is normal, while the second text prompt feature is a negative text prompt, indicating that the product in the image is abnormal.

[0080] In some embodiments, the second model can construct an attention-based prompt generator to focus on a normal sample memory based on the first visual features and dynamically combine the sample features in the memory according to the attention weights to generate prompts; or it can directly use a lightweight language model to generate prompts based on the first visual features and the text description (caption), thereby obtaining two target text prompt features of the image samples in the training data.

[0081] In some embodiments, a second model can be used to align the first visual features with pre-stored general text prompt embedding features for each image category to obtain target text prompt embedding features; then, the target text prompt embedding features are text-encoded to obtain two target text prompt features for the image samples in the training data.

[0082] In some embodiments, to generate adaptive text prompts that are highly relevant to the instance content and image category of the image sample, some dedicated prompt parameters can be designed for each image category to obtain target-specific text prompt embedding features. The target general text prompt embedding features and the target-specific text prompt embedding features are aggregated to obtain target text prompt embedding features. Then, the target text prompt embedding features are text-encoded to obtain two target text prompt features for the image samples in the training data.

[0083] In some embodiments, the second model includes a first sub-model, a second sub-model, and a third sub-model, and step 102 specifically includes:

[0084] Based on the first sub-model, the first visual feature is aligned with the pre-stored general text prompt embedding features for each image category to obtain two target general text prompt embedding features for the target image category corresponding to the image sample in the training data.

[0085] For the first image sample and the second image sample respectively, based on the second sub-model, the target general text prompt embedding feature of the target image category and the target specific text prompt embedding feature of the target image category are integrated to obtain the target text embedding feature of the target image category;

[0086] The target text embedding features of the target image category are input into the third sub-model for text encoding to obtain two target text prompt features of the image samples in the training data. The two target text prompt features correspond to the target image category.

[0087] The second model may include a DCF module and a third sub-model. The DCF module may include a first sub-model and a second sub-model, while the third sub-model may be a text encoder.

[0088] In contrast to related technologies where all image categories and all image samples share the same pair of "normal / abnormal" prompt vectors, this embodiment generates unique prompts for each input image and each different image category through instance awareness and image category adaptation. This can adapt to subtle differences between different image categories and different instances, improving the accuracy of multi-category anomaly detection.

[0089] In some embodiments, the text alignment includes: the first sub-model outputting weight information based on the first visual feature; and weighting the weight information with each general text cue embedding feature to obtain the target general text cue embedding feature of the target image category, wherein the weight information is used to indicate the degree of alignment between the first visual feature and each general text cue embedding feature.

[0090] To enable text prompts to adapt to different instances, the first sub-model is the shared concept part, which can be implemented using a hybrid expert network (MoE) framework. Figure 3 This is a schematic diagram of the DCF module, as shown below. Figure 3 As shown, the DCF module may include a first sub-model 301 (such as...) Figure 3 In the moe-shared model, the first sub-model 301 obtains two target general text prompt embedding features corresponding to the target image category of the image samples in the training data by weighting the general text prompt embedding features applicable to various image categories. Here, P represents a positive text prompt, and N represents a negative text prompt, while the target image category can be obtained through instance perception.

[0091] The first sub-model can maintain an expert dictionary, such as a dictionary consisting of K experts. Learnable general text hint embedding features, where K can be set according to the actual situation, such as K=4. General text hint embedding features can be understood as encoding a basic general semantic, such as normal / abnormal in terms of texture, structure, color, etc.

[0092] The first sub-model may include a gating network, which is a multi-layer perceptron (MLP) that receives global visual feature vectors from the first model. This is taken as input, and a set of weights is output to weight the various general text cue embedding features. The calculation process is as follows: .

[0093] The ultimate goal is to embed general text prompt features. Based on weight information The weighted sum of the embedding features of each general text prompt is obtained, i.e. This allows the model to dynamically combine different expert general knowledge based on the specific image content, thereby adapting to different instances and improving the model's adaptability.

[0094] Furthermore, such as Figure 3 As shown, the DCF module may also include a second sub-model 302, such as Figure 3As shown in module 3021, class-specific features are part of the second sub-model 302. These features maintain target-specific text cue embedding features for each image category, are applicable only to a single image category, and are learnable. The second sub-model 302 may also include cue assembly logic unit 3022, which can concatenate (e.g., ...) text cue embedding features. Figure 3 (As shown by symbol c) Target-specific text hint embedding features and target-specific text hint embedding features. This allows for highly semantically adaptive features for different instances and image categories.

[0095] In some embodiments, to further enhance the expressiveness of the prompts, the text prompt embedding features can be modulated at a more refined instance level. The step of integrating the target general text prompt embedding features and the target specific text prompt embedding features of the target image category based on the second sub-model, respectively targeting the first image sample and the second image sample, to obtain the target text embedding features of the target image category, includes:

[0096] For the first image sample and the second image sample respectively, based on the first visual feature, the second sub-model is used to generate a first offset feature for the target general text prompt embedding feature and a second offset feature for the target specific text prompt embedding feature;

[0097] By fusing the first offset feature with the target general text cue embedding feature, a first text embedding feature for the target image category is obtained; and by fusing the second offset feature with the target specific text cue embedding feature, a second text embedding feature for the target image category is obtained.

[0098] The first text embedding feature and the second text embedding feature are concatenated to obtain the third text embedding feature of the target image category. The target text embedding feature of the image samples in the training data includes the third text embedding feature.

[0099] like Figure 3 As shown, the second sub-model 302 can also include a hybrid activation function (SwiGLU) feed-forward network (FFN) that combines the Swish activation function and the gated linear unit (GLU). Two independent feed-forward networks (FFNs) based on the Swish activation function can be used, similarly... Using the target general text hint embedding feature and the target specific text hint embedding feature as input, a small offset feature is generated, which is the first offset feature. (like Figure 3The output of Swigluffn (shown in Figure 3023) and the second offset feature (like Figure 3 The output of Swigluffn (shown as 3024) is used and express.

[0100] The 3022 logic unit can also be integrated (add, such as...) Figure 3 The first offset feature (shown by the symbol +) is combined with the target general text cue embedding feature to obtain the first text embedding feature of the target image category; and fusion (add, as shown) is performed. Figure 3 The second offset feature (shown by the symbol +) and the target-specific text cue embedding feature are used to obtain the second text embedding feature of the target image category.

[0101] Next, the first and second text embedding features are concatenated to obtain the third text embedding feature for the target image category. Correspondingly, the target text cue embedding feature can be input into the third sub-model for text encoding to obtain two target text cue features for the image samples in the training data.

[0102] In some embodiments, the third text prompt embedding features can be input into a third sub-model for text encoding to obtain two target text prompt features from the image samples in the training data.

[0103] In some embodiments, to further enhance the expressiveness of the prompts, the text prompt embedding features can be assembled in a hierarchical manner. A hierarchical design can be adopted, generating shallow prompts injected into the text encoder input layer and deep prompts injected into the intermediate layer, respectively.

[0104] In some embodiments, the step of inputting the target text embedding features of the target image category into a third sub-model for text encoding to obtain two target text prompt features of the image samples in the training data includes:

[0105] The third text embedding feature is input into the first layer of the third sub-model for text encoding, and the fourth text embedding feature of the target image category is input into the second layer of the third sub-model for text encoding, thereby obtaining two target text prompt features for the image samples in the training data;

[0106] The target text embedding feature of the target image category further includes a fourth text embedding feature, which is obtained by concatenating the target general text prompt embedding feature and the target specific text prompt embedding feature. The second layer network is a deep network relative to the first layer network.

[0107] The first layer of the network is a shallow network. This shallow layer integrates all information and serves as the third text embedding feature. The third text embedding feature includes a shared component, a specific component, and instance-level modulation. It is formed by adding these components and concatenating them along a specific dimension, and is used to inject into the lowest layer of the text encoder. The third text embedding feature is used... express.

[0108] The second layer is a deep network. In some embodiments, a 7-layer deep cue design can be used. Deep cues are injected into the middle layer of the text encoder. To maintain signal stability, instance-level modulation can be omitted. Each deep layer i has a shared part and a unique part. A fourth text embedding feature is injected. express.

[0109] In some embodiments, for the MVTec-AD dataset, the total token length output by the DCF module can be 11 tokens, of which 4 are image-specific text hints. For the VisA dataset, the total token length output by the DCF module is 19 tokens, of which 5 are image-specific text hints.

[0110] In this embodiment, a DCF module is designed to achieve lightweight, instance-aware semantic adaptation. This module abandons the static, fixed text prompts found in related technologies, innovatively combining the MoE framework with target-specific text embedding features for image categories. MoE can dynamically generate general shared semantics based on the visual features of the input image, while the target-specific text embedding features are learnable and are responsible for encoding attributes specific to a particular image category when subsequently updating the target text prompt features. This design achieves high semantic adaptation to different instances and image categories without introducing heavyweight external modules, balancing performance and efficiency.

[0111] In step 103, in some embodiments, the first visual feature and two target text cue features of the first image sample in the training data can be input into the third model. The third model generates fence features by comparing the first visual feature and the two target text cue features. For example, Figure 2 As shown, the third model 203 can be an Adversarial Boundary Forging (ABF) module.

[0112] In some embodiments, samples near the boundary can be synthesized using a third model based on the first visual features of a first image sample in the training data and two target text cue features. For example, a variant of a generative adversarial network can be used to train a generator specifically for generating boundary samples, or a variational autoencoder can be used to interpolate in random directions in the latent space of normal samples in order to find boundary points. Accordingly, fence features are generated based on samples near the boundary.

[0113] In some embodiments, instead of generating new boundary samples, the first visual features of the first image sample in the training data and the features of two target text prompts are input into a third model. The third model indirectly enhances the boundary by constructing a loss function for the original samples, thereby obtaining the fence features. For example, a dynamic boundary distance loss can be used to adaptively adjust the minimum distance from the normal image sample to the decision boundary during training; or a spherical space-based loss function (such as a variant of SphereFace Loss) can be used to force the features of all normal image samples to be distributed within a hypersphere with a smaller radius and clearer boundaries.

[0114] In some embodiments, the third model can receive global visual features of the image and actively synthesize fence features located on the decision boundary in the feature space through adversarial attacks such as PGD adversarial attacks. These fence features can be used to train the second model and the first model, so that the second model can sculpt a clear and sharp decision boundary in the text space, and the first model in the visual space.

[0115] In some embodiments, step 103 specifically includes:

[0116] The first visual feature is input into the third model for iterative perturbation to generate adversarial features of the first visual feature;

[0117] Based on the adversarial features and the two target text prompt features, a second loss of the third model is generated. The second loss includes the balance loss of the adversarial features relative to the two target text prompt features and the discrete loss between different adversarial features.

[0118] The third model is updated based on the second loss so that the third model generates fence features.

[0119] The third model can include a Projected Gradient Descent (PGD) attacker. Figure 4 This is a diagram illustrating the adversarial attack of the third model, such as... Figure 4 As shown, the PGD attack algorithm can be used to attack the input. Multiple iterative perturbations are performed to generate adversarial features of the first visual feature. The optimization objective of a PGD attacker is to minimize a composite loss function. This refers to the second loss, which can be based on adversarial features. First text prompt features Second text prompt features The generated adversarial features are precisely moved to the decision boundary by the second loss.

[0120] The second loss of the third model can be used as follows: This indicates that β is a hyperparameter used to balance the two loss terms.

[0121] The second loss can include a balance loss of the adversarial features relative to the two target text prompt features. ,use The goal of balancing the loss is to make the adversarial features... Features of the first text prompt Second text prompt features The cosine similarity is exactly equal. By minimizing the absolute value of this difference, the features are effectively pushed onto the decision hyperplane defined by the two semantic concepts, thus forming fence features.

[0122] The second loss can also include discrete losses between different adversarial features. The goal of discrete loss is to prevent all generated boundary features from collapsing to the same point. This loss term encourages the adversarial features to be as dispersed as possible. It achieves this by minimizing the negative of the average L2 distance between each pair of N adversarial features, i.e., maximizing their average distance. express.

[0123] Figure 5 The three sub-images vividly illustrate the differences in the quality of decision boundaries formed by different technical solutions in the feature space. The squares represent the features of the first image sample, the triangles represent the features of the second image sample, the solid-lined circle represents the true ideal boundary, and the area within the dashed-lined circle represents the decision boundary learned by the model.

[0124] In related technologies, the original CLIP model, such as Figure 5 As shown in the middle left figure, the decision boundary (dashed circular area) is too loose, much larger than the true boundary (solid circular area), which causes a large number of abnormal features (features represented by triangles) to be incorrectly included, resulting in missed detections.

[0125] In related technologies, the decision boundaries are often poorly defined or overly restrictive, such as... Figure 5As shown in the middle image, its boundaries are irregular and poorly match the real boundaries. This may exclude some normal image samples of atypical objects, leading to false alarms, and it also cannot effectively identify all abnormal image samples of objects.

[0126] The fence features formed in this embodiment, such as Figure 5 As shown in the right-middle figure, the model can proactively generate fence features near the real boundary, represented by diamonds. By training with these synthetic image samples precisely located on the boundary, the model is forced to learn a clear and sharp decision boundary (dashed circle) that closely matches the real boundary (i.e., the solid-lined circular region), thereby enabling extremely accurate differentiation between normal and abnormal features and achieving highly robust detection.

[0127] This embodiment proposes an Adversarial Boundary Forging (ABF) mechanism for actively and explicitly sculpting decision boundaries. This mechanism iteratively synthesizes fence features precisely positioned between normal and anomalous semantic concepts in the feature space using adversarial attack methods such as projected gradient descent. This process can be guided by a dual loss function (including balanced loss and discrete loss) to ensure that the synthesized boundary samples are both semantically centered and diverse. By utilizing these high-quality boundary samples to train the model, the technical pain points of ambiguous and passively formed decision boundaries in related technologies are fundamentally addressed.

[0128] In step 104, the first loss of the second model can be constructed based on the fence features and the two target text prompt features.

[0129] In some embodiments, the first loss may include a balance loss of the fence features relative to the two target text cue features.

[0130] In some embodiments, the first loss includes at least one of the following:

[0131] The balance loss of the fence feature relative to the two target text prompt features;

[0132] The classification uncertainty loss for the fence feature is measured by the Shannon entropy of the predicted probability of the fence feature being classified into the two target text prompt features.

[0133] In other words, in some embodiments, two additional parameters can be added to the main loss function of the second model, i.e., the first loss. The regularization objectives are the balance loss of the fence feature relative to the two target text prompt features and the classification uncertainty loss for the fence feature.

[0134] In some embodiments, the first loss of the second model may include the following:

[0135] Pixel-level segmentation loss involves synthesizing images of abnormal objects by cutting / pasting from normal image samples, and then calculating Focal Loss and Dice Loss to provide pixel-level supervision signals. Focal Loss primarily addresses the imbalance between normal and abnormal samples, while Dice Loss mainly measures the degree of overlap between the predicted and ground truth regions in the segmentation task.

[0136] The cue alignment loss can align the generated text cue with normal visual features and a set of manually defined text cues, respectively.

[0137] The training objective of the fence feature's balance loss relative to the two target text cue features is to minimize the effect on the The balance loss on the boundary forces the parameters of the second and third models to be adjusted so that their decision boundaries pass precisely through these adversarial points, thus making the local geometry of the boundary smoother and more stable.

[0138] The training objective for the classification uncertainty loss of the aforementioned fence features is to maximize its value for... The classification uncertainty can be measured using the Shannon entropy of the predicted probability; therefore, this loss term is negative entropy. The objective is to minimize the negative entropy, i.e., maximize the entropy. This objective causes the model to form a low-confidence buffer zone around the decision boundary, which in turn makes the model more confident in predicting whether an image sample of a clearly normal or abnormal object is far from the boundary, thus sharpening the entire decision boundary.

[0139] In some embodiments, visual features, two target text cue features, and fence features generated by the ABF module are collectively incorporated into an objective function module, such as... Figure 2 Module 204 is shown. This module may contain components such as Balance Loss and Discretion Loss, which are used to calculate the model training loss and guide the optimization of the entire model.

[0140] In some embodiments, such as Figure 6 As shown, P represents the first text prompt feature. N represents the second text prompt feature. Squares represent normal features, triangles represent abnormal features, rhombuses represent fence features, circles represent feature anomalous synthesis, stars represent pixel anomalous synthesis, and double arrows indicate pushing the synthesized anomalous image and synthesized anomalous features away from the first text prompt feature. Pull the normal features closer to the first text prompt feature. And thus push the normal features away from the second text prompt features. The synthesized abnormal image and synthesized abnormal features are then pulled closer to the second text prompt features. Crossed double arrows indicate adversarial features. Features of the first text prompt Second text prompt features The cosine similarity is completely equal, balancing the distance between the two. Thus, through multiple iterative attacks, guided by the second loss, the final fence features can be generated, and guided by the first loss, the parameters of the DCF module can be updated, that is, the two target text prompt features can be updated, so that the DCF module can generate adaptive text prompts that are highly correlated with the instance content and image category of the input image, resulting in the final two target text prompt features.

[0141] It should be noted that the target model may also include a pixel-level anomaly synthesis module, which is used to synthesize anomalous images during training to provide stronger supervision signals. For example, this module can synthesize anomalous images and anomalous features, such as... Figure 2 As shown, abnormal image 205 can be generated by the pixel-level abnormal synthesis module, and abnormal image refers to an image of an object that is abnormal.

[0142] This embodiment employs a decoupled two-stage training strategy. First, the second model is trained. During this stage, all parameters of the first model are frozen, and only the parameters of the DCF module are updated. The second stage involves training the first model. In this stage, all parameters of the DCF module trained in the first stage are frozen. Simultaneously, the top-level portion of the first model is unfrozen; in some embodiments, this could be the last four Transformer blocks, the LN post layer, and the projection layer.

[0143] In some embodiments, in step 105, the target loss function of the first model can be the same as that of the second model, and the unfrozen visual layer in the first model can be fine-tuned. This stage aims to allow the visual feature space to actively adapt to the solidified high-quality semantic concepts and complete the final, refined decision boundary sculpting.

[0144] This embodiment proposes a decoupled two-stage training framework to resolve the conflict between adaptive and discriminative optimization. By clearly dividing the training process into two independent stages, the first stage freezes the visual encoder, focusing on training the DCF module to learn high-quality semantic concepts; the second stage freezes the DCF module, using the solidified high-quality concepts to fine-tune the higher levels of the visual encoder, completing the refined shaping of the feature space. This decoupled architecture of learning concepts first and then tuning features avoids the mutual constraint of two conflicting optimization objectives in end-to-end training, which is key to achieving high performance.

[0145] Once the target model has been trained, anomaly detection of objects in an image can be performed using two target text cue features from the text memory and a second visual feature from the first image sample in the visual memory. The inference process is as follows:

[0146] Step 1: Image category recognition.

[0147] For each trained image category k, a multivariate normal distribution parameter (mean) can be pre-calculated and stored based on the global visual features of the first image sample. Covariance For an image to be detected, the log-likelihood of its global visual features under each image category distribution can be calculated, and the image category with the highest log-likelihood is identified. It is identified as the most likely image category to which the image belongs.

[0148] Step 2: Calculation of anomaly scores in memory bank fusion.

[0149] Once the image category is determined The system can retrieve the visual memory and text memory corresponding to the image category. It can calculate the similarity between the image patch features of the image to be detected and the two target text cue features in the text memory, obtaining a text anomaly score. It can also calculate the distance between the image patch features of the image to be detected and the nearest neighbor second visual feature in the visual memory, obtaining a visual anomaly score. Finally, it can perform a weighted sum of the text and visual anomaly scores for each image patch to obtain the final pixel-level anomaly map. The maximum value on this pixel-level anomaly map is the final image-level anomaly score for the image to be detected. In this way, anomaly detection of objects in an image can be achieved.

[0150] This application addresses key technical challenges in few-shot anomaly detection, such as ambiguous decision boundaries, poor model robustness, inability of static prompts to adapt to complex scenarios, and difficulty in balancing model complexity and performance. It propels few-shot anomaly detection technology towards greater accuracy, robustness, and efficiency. Compared to existing technologies, it possesses significant and fundamental technical advantages, primarily in the following aspects:

[0151] 1. By actively sculpting boundaries, the robustness of detection is fundamentally improved.

[0152] In related technologies, the use of static or globally shared prompts and passive training methods results in unclear decision boundaries. This makes the model prone to misjudgment when faced with atypical normal images that differ from the training samples or subtle anomalies with indistinct defects, leading to poor robustness. This application's embodiment introduces an ABF module. This module synthesizes fence features on the decision boundary through PGD adversarial attacks and uses balanced loss and discrete loss to guide model training. This proactive rather than passive learning approach forces the model to form an extremely clear and sharp decision boundary, fundamentally solving the boundary ambiguity problem and significantly improving detection accuracy and overall model robustness.

[0153] 2. The decoupled training architecture effectively resolves the optimization conflict between adaptability and discriminability.

[0154] In related technologies, end-to-end joint training is commonly used. This requires the model to simultaneously optimize two conflicting objectives: ensuring the distribution area of ​​normal features is large enough to accommodate all normal variations, and ensuring the boundary between normal and abnormal features is clear enough to achieve high discriminativity. These two objectives interfere with each other during optimization, resulting in the final model performance being a compromise between the two, making it difficult to achieve optimal performance. This application's embodiment employs a decoupled two-stage training framework. The first stage focuses on learning semantic concepts, aiming to enable the DCF module to generate highly discriminative text prompts; the second stage focuses on fine-tuning visual features, aiming to align these high-quality semantic concepts with the visual space. This separate optimization strategy effectively avoids conflicts between optimization objectives, allowing the model to first ensure discriminativity, and then expand its adaptability on this basis, ultimately achieving comprehensive performance far exceeding that of integrated training schemes.

[0155] 3. Through a lightweight dynamic prompt design, highly efficient semantic adaptation is achieved.

[0156] In related technologies, to address the limitations of static prompts, additional, pre-trained heavyweight modules such as Q-Former are introduced to enhance the model's adaptability. While this improves performance, it also significantly increases the number of model parameters, computational overhead, and deployment complexity. This application's embodiment utilizes a DCF module, which offers a more efficient solution. Through a clever combination of the MoE mechanism and dedicated text embedding features for image categories, it achieves powerful instance awareness and category adaptation capabilities with minimal parameter increments. Therefore, it avoids dependence on large external modules, maintaining a lightweight model while achieving high performance, demonstrating excellent power efficiency and making it more suitable for deployment in real-world industrial scenarios.

[0157] The embodiments of this application demonstrate significant commercial value and broad market application prospects in the current wave of intelligent and flexible transformation in the manufacturing industry. However, the development of traditional AI quality inspection models faces two major bottlenecks: data scarcity and long development cycles, which are particularly prominent in industries with a wide variety of products and rapid product updates (such as consumer electronics and semiconductors).

[0158] This application embodiment directly addresses the aforementioned industry pain points by enabling the rapid deployment of a high-precision detection model with only a single-digit number of normal image samples, thus possessing the following clear commercial application value:

[0159] 1. Application in multi-variety, small-batch production mode: For high value-added manufacturing industries such as semiconductors, aerospace, and precision instruments, products are often highly customized. It can quickly build a dedicated quality inspection model for each batch of new products or designs, which greatly improves the flexibility of the production line and the speed of response to market demands.

[0160] 2. Rapid iteration support for consumer electronics product lines: In fields such as smartphones and wearable devices, product lifecycles are extremely short. This allows companies to deploy corresponding defect detection models on the production line on the same day they release a new product, eliminating the need to spend weeks or even months collecting thousands of samples for model training, thus greatly shortening the time from R&D to mass production.

[0161] 3. Lower the threshold for AI applications for SMEs. Many small and medium-sized manufacturing enterprises often lack the ability and resources to acquire large-scale labeled data. This will enable them to enjoy high-level AI quality inspection capabilities at a very low cost by using limited samples, thus promoting the popularization of AI technology in the manufacturing industry.

[0162] 4. Empowering AI-powered Quality Inspection Platform as a Service (PaaS): This can be integrated as a core algorithm into cloud-based AI industrial quality inspection platforms. Platform service providers can offer online services to clients across various industries, allowing them to upload small samples and automatically generate models, creating entirely new business models.

[0163] The embodiments of this application not only solve key technical bottlenecks but also directly address the urgent needs of the industry. They offer significant advantages in improving production efficiency, reducing quality inspection costs, and accelerating product iteration, demonstrating the potential to become the core engine of the next-generation intelligent industrial vision platform and possessing broad commercial prospects.

[0164] See Figure 7 , Figure 7 This is a structural diagram of a model training device provided in an embodiment of this application, as shown below. Figure 7 As shown, the model training device 700 includes:

[0165] The visual feature extraction module 701 is used to input training data into the first model of the target model to extract visual features and obtain the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image.

[0166] The feature alignment module 702 is used to generate text features aligned with the first visual feature based on the first visual feature and using the second model in the target model to obtain two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal.

[0167] The generation module 703 is used to generate fence features based on the first visual features of the first image sample in the training data and the two target text prompt features, using the third model in the target model. The first image sample is an image sample with normal objects, and the fence features are used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects.

[0168] Construction module 704 is used to construct a first loss of the second model based on the fence features and the two target text prompt features, wherein the first loss is used to update the two target text prompt features;

[0169] The training module 705 is used to train the first model based on the fence features and the two target text prompt features after the second model has been trained. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

[0170] Optionally, the second model includes a first sub-model, a second sub-model, and a third sub-model, and the feature alignment module 702 is specifically used for:

[0171] Based on the first sub-model, the first visual feature is aligned with the pre-stored general text prompt embedding features for each image category to obtain two target general text prompt embedding features for the target image category corresponding to the image sample in the training data.

[0172] For the first image sample and the second image sample respectively, based on the second sub-model, the target general text prompt embedding feature of the target image category and the target specific text prompt embedding feature of the target image category are integrated to obtain the target text embedding feature of the target image category;

[0173] The target text embedding features of the target image category are input into the third sub-model for text encoding to obtain two target text prompt features of the image samples in the training data. The two target text prompt features correspond to the target image category.

[0174] Optionally, the feature alignment module 702 is further configured to:

[0175] The first sub-model outputs weight information based on the first visual feature; and performs weighted processing on the weight information and each general text prompt embedding feature to obtain the target general text prompt embedding feature of the target image category, wherein the weight information is used to indicate the degree of alignment between the first visual feature and each general text prompt embedding feature.

[0176] Optionally, the feature alignment module 702 is further configured to:

[0177] For the first image sample and the second image sample respectively, based on the first visual feature, the second sub-model is used to generate a first offset feature for the target general text prompt embedding feature and a second offset feature for the target specific text prompt embedding feature;

[0178] By fusing the first offset feature with the target general text cue embedding feature, a first text embedding feature for the target image category is obtained; and by fusing the second offset feature with the target specific text cue embedding feature, a second text embedding feature for the target image category is obtained.

[0179] The first text embedding feature and the second text embedding feature are concatenated to obtain the third text embedding feature of the target image category. The target text embedding feature of the image samples in the training data includes the third text embedding feature.

[0180] Optionally, the feature alignment module 702 is further configured to:

[0181] The third text embedding feature is input into the first layer of the third sub-model for text encoding, and the fourth text embedding feature of the target image category is input into the second layer of the third sub-model for text encoding, thereby obtaining two target text prompt features for the image samples in the training data;

[0182] The target text embedding feature of the target image category further includes a fourth text embedding feature, which is obtained by concatenating the target general text prompt embedding feature and the target specific text prompt embedding feature. The second layer network is a deep network relative to the first layer network.

[0183] Optionally, the generation module 703 is specifically used for:

[0184] The first visual feature is input into the third model for iterative perturbation to generate adversarial features of the first visual feature;

[0185] Based on the adversarial features and the two target text prompt features, a second loss of the third model is generated. The second loss includes the balance loss of the adversarial features relative to the two target text prompt features and the discrete loss between different adversarial features.

[0186] The third model is updated based on the second loss so that the third model generates fence features.

[0187] Optionally, the first loss includes at least one of the following:

[0188] The balance loss of the fence feature relative to the two target text prompt features;

[0189] The classification uncertainty loss for the fence feature is measured by the Shannon entropy of the predicted probability of the fence feature being classified into the two target text prompt features.

[0190] The model training device 700 can implement all the processes implemented in the above-described model training method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0191] See Figure 8 The figure shows a structural diagram of an electronic device provided in an embodiment of the present invention. Figure 8 As shown, the electronic device 800 includes: a processor 801, a memory 802, a user interface 803, and a bus interface 804.

[0192] Processor 801 is used to read the program from memory 802 and execute the following procedures:

[0193] The training data is input into the first model of the target model to extract visual features, thereby obtaining the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image.

[0194] Based on the first visual feature, text features aligned with the first visual feature are generated using the second model in the target model, thereby obtaining two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal.

[0195] Based on the first visual features of the first image sample in the training data and the two target text prompt features, a fence feature is generated using the third model in the target model. The first image sample is an image sample with normal objects, and the fence feature is used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects.

[0196] Based on the fence features and the two target text prompt features, a first loss of the second model is constructed, and the first loss is used to update the two target text prompt features.

[0197] Once the second model has been trained, the first model is trained based on the fence features and the two target text prompt features. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

[0198] exist Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 801 and memory represented by memory 802 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 804 provides an interface. For different user devices, user interface 803 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0199] The processor 801 is responsible for managing the bus architecture and general processing, while the memory 802 can store the data used by the processor 801 when performing operations.

[0200] In some embodiments, the second model includes a first sub-model, a second sub-model, and a third sub-model, and the processor 801 is further configured to:

[0201] Based on the first sub-model, the first visual feature is aligned with the pre-stored general text prompt embedding features for each image category to obtain two target general text prompt embedding features for the target image category corresponding to the image sample in the training data.

[0202] For the first image sample and the second image sample respectively, based on the second sub-model, the target general text prompt embedding feature of the target image category and the target specific text prompt embedding feature of the target image category are integrated to obtain the target text embedding feature of the target image category;

[0203] The target text embedding features of the target image category are input into the third sub-model for text encoding to obtain two target text prompt features of the image samples in the training data. The two target text prompt features correspond to the target image category.

[0204] In some embodiments, the text alignment includes: the first sub-model outputting weight information based on the first visual feature; and weighting the weight information with each general text cue embedding feature to obtain the target general text cue embedding feature of the target image category, wherein the weight information is used to indicate the degree of alignment between the first visual feature and each general text cue embedding feature.

[0205] In some embodiments, the processor 801 is further configured to:

[0206] For the first image sample and the second image sample respectively, based on the first visual feature, the second sub-model is used to generate a first offset feature for the target general text prompt embedding feature and a second offset feature for the target specific text prompt embedding feature;

[0207] By fusing the first offset feature with the target general text cue embedding feature, a first text embedding feature for the target image category is obtained; and by fusing the second offset feature with the target specific text cue embedding feature, a second text embedding feature for the target image category is obtained.

[0208] The first text embedding feature and the second text embedding feature are concatenated to obtain the third text embedding feature of the target image category. The target text embedding feature of the image samples in the training data includes the third text embedding feature.

[0209] In some embodiments, the processor 801 is further configured to:

[0210] The third text embedding feature is input into the first layer of the third sub-model for text encoding, and the fourth text embedding feature of the target image category is input into the second layer of the third sub-model for text encoding, thereby obtaining two target text prompt features for the image samples in the training data;

[0211] The target text embedding feature of the target image category further includes a fourth text embedding feature, which is obtained by concatenating the target general text prompt embedding feature and the target specific text prompt embedding feature. The second layer network is a deep network relative to the first layer network.

[0212] In some embodiments, the processor 801 is further configured to:

[0213] The first visual feature is input into the third model for iterative perturbation to generate adversarial features of the first visual feature;

[0214] Based on the adversarial features and the two target text prompt features, a second loss of the third model is generated. The second loss includes the balance loss of the adversarial features relative to the two target text prompt features and the discrete loss between different adversarial features.

[0215] The third model is updated based on the second loss so that the third model generates fence features.

[0216] In some embodiments, the first loss includes at least one of the following:

[0217] The balance loss of the fence feature relative to the two target text prompt features;

[0218] The classification uncertainty loss for the fence feature is measured by the Shannon entropy of the predicted probability of the fence feature being classified into the two target text prompt features.

[0219] Preferably, the present invention also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the computer program is executed by the processor 801, it implements the various processes of the above-described model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0220] This invention also provides a readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described model training method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0221] This application also provides a computer program product, including computer instructions. When these computer instructions are executed by a processor, they implement the various processes of the above-described model training method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0222] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0223] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0224] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0226] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0227] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0228] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method includes: The training data is input into the first model of the target model to extract visual features, thereby obtaining the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image. Based on the first visual feature, text features aligned with the first visual feature are generated using the second model in the target model, thereby obtaining two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal. Based on the first visual features of the first image sample in the training data and the two target text prompt features, a fence feature is generated using the third model in the target model. The first image sample is an image sample with normal objects, and the fence feature is used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects. Based on the fence features and the two target text prompt features, a first loss of the second model is constructed, and the first loss is used to update the two target text prompt features. Once the second model has been trained, the first model is trained based on the fence features and the two target text prompt features. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

2. The method of claim 1, wherein, The second model includes a first sub-model, a second sub-model, and a third sub-model. Based on the first visual feature, the second model in the target model generates text features aligned with the first visual feature, resulting in two target text prompt features for the image samples in the training data. These features include: Based on the first sub-model, the first visual feature is aligned with the pre-stored general text prompt embedding features for each image category to obtain two target general text prompt embedding features for the target image category corresponding to the image sample in the training data. For the first image sample and the second image sample respectively, based on the second sub-model, the target general text prompt embedding feature of the target image category and the target specific text prompt embedding feature of the target image category are integrated to obtain the target text embedding feature of the target image category; The target text embedding features of the target image category are input into the third sub-model for text encoding to obtain two target text prompt features of the image samples in the training data. The two target text prompt features correspond to the target image category.

3. The method of claim 2, wherein, The text alignment includes: the first sub-model outputs weight information based on the first visual feature; and the weight information is weighted with each general text prompt embedding feature to obtain the target general text prompt embedding feature of the target image category, wherein the weight information is used to indicate the degree of alignment between the first visual feature and each general text prompt embedding feature.

4. The method of claim 2, wherein, The step involves, for the first image sample and the second image sample respectively, integrating the target general text cue embedding features and the target specific text cue embedding features of the target image category based on the second sub-model to obtain the target text embedding features of the target image category, including: For the first image sample and the second image sample respectively, based on the first visual feature, the second sub-model is used to generate a first offset feature for the target general text prompt embedding feature and a second offset feature for the target specific text prompt embedding feature; By fusing the first offset feature with the target general text cue embedding feature, a first text embedding feature for the target image category is obtained; and by fusing the second offset feature with the target specific text cue embedding feature, a second text embedding feature for the target image category is obtained. The first text embedding feature and the second text embedding feature are concatenated to obtain the third text embedding feature of the target image category. The target text embedding feature of the image samples in the training data includes the third text embedding feature.

5. The method according to claim 4, characterized in that, The step of inputting the target text embedding features of the target image category into the third sub-model for text encoding yields two target text prompt features for the image samples in the training data, including: The third text embedding feature is input into the first layer of the third sub-model for text encoding, and the fourth text embedding feature of the target image category is input into the second layer of the third sub-model for text encoding, thereby obtaining two target text prompt features for the image samples in the training data; The target text embedding feature of the target image category further includes a fourth text embedding feature, which is obtained by concatenating the target general text prompt embedding feature and the target specific text prompt embedding feature. The second layer network is a deep network relative to the first layer network.

6. The method according to claim 1, characterized in that, The step of generating fence features based on the first visual features of the first image sample in the training data and the two target text prompt features, using the third model in the target model, includes: The first visual feature is input into the third model for iterative perturbation to generate adversarial features of the first visual feature; Based on the adversarial features and the two target text prompt features, a second loss of the third model is generated. The second loss includes the balance loss of the adversarial features relative to the two target text prompt features and the discrete loss between different adversarial features. The third model is updated based on the second loss so that the third model generates fence features.

7. The method according to claim 1, characterized in that, The first loss includes at least one of the following: The balance loss of the fence feature relative to the two target text prompt features; The classification uncertainty loss for the fence feature is measured by the Shannon entropy of the predicted probability of the fence feature being classified into the two target text prompt features.

8. A model training device, characterized in that, The device includes: The visual feature extraction module is used to input training data into the first model of the target model to extract visual features and obtain the first visual features of the image samples in the training data. The target model is used to perform anomaly detection of objects in the image. The feature alignment module is used to generate text features aligned with the first visual feature based on the first visual feature and using the second model in the target model, thereby obtaining two target text prompt features for the image samples in the training data. The two target text prompt features include: a first text prompt feature for indicating that the object in the image sample is normal and a second text prompt feature for indicating that the object in the image sample is abnormal. The generation module is used to generate fence features based on the first visual features of the first image sample in the training data and the two target text prompt features, using the third model in the target model. The first image sample is an image sample with normal objects, and the fence features are used to indicate the decision boundary between the first image sample and the second image sample. The second image sample is an image sample with abnormal objects. A construction module is used to construct a first loss for the second model based on the fence features and the two target text prompt features, wherein the first loss is used to update the two target text prompt features; The training module is used to train the first model based on the fence features and the two target text prompt features after the second model has been trained. The two target text prompt features are used to perform anomaly detection of objects in the image by combining the second visual features of the first image sample extracted by the first model after training.

9. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the model training method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the model training method as described in any one of claims 1 to 7.

Images