An industrial anomaly detection method based on multi-agent hint learning

Abstract

The application provides an industrial anomaly detection method based on multi-agent prompt learning, which performs the following processing through a multi-agent collaborative framework and an improved multi-modal model: an image annotator agent is used to analyze an input industrial image, and an item class name is generated through a multi-modal large model; a description generator agent is used to generate normal visual guide descriptions and abnormal visual guide descriptions based on the item class name; an improved multi-modal pre-training model is used to extract visual features of the industrial image; through semantic connection strengthening processing, the text features of the normal visual guide descriptions are aligned with the visual features of normal samples, and the text features of the abnormal visual guide descriptions are far away from the visual features of the normal samples; a similarity is calculated based on the improved multi-modal pre-training model, and a detection result is output; an anomaly detection evaluator agent is used to calculate an anomaly index AUROC as a feedback signal, and the above processing is iteratively performed until a preset condition is met.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of artificial intelligence and industrial inspection, specifically relating to an industrial anomaly detection method based on multi-agent prompting learning. Background Technology

[0002] Industrial anomaly detection, a crucial link in ensuring the quality of industrial production, has long faced technical bottlenecks in complex scenarios. In actual industrial manufacturing, the scarcity of anomaly samples not only leads to a severe imbalance in data distribution but also makes it difficult for models to learn representative anomaly patterns. In most cases, normal samples account for the vast majority of the total data, while anomaly samples may only account for a few per thousand or even a few per ten thousand. This extremely imbalanced data distribution makes traditional supervised learning methods prone to the dilemma of "overfitting to normal samples," resulting in a persistently high rate of missed detections for rare anomalies.

[0003] Meanwhile, the manifestations of anomalies in industrial settings are highly diverse and difficult to detect. From microscopic surface scratches and material impurities to macroscopic structural deformations, assembly misalignments, and even subtle changes in visual features such as color deviations and texture anomalies, all can point to potential quality problems. These anomalies are often closely related to the manufacturing process and material properties of specific products. For example, solder joint defects in electronic components and surface corrosion on automotive parts have significantly different characteristics, making it difficult for a single model to cover the testing needs of multiple types of industrial products, thus limiting its versatility.

[0004] Traditional anomaly detection methods have significant limitations in addressing these challenges. Early rule-based methods relied on manually set feature thresholds, which could only handle simple scenarios and were almost ineffective against complex anomalies. Classical machine learning methods, while extracting handcrafted features (such as edges and textures) through feature engineering, relied excessively on domain expert experience and struggled to capture the implicit nonlinear relationships in high-dimensional images, making them poorly adaptable to the inspection of new industrial products. Even the deep learning methods that have emerged in recent years mostly require large-scale labeled data for training, while the labeling cost of anomaly samples in industrial scenarios is extremely high—not only do professional quality inspectors need to judge each sample individually, but some subtle defects also require the assistance of precision instruments for identification, making data acquisition a major obstacle in practical applications.

[0005] The rise of visual-language models has provided a new approach to recognition tasks in low-sample scenarios. By associating visual features with textual semantics, they aim to guide the model to focus on key features using natural language descriptions, reducing reliance on labeled data. However, transferring these models to industrial anomaly detection still faces unique challenges: the technical terminology system in the industrial field differs from the pre-training corpus of general-purpose language models, resulting in insufficient alignment accuracy between textual descriptions and visual features; simultaneously, the subtlety of industrial anomalies requires models to have extremely strong local feature discrimination capabilities, while existing visual-language models mostly focus on global semantic matching, with limited sensitivity to subtle local differences, making it difficult to accurately locate and describe minute defects. Furthermore, dynamic changes in different production lines (such as lighting conditions and image style shifts caused by equipment wear) further exacerbate the difficulty of model adaptation, significantly reducing the generalization performance of general-purpose visual-language models in industrial scenarios.

[0006] These issues collectively constitute the technical pain points in the field of industrial anomaly detection: how to achieve efficient and accurate anomaly detection without extensive human intervention in industrial scenarios with few samples, high diversity, and strong dynamism remains a core challenge that existing technologies have not yet fully resolved. Summary of the Invention

[0007] To address the shortcomings and deficiencies of existing technologies, this invention provides an industrial anomaly detection method based on multi-agent cue learning, aiming to solve problems in industrial anomaly detection such as reliance on large amounts of labeled data, high cost of manually designing cue words, weak model generalization ability, and insufficient accuracy in detecting local subtle anomalies.

[0008] This method constructs a multi-agent collaborative framework. An image annotator agent parses industrial images to generate item class names, while a description agent automatically generates normal visual guidance descriptions containing learnable prefix sequences based on the class names, as well as abnormal visual guidance descriptions that fuse learnable prefixes, class names, and anomaly-related suffixes, reducing human intervention. An improved multimodal pre-trained model is introduced, employing a dual-value self-attention mechanism based on value vector self-similarity calculation to enhance attention to local features and improve the accuracy of subtle anomaly detection. Semantic connectivity is used to strengthen processing, aligning the textual features of normal visual guidance descriptions with the visual features of normal samples, while keeping the features of abnormal descriptions away from normal sample features, thus optimizing feature matching. An anomaly detection evaluator agent calculates an anomaly index (AUROC) as a feedback signal, driving the preceding agent to iteratively adjust the generation logic until preset conditions are met.

[0009] This method is suitable for scenarios with few samples, requiring fewer than four normal samples to achieve detection, and supports the application of a single normal sample. While improving the efficiency and accuracy of anomaly detection, it also enhances the adaptability to different industrial products and new fields.

[0010] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0011] An industrial anomaly detection method based on multi-agent cue learning performs the following processing through a multi-agent collaborative framework and an improved multimodal model:

[0012] The input industrial image is parsed using an image annotation agent, and item class names are generated through a multimodal large model.

[0013] Based on the item class name, the member intelligent agent generates normal visual guidance description and abnormal visual guidance description. The normal visual guidance description consists of a learnable prefix sequence and the item class name, while the abnormal visual guidance description includes a learnable prefix sequence, the item class name, and abnormal related suffixes.

[0014] Visual features of industrial images are extracted using an improved multimodal pre-trained model, and local feature representation is enhanced through value vector self-similarity calculation.

[0015] Through semantic connection enhancement processing, the text features of normal visual guidance descriptions are aligned with the visual features of normal samples, while the text features of abnormal visual guidance descriptions are moved away from the visual features of normal samples.

[0016] The similarity between visual features and normal and abnormal visual guided description features is calculated based on an improved multimodal pre-trained model. When the similarity of normal features is lower than a preset threshold, it is judged as abnormal and the detection result is output.

[0017] The anomaly detection evaluator agent calculates the anomaly index AUROC as a feedback signal, which drives the image annotator agent to adjust the item category name generation logic and the description generation agent to optimize the visually guided description generation logic. The above processing is executed iteratively until the preset conditions are met.

[0018] Furthermore, the image labeler agent and the description agent in the multi-agent collaborative framework are both implemented based on a multimodal large model, which is used to automatically generate item class names and visual guidance description prompts.

[0019] Furthermore, the specific structure of the normal visual guidance description is as follows: it consists of a learnable prefix sequence concatenated with an item category name, wherein the prefix sequence contains multiple prompt words, the sequence length is a learnable parameter, and the learnable parameter is optimized through model training.

[0020] Furthermore, the abnormal visual guidance description includes two types:

[0021] The first category consists of a learnable prefix sequence, an item class name, and abnormal attribute connectors concatenated together. The abnormal attribute connectors include words indicating feature association, words indicating color features, and other abnormal attribute descriptive words.

[0022] The second category consists of a learnable prefix sequence, an item class name, and a learnable anomaly suffix sequence concatenated together. The learnable anomaly suffix sequence contains multiple anomaly feature descriptors, and the sequence length is a learnable parameter.

[0023] Furthermore, the value vector self-similarity calculation of the improved multimodal pre-trained model is specifically a two-valued self-attention mechanism, which generates attention weights by the similarity distribution between the value vector and its transpose matrix, and then aggregates the value vectors based on the weights to enhance the local feature expression; wherein, the similarity distribution is calculated by the softmax function, and a scaling factor is introduced in the calculation process to adjust the similarity scale.

[0024] Furthermore, the semantic connection enhancement processing achieves feature alignment optimization through a contrastive loss function. This loss function strengthens the similarity between text features of normal visual guidance descriptions and normal visual features, while weakening the similarity between text features of abnormal visual guidance descriptions and normal visual features, thereby achieving feature alignment. Specifically, the loss function adjusts the intensity of feature alignment through exponential and logarithmic operations based on the similarity between the mean of normal visual features and normal text features, as well as the similarity between normal visual features and all abnormal text features. A temperature hyperparameter is introduced to control the weight ratio of similarity, and the mathematical expectation of normal visual features is used as the calculation benchmark.

[0025] The mean of the normal text features is the average value of the features extracted by the text encoder for all normal visual guidance descriptions, and the set of abnormal text features is the set of features extracted by the text encoder for all abnormal visual guidance descriptions.

[0026] Furthermore, the preset threshold is 0.7. When the similarity between the visual features and the normal visual guidance description features is higher than 0.7, it is judged as normal; otherwise, it is judged as abnormal.

[0027] When the anomaly detection evaluator agent is below the empirical threshold, it sends targeted feedback signals to the image annotator agent and the description generation agent. The feedback signals include suggestions for adjusting class name accuracy and description logic deviation to drive the optimization of generation logic.

[0028] Furthermore, the improved multimodal pre-trained model adopts a dual-path feature fusion strategy: one path retains the original feature extraction structure, while the other path replaces it with the dual-value self-attention mechanism starting from a specific network depth, so as to simultaneously retain global and local features.

[0029] Furthermore, an industrial anomaly detection system based on multi-agent cue learning includes:

[0030] The multi-agent collaborative module includes an image annotator unit, a description generation unit, and an anomaly detection evaluator unit; among them,

[0031] The image annotator unit is used to parse the input industrial images and generate item class names through a multimodal large model;

[0032] The description member unit is used to generate normal visual guidance description and abnormal visual guidance description based on the item class name. The normal visual guidance description consists of a learnable prefix sequence and the item class name, and the abnormal visual guidance description includes a learnable prefix sequence, the item class name, and abnormal related suffixes.

[0033] The anomaly detection and evaluation unit is used to calculate the anomaly index AUROC as a feedback signal, which drives the image annotation unit to adjust the item class name generation logic and the description generation unit to optimize the visually guided description generation logic.

[0034] An improved multimodal processing module is used to extract visual features from industrial images. It enhances the expression of local features through value vector self-similarity calculation and calculates the similarity between visual features and normal and abnormal visual guided descriptive features. When the similarity of normal features is lower than a preset threshold, it is judged as abnormal and the detection result is output.

[0035] The semantic connection enhancement module is used to align the text features of normal visual guidance descriptions with the visual features of normal samples through feature alignment processing, and to keep the text features of abnormal visual guidance descriptions away from the visual features of normal samples.

[0036] The system iteratively executes the processing of the above modules until the preset conditions are met.

[0037] And an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0038] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0039] Compared to existing technologies, this invention and its preferred solution achieve automatic generation of prompt words and visual text descriptions by constructing a multi-agent collaborative framework, significantly reducing the intervention cost of manually designing prompt words and improving the model's adaptability in different industrial products and new fields. It introduces an improved multimodal pre-trained model based on value vector self-similarity calculation, strengthening the focus on subtle local features and helping to improve the detection accuracy of common subtle anomalies in industrial scenarios. Semantic connectivity enhances processing, further improving the accuracy of anomaly detection by optimizing the matching relationship between normal and abnormal features. Combined with an anomaly index-driven closed-loop feedback mechanism, the model can continuously improve performance through iterative optimization. Simultaneously, the solution is applicable to scenarios with few samples, achieving effective detection with limited normal samples, reducing dependence on large amounts of labeled data, thereby improving detection efficiency while enhancing the model's practicality and generalization ability. Attached Figure Description

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] Figure 1 This is a schematic diagram of the overall process framework of an embodiment of the present invention;

[0042] Figure 2 This is the original self-attention module structure diagram;

[0043] Figure 3 This is a new module diagram for an embodiment of the present invention that uses dual-valued self-attention and has no feedforward network. Detailed Implementation

[0044] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:

[0045] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0046] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0047] This invention discloses an industrial anomaly detection method based on multi-agent cue learning, enabling fully automated generation of cue engineering. Its core comprises a multi-agent cue learning framework, integrating advanced language models and image processing techniques, aiming to address the efficiency and accuracy issues in industrial anomaly detection. Based on the multi-agent cue learning model, by constructing a multi-agent collaborative framework, the automatic generation of cue engineering and visual text descriptions is achieved. This innovation significantly reduces manual intervention and improves the automation level of the model. The various agents within the multi-agent system collaborate to complete tasks, thereby improving the overall system efficiency and accuracy.

[0048] The method of this invention is based on an industrial anomaly detection model that incorporates multi-agent prompting learning. This model combines multimodal pre-trained model improvement, multi-agent collaboration, and semantic connection enhancement techniques, enabling it to automatically adapt to anomaly detection tasks in various industrial products and new fields in scenarios with few samples.

[0049] The core of the multi-agent prompting learning model lies in constructing a multi-agent collaborative framework that automatically constructs prompt words and visual text descriptions through a large language model. This framework includes an image annotator agent, a description generator agent, and an anomaly detection evaluator agent. The image annotator agent receives the input image and generates the corresponding item class name; the description generator agent generates two types of visual guidance descriptions—normal and abnormal—for the corresponding class name; and the anomaly detection evaluator agent analyzes the anomaly detection results given by the multi-agent prompting learning model based on the class names and visual guidance descriptions generated by the agents.

[0050] By introducing a dual-value self-attention mechanism, the multi-agent cue-based learning model can enhance the model's focus on local features, thereby improving the accuracy of anomaly detection. Furthermore, the multi-agent collaborative system can guide image annotators and descriptors to adjust and optimize the generation of class names and visually guided descriptions based on feedback from anomaly detection evaluators, further improving the overall anomaly detection performance.

[0051] The following describes the construction process of this model, including the following steps:

[0052] S1: Construct a multi-agent collaborative framework, which includes three agents: an image annotator agent, a descriptor agent, and an anomaly detection evaluator agent;

[0053] S2: Using the Tongyi Qianwen 2.5-VL-32B multimodal large model, the image annotation agent receives the input industrial image and generates the corresponding item category name;

[0054] S3: Describes how the agent generates two types of visual guidance descriptions—normal and abnormal—using the Tongyi Qianwen 2.5-VL-32B multimodal large model and item class names. Normal visual guidance... The specific design is described as follows:

[0055]

[0056] Where P1 represents prompt word 1, and P2 represents prompt word 2. Indicates the length of learnable words. These are item class names generated by the image annotation agent. Anomaly visual guidance descriptions are divided into... and ,

[0057] The specific design is as follows: Here, "with" indicates words related to connection, "color" indicates words related to color, and "..." indicates related words that follow.

[0058] The specific design is as follows:

[0059] Where A1 represents an abnormal suffix, E A It is a learnable suffix;

[0060] S4: Anomaly detection evaluator agent analysis is based on a multimodal pre-trained model, specifically an open-source pre-trained model based on comparing text-image pairs for similarity scoring; based on a multi-agent cue learning model, the similarity between the class name and visual guidance description generated by the agent and normal features is spatially judged to be above 0.7. If it is above 0.7, it is judged as normal; otherwise, it is anomaly, thus providing the anomaly detection result; real-time computation. As feedback, the first two agents are adjusted, where AUROC is the anomaly index, TPR is the proportion of samples that are actually positive that are correctly predicted as positive by the model, and FPR is the proportion of samples that are actually negative that are incorrectly predicted as positive by the model.

[0061] S5: The industrial anomaly detection model based on multi-agent cue learning uses a dual-valued multimodal pre-trained model, requiring fewer than four normal images. In this model, one normal image is used to obtain global image features. During training, normal cue features are made close to normal visual features, while anomaly cue features are made far from normal visual features. The loss function during training is then... Specifically as follows:

[0062]

[0063] Among them, E zLet exp represent the expected value of a mathematical function. The temperature hyperparameter is denoted by z, which represents normal visual characteristics. , represents all normal visual guidance description features extracted by the text encoder, the subscript i is an auxiliary parameter used to accumulate from 1 to N, g(s) represents the process of converting the input text prompt s into a numerical feature vector using the text encoder, and N represents the number of normal visual prompts, while It is a collection of all abnormal visual guidance description features extracted by the text encoder; the visual text is automatically constructed based on the multi-agent prompting learning model, which not only improves the efficiency and accuracy of anomaly detection, but also significantly enhances the model's generalization ability.

[0064] S6: The semantic connection enhancement module uses text generated by multiple agents to construct normal visual guidance descriptions and abnormal visual guidance descriptions, and optimizes the quality of multimodal large model generation by iteratively optimizing the prompt words of the agents;

[0065] S7: Repeat steps S2 to S6 until the upper limit of the number of model iterations is reached. In this scheme, the upper limit of the number of iterations is 100.

[0066] In the above solutions of the present invention, the multimodal large model is used to automatically construct prompt words and visual text descriptions.

[0067] The core of the dual-value multimodal pre-trained model mentioned in step S5 above lies in modifying the self-attention module in standard vision:

[0068] Standard QK self-attention Attn qk The goal is to calculate the relationship between the query of each image patch and the keys of all other patches. The specific calculation formula is as follows:

[0069] Where softmax represents the attention distribution, Q represents the query vector, and K... T represents the key vector, scale represents the scaling factor, and V represents the value vector;

[0070] The present invention's dual self-attention Attn vv The similarity between the content features of each image patch is directly calculated to enhance key visual patterns in the image. The specific calculation formula is as follows:

[0071] Where softmax represents the attention distribution, V represents the value vector, and V T This represents the transpose of a value vector, and scale represents the scaling factor.

[0072] In step S6 above, the semantic connection enhancement module aligns the descriptive features with the features of normal sample images through normal visual guidance, while simultaneously using the descriptive features with abnormal visual guidance to move away from the features of normal sample images, thus providing more accurate semantic targets for the abnormal features. Specifically, this is achieved through comparative loss. Optimize tooltip alignment.

[0073] Based on the above design, the overview of the multi-agent prompting learning model obtained is as follows: Figure 1 As shown, normal visual features are obtained through a dual-valued multimodal pre-trained model. Then, a dual-valued self-attention mechanism is introduced through the dual-valued multimodal pre-trained model to enhance the model's attention to local features and improve the accuracy of anomaly detection.

[0074] The model-based multi-agent collaborative system consists of an image annotator, a description generator, and an anomaly detection evaluator. The image annotator agent receives the input image and generates the corresponding item class name. The description generator agent generates two types of visual guidance descriptions for each class name: normal and abnormal. The anomaly detection evaluator agent analyzes the class names and visual guidance descriptions generated by the agents based on the multi-agent prompting learning model and provides the AUROC level for anomaly detection. The agents within the multi-agent collaborative system cooperate through a closed-loop feedback mechanism: after generating the item class name, the image annotator agent passes it to the description generator; after the description generator outputs the visual guidance description, the anomaly detection evaluator calculates... The score serves as feedback. If... If the value is less than 0.85, where 0.85 is an empirical threshold, the evaluator will send feedback signals such as "the class name is inaccurate, causing description deviation" back to the annotators and new members. The annotators adjust the prompt words based on the feedback, while the new members optimize the description generation logic, forming a closed-loop system with feedback and iterative optimization capabilities.

[0075] The semantic connectivity enhancement module further utilizes text generated by multiple agents, such as normal prefixes, item class names, and abnormal suffixes, to construct normal prompts (NP), abnormal prompts (LAP), and learnable abnormal prompts (LLAP).

[0076] In the above model scheme, the anomaly score is calculated by using the global or local image features output by the dual-value multimodal pre-trained model and the normal text prototype features and abnormal text prototypes generated by the semantic connection enhancement module. Then, the anomaly result is returned to the anomaly detection evaluator agent to optimize the agent's prompt words.

[0077] Multimodal pre-trained models are a type of pre-training method or model based on contrastive text-image pairs, given an unknown image. and Text prompt Multimodal pre-trained models can predict which image belongs to which modality. Text prompt The probability distribution of each of them:

[0078]

[0079] in and These are visual and text encoders, respectively. To represent cosine similarity, It is a temperature hyperparameter.

[0080] As a classification model, multimodal pre-trained models perform significantly worse in few-shot anomaly detection tasks without fine-tuning. To understand why multimodal pre-trained models fail in image localization, this study analyzes the mechanism by which they extract visual features through architecture optimization. Architecture optimization is a method to improve the interpretability of multimodal pre-trained models, primarily addressing two issues: inverse visualization and noisy activation. In terms of architecture, the original multimodal pre-trained model performs poorly in image localization, mainly due to the global feature extraction impact caused by the QK self-attention mechanism. Architecture optimization, by modifying the attention mechanism, significantly improves the model's localization ability.

[0081] The model enhances its focus on local features by introducing a dual-valued self-attention mechanism without disrupting the original global feature extraction structure. This is particularly important for industrial anomaly detection, as industrial anomalies often manifest as subtle local changes. A dual-path strategy is used to fuse multi-layer features. Specifically, it preserves the original path while introducing a new path. In the new path, starting from a specific depth, the original self-attention module structure is replaced by a new dual-valued self-attention mechanism module in the new architecture. The original self-attention module structure is as follows: Figure 2 As shown, the structure of the dual-valued self-attention mechanism is as follows: Figure 3 As shown. The structure-optimized multimodal pre-trained model can better adapt to anomaly detection tasks, especially in few-shot learning scenarios, where it can more effectively utilize limited normal samples for learning. Therefore, the modified multimodal pre-trained model is used as the backbone and is called the dual-valued multimodal pre-trained model.

[0082] The industrial anomaly detection model based on multi-agent prompting learning provided by this invention can be applied to anomaly detection scenarios with fewer than 4 samples, and can even achieve anomaly detection with only one normal sample.

[0083] Compared with the prior art, the above solution of the present invention has the following advantages:

[0084] 1. No need for extensive manual design of prompts; all text is automatically generated, greatly improving the scalability and usability of the method.

[0085] 2. It can automatically adapt to anomaly detection tasks in various industrial products and new fields, and has wide applicability.

[0086] 3. It has achieved excellent results in image-level / pixel-level anomaly detection, demonstrating high efficiency and accuracy.

[0087] The following two specific application examples further demonstrate and introduce the specific implementation process of the present invention:

[0088] Example 1: Image-level anomaly detection on the MVTEC dataset based on a multi-agent prompting learning model

[0089] This embodiment aims to verify the performance of a multi-agent cue learning model in image-level anomaly detection tasks. The specific steps are as follows:

[0090] 1. Data preparation: The MVTec dataset was selected as the test object. This dataset contains images of various industrial products, in which the number of normal samples far exceeds the number of abnormal samples.

[0091] 2. Image Annotation: The input image is preprocessed using an image annotator agent to generate corresponding item category names. This step provides basic information for subsequent anomaly detection, ensuring that the model can accurately identify and process different types of industrial products.

[0092] 3. Description Generation: After receiving the item class names generated by the image annotator agent, the description generation agent further generates two types of visual guidance descriptions: normal and abnormal. These descriptions are constructed based on the class names and aim to guide the multi-agent cue-based learning model to pay more attention to key features in the image.

[0093] 4. Anomaly Detection: Anomaly detection is performed on the input image using a multi-agent cue-based learning model. The model identifies and locates abnormal regions in the image based on generated class names and visual guidance descriptions.

[0094] 5. Feedback and Optimization: Based on feedback from anomaly detection evaluators, the generation methods of image annotators and description agents are adjusted and optimized to improve the accuracy and robustness of anomaly detection.

[0095] Experimental results show that the multi-agent prompting learning model achieves excellent anomaly detection performance on the MVTec dataset, demonstrating its effectiveness in image-level anomaly detection tasks.

[0096] Example 2: Pixel-level anomaly detection on the VisA dataset based on a multi-agent cue learning model

[0097] This embodiment aims to verify the performance of a multi-agent prompting learning model in pixel-level anomaly detection tasks. The specific steps are as follows:

[0098] 1. Data preparation: The VisA dataset was selected as the test object. This dataset contains high-resolution images of industrial products, in which pixel-level anomaly annotations provide a fine benchmark for anomaly detection.

[0099] 2. Image annotation and description generation: Similar to Example 1, the image annotation agent is first used to generate item class names, and then the description generation agent is used to generate normal and abnormal visual guidance descriptions.

[0100] 3. Anomaly Detection: A multi-agent prompting learning model is used to perform pixel-level anomaly detection on the input image. The model classifies each pixel in the image as an anomaly based on the generated class name and visual guidance description.

[0101] 4. Performance Evaluation: The accuracy and generalization ability of the model in pixel-level anomaly detection tasks are evaluated by comparing the detection results of the multi-agent prompting learning model with the real annotations provided by the VisA dataset.

[0102] 5. Experimental results show that the multi-agent prompting learning model also achieves excellent anomaly detection performance on the VisA dataset, demonstrating its effectiveness and generalization ability in pixel-level anomaly detection tasks.

[0103] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.

[0104] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0106] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

[0107] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of industrial anomaly detection methods based on multi-agent prompting learning. All equivalent variations and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.

Claims

1. An industrial anomaly detection method based on multi-agent cue learning, characterized in that: The following processing is performed using a multi-agent cooperative framework and an improved multimodal model: The input industrial image is parsed using an image annotation agent, and item class names are generated through a multimodal large model. Based on the item class name, the member intelligent agent generates normal visual guidance description and abnormal visual guidance description. The normal visual guidance description consists of a learnable prefix sequence and the item class name, while the abnormal visual guidance description includes a learnable prefix sequence, the item class name, and abnormal related suffixes. Visual features of industrial images are extracted using an improved multimodal pre-trained model, and local feature representation is enhanced through value vector self-similarity calculation. Through semantic connection enhancement processing, the text features of normal visual guidance descriptions are aligned with the visual features of normal samples, while the text features of abnormal visual guidance descriptions are moved away from the visual features of normal samples. The similarity between visual features and normal and abnormal visual guided description features is calculated based on an improved multimodal pre-trained model. When the similarity of normal features is lower than a preset threshold, it is judged as abnormal and the detection result is output. The anomaly detection evaluator agent calculates the anomaly index AUROC as a feedback signal, which drives the image annotator agent to adjust the item category name generation logic and the description generation agent to optimize the visually guided description generation logic. The above processes are iteratively executed until the preset conditions are met. The improved multimodal pre-trained model calculates the value vector self-similarity using a two-valued self-attention mechanism. Attention weights are generated by the similarity distribution between the value vector and its transpose matrix, and then the value vectors are aggregated based on these weights to enhance the expression of local features. The similarity distribution is calculated using the softmax function, and a scaling factor is introduced during the calculation process to adjust the similarity scale. The semantic connection enhancement process optimizes feature alignment through a contrastive loss function. This loss function strengthens the similarity between text features of normal visual guidance descriptions and normal visual features, while weakening the similarity between text features of abnormal visual guidance descriptions and normal visual features. Specifically, the loss function adjusts the strength of feature alignment based on the similarity between the mean of normal visual features and normal text features, as well as the similarity between normal visual features and all abnormal text features, through exponential and logarithmic operations. A temperature hyperparameter is introduced to control the weighting of similarity, and the mathematical expectation of normal visual features is used as the calculation benchmark. The mean of the normal text features is the average value of the features extracted by the text encoder for all normal visual guidance descriptions, and the set of abnormal text features is the set of features extracted by the text encoder for all abnormal visual guidance descriptions.

2. The industrial anomaly detection method based on multi-agent cue learning according to claim 1, characterized in that: The image labeler agent and the description agent in the multi-agent collaborative framework are both implemented based on a multimodal large model, which is used to automatically generate item class names and visual guidance description prompts.

3. The industrial anomaly detection method based on multi-agent cue learning according to claim 1, characterized in that: The specific structure of the normal visual guidance description is as follows: it consists of a learnable prefix sequence concatenated with an item category name, wherein the prefix sequence contains multiple prompt words, the sequence length is a learnable parameter, and the learnable parameter is optimized through model training.

4. The industrial anomaly detection method based on multi-agent cue learning according to claim 1, characterized in that: The abnormal visual guidance descriptions include two types: The first category consists of a learnable prefix sequence, an item class name, and abnormal attribute connectors concatenated together. The abnormal attribute connectors include words indicating feature association, words indicating color features, and other abnormal attribute descriptive words. The second category consists of a learnable prefix sequence, an item class name, and a learnable anomaly suffix sequence concatenated together. The learnable anomaly suffix sequence contains multiple anomaly feature descriptors, and the sequence length is a learnable parameter.

5. The industrial anomaly detection method based on multi-agent cue learning according to claim 1, characterized in that: The preset threshold is 0.

7. When the similarity between the visual feature and the normal visual guidance description feature is higher than 0.7, it is judged as normal; otherwise, it is judged as abnormal. When the anomaly detection evaluator agent is below the empirical threshold, it sends targeted feedback signals to the image annotator agent and the description generation agent. The feedback signals include suggestions for adjusting class name accuracy and description logic deviation to drive the optimization of generation logic.

6. The industrial anomaly detection method based on multi-agent cue learning according to claim 1, characterized in that: The improved multimodal pre-trained model adopts a dual-path feature fusion strategy: one path retains the original feature extraction structure, while the other path replaces it with the dual-value self-attention mechanism starting from a specific network depth, so as to retain both global and local features at the same time.

7. An industrial anomaly detection system based on multi-agent cue learning, used to implement the method as described in claim 1, characterized in that, include: The multi-agent collaborative module includes an image annotator unit, a description generation unit, and an anomaly detection evaluator unit; among them, The image annotator unit is used to parse the input industrial images and generate item class names through a multimodal large model; The description member unit is used to generate normal visual guidance description and abnormal visual guidance description based on the item class name. The normal visual guidance description consists of a learnable prefix sequence and the item class name, and the abnormal visual guidance description includes a learnable prefix sequence, the item class name, and abnormal related suffixes. The anomaly detection and evaluation unit is used to calculate the anomaly index AUROC as a feedback signal, which drives the image annotation unit to adjust the item class name generation logic and the description generation unit to optimize the visually guided description generation logic. An improved multimodal processing module is used to extract visual features from industrial images. It enhances the expression of local features through value vector self-similarity calculation and calculates the similarity between visual features and normal and abnormal visual guided descriptive features. When the similarity of normal features is lower than a preset threshold, it is judged as abnormal and the detection result is output. The semantic connection enhancement module is used to align the text features of normal visual guidance descriptions with the visual features of normal samples through feature alignment processing, and to keep the text features of abnormal visual guidance descriptions away from the visual features of normal samples. The system iteratively executes the processing of the above modules until the preset conditions are met.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the industrial anomaly detection method based on multi-agent prompting learning as described in any one of claims 1-6.

Images