Industrial anomaly detection method and system, storage medium and electronic equipment

Abstract

The invention provides an industrial anomaly detection method and system, a storage medium and electronic equipment. The method comprises the steps that a to-be-detected industrial image is acquired; performing visual similarity retrieval based on the to-be-detected industrial image to obtain a similar visual anomaly prompt; triggering knowledge retrieval based on the similar visual anomaly prompts to obtain dynamic knowledge prompts; and performing industrial anomaly detection on the to-be-detected industrial image based on the similar visual anomaly prompt and the dynamic knowledge prompt to obtain an industrial anomaly detection result. The method can remarkably improve the recognition and processing capability of the model for complex abnormal events in industrial scenes, has the characteristics of high expandability, high universality, low deployment threshold and the like, and is particularly suitable for professional fields with extremely high requirements for abnormal detection precision and response timeliness, such as power substations, traffic inspection, military security and the like.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to an industrial anomaly detection method, system, storage medium, and electronic device. Background Technology

[0002] Industrial anomaly detection refers to the critical task of real-time monitoring and early warning of equipment operating status, personnel behavior, and environmental risks in industrial production using automation and intelligent technologies. The system, leveraging IoT sensors, high-definition cameras, drones, and artificial intelligence algorithms, can automatically identify equipment malfunctions (such as abnormal temperature and excessive vibration), safety hazards (such as personnel not wearing protective equipment or trespassing into dangerous areas), and environmental anomalies (such as gas leaks and signs of fire), and issue timely warnings. This technology has gradually replaced traditional manual inspection methods, achieving all-weather, high-precision, and high-efficiency risk management, significantly improving production safety and intelligent operation and maintenance, and becoming a core support for industrial safety production and preventative maintenance.

[0003] Currently, most mainstream industrial anomaly detection systems rely on a single model to complete image perception tasks. As multimodal large language models (MLLMs) demonstrate increasingly powerful reasoning capabilities in general scenarios, academia and industry are actively exploring their application potential in vertical fields, particularly in critical tasks such as industrial inspection and safety anomaly detection. Therefore, how to efficiently detect industrial anomalies using MLLMs is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides an industrial anomaly detection method, system, storage medium, and electronic device that can efficiently detect industrial anomalies in complex scenarios using MLLM.

[0005] In a first aspect, this application provides an industrial anomaly detection method, the method comprising:

[0006] Acquire the industrial image to be inspected;

[0007] Visual similarity retrieval is performed based on the industrial image to be detected to obtain similar visual anomaly alerts;

[0008] Knowledge retrieval is triggered based on the similar visual anomaly prompts to obtain dynamic knowledge prompts;

[0009] Based on the similar visual anomaly prompts and the dynamic knowledge prompts, industrial anomaly detection is performed on the industrial image to be detected to obtain industrial anomaly detection results.

[0010] In one implementation of the first aspect, visual similarity retrieval based on the industrial image to be detected includes:

[0011] The industrial image to be detected is subjected to visual semantic encoding to obtain a visual semantic representation;

[0012] Visual similarity retrieval is performed on the visual semantic representation based on a visual experience database to obtain visual precedents;

[0013] Based on the aforementioned visual precedents, construct similar visual anomaly prompts.

[0014] In one implementation of the first aspect, obtaining dynamic knowledge hints includes:

[0015] Based on the similar visual anomaly cues, candidate industrial anomaly types are identified;

[0016] Based on the conceptual knowledge base, knowledge retrieval is performed on the candidate industrial anomaly types to obtain relevant knowledge entries;

[0017] Dynamic knowledge prompts are constructed based on the relevant knowledge entries.

[0018] In one implementation of the first aspect, industrial anomaly detection of the industrial image to be detected based on the similar visual anomaly cues and the dynamic knowledge cues includes:

[0019] The image to be processed and its visual semantic representation are perceived to obtain perceptual information;

[0020] Based on the perceptual information, the similar visual anomaly prompts are compared with the dynamic knowledge prompts to obtain industrial anomalies in the image to be processed for inference, and a structured industrial anomaly detection result is output.

[0021] In one implementation of the first aspect, the method further includes constructing a visual experience database and a concept knowledge base; wherein, it includes

[0022] Acquire several industrial image samples and label the industrial image samples with target and category tags to obtain an image dataset;

[0023] The category labels marked in the industrial image samples are mapped to the corresponding target location information to obtain a spatial location-category mapping set;

[0024] The abnormal targets and abnormal category labels marked in the industrial image samples are used as an abnormal label set;

[0025] The image dataset, the spatial location and category mapping set, and the anomaly annotation set are used as the industrial dataset;

[0026] The visual experience database and the conceptual knowledge base are constructed based on the industrial dataset.

[0027] In one implementation of the first aspect, constructing the visual experience database and the concept knowledge base based on the industrial dataset includes:

[0028] The industrial image samples are subjected to visual semantic encoding to obtain visual semantic empirical representations;

[0029] The abnormal target, the abnormal category, and the corresponding target location information labeled in the industrial image sample are used as semantic triples;

[0030] The visual semantic experience representation and the corresponding semantic triple are stored as visual precedents to construct the visual experience database.

[0031] Based on the industrial image and the corresponding semantic triples and external prior knowledge, domain triples are generated, and the domain triples are stored as knowledge entries to construct the concept knowledge base.

[0032] In one implementation of the first aspect, the generation of domain triples based on the industrial image and the corresponding semantic triples, and external prior knowledge includes:

[0033] Based on the industrial image and the corresponding semantic triple, the anomaly category is obtained, and an anomaly description is generated;

[0034] Anomaly handling suggestions are generated based on the aforementioned external prior knowledge;

[0035] The anomaly category, the anomaly description, and the anomaly handling suggestion are used as the domain triple.

[0036] Secondly, this application provides an industrial anomaly detection system, the system comprising:

[0037] The acquisition module is configured to acquire the industrial image to be inspected;

[0038] The visual precedent retrieval module is configured to perform visual similarity retrieval based on the industrial image to be detected in order to obtain similar visual anomaly prompts.

[0039] The domain knowledge enhancement module is configured to trigger knowledge retrieval based on the similar visual anomaly prompts to obtain dynamic knowledge prompts;

[0040] The industrial anomaly detection module is configured to perform industrial anomaly detection on the industrial image to be detected based on the similar visual anomaly prompts and the dynamic knowledge prompts, so as to obtain industrial anomaly detection results.

[0041] Thirdly, this application provides a storage medium on which a computer program is stored, which, when executed by a processor, implements the above-described industrial anomaly detection method.

[0042] Fourthly, this application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the electronic device performs the above-described industrial anomaly detection method.

[0043] As described above, the industrial anomaly detection method, system, storage medium, and electronic device of this application have the following beneficial effects:

[0044] This application can effectively solve the key problems in existing industrial anomaly detection technologies, such as difficulty in multimodal data fusion, weak semantic understanding, and delayed response decisions, and overcome the limitations of traditional systems that rely on only a single visual model and lack task adaptability.

[0045] This application possesses capabilities such as task-aware dynamic scheduling and adaptive inference path planning. Especially in high-risk and high-complexity application scenarios such as substations and security inspections, this application significantly improves the model's ability to identify and process complex abnormal events in industrial scenarios, reduces the reliance on large-scale manual annotation, achieves robust detection and semantic closed-loop response to multi-type and multi-scale risks in professional scenarios, and supports interpretable decision-making, customized response, and multi-task linkage.

[0046] This application not only improves the accuracy of detection tasks (detection, classification, localization), but also achieves consistent improvement across models in description and suggestion generation tasks that require semantic understanding and reasoning, effectively alleviating the problems of model illusion and semantic bias.

[0047] This application features high scalability, high versatility, and low deployment threshold, making it particularly suitable for professional fields with extremely high requirements for anomaly detection accuracy and response time, such as power substations, traffic inspection, and military security. It breaks through the technical bottlenecks of traditional industrial detection models in terms of generalization ability, cognitive depth, and semantic feedback, providing fundamental support for the application of multimodal artificial intelligence in safety-critical scenarios.

[0048] In summary, this application demonstrates strong robustness and universality in industrial visual security inspection scenarios. Technically, it breaks through the dependence of existing methods on model fine-tuning, significantly improves the zero-shot detection and inference capabilities of multimodal large models, and has broad application prospects and industrial value. Attached Figure Description

[0049] Figure 1 The diagram shown is a flowchart of an industrial anomaly detection method provided in an embodiment of this application.

[0050] Figure 2 The diagram shown is a schematic representation of the process for visual similarity retrieval based on the industrial image to be detected, as provided in an embodiment of this application.

[0051] Figure 3 The diagram shown is a flowchart illustrating the process of obtaining dynamic knowledge prompts as provided in an embodiment of this application.

[0052] Figure 4 The diagram shows a flowchart illustrating the process of constructing a visual experience database and a conceptual knowledge base, as provided in an embodiment of this application.

[0053] Figure 5 The diagram shows a process for constructing the visual experience database and the conceptual knowledge base based on the industrial dataset, as provided in an embodiment of this application.

[0054] Figure 6 The diagram shown is a structural schematic of an industrial anomaly detection system provided in an embodiment of this application.

[0055] Figure 7 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0057] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0058] Current mainstream industrial anomaly detection systems mostly employ a single model to complete image perception tasks, lacking task decomposition and collaborative mechanisms, making it difficult to effectively process multi-target and multi-modal information in complex scenarios. Furthermore, traditional methods generally lack the ability to provide causal explanations, deductive reasoning, or semantic responses to anomalies, limiting their practicality in scenarios with strong logical and safety requirements. For example, in scenarios such as power substations, the system needs to identify complex events such as high-temperature hotspots, foreign object obstructions, and cable aging, and make interpretable judgments based on operating procedures and inspection tasks; a single perception model is ill-suited for such multi-stage tasks.

[0059] Traditional visual detection models rely on large-scale supervised learning, which performs well on specific tasks but has limited capabilities in cross-task generalization, semantic understanding, and contextual reasoning. Meanwhile, existing multimodal large language models (MLLMs), while possessing natural language understanding capabilities, still lag significantly behind in terms of highly reliable perception, knowledge support, and decision-making response capabilities for professional scenarios.

[0060] Furthermore, most existing multimodal model systems employ static prompts or fixed knowledge inputs, making it difficult to dynamically schedule knowledge or reasoning paths based on perceived content. Faced with the increasing complexity of tasks and the growing diversity of inputs, traditional solutions suffer from bottlenecks in terms of flexibility, accuracy, and responsiveness.

[0061] To at least address the aforementioned technical issues, embodiments of this application provide an industrial anomaly detection method, system, storage medium, and electronic device. These can be applied to high-risk and highly complex application scenarios such as substations and security inspections, enabling robust detection and semantic closed-loop responses to multi-type and multi-scale risks in professional scenarios. They support interpretable decision-making, customized responses, and multi-task linkage, significantly improving the model's ability to identify and process complex anomalies in industrial scenarios. This breaks through the technical bottlenecks of traditional industrial detection models in terms of generalization ability, cognitive depth, and semantic feedback.

[0062] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] Figure 1 The diagram shown is a flowchart illustrating the industrial anomaly detection method provided in an embodiment of this application. Figure 1 As shown, in one embodiment, the industrial anomaly detection method of this application includes steps S1 to S4.

[0064] Step S1: Obtain the industrial image to be inspected.

[0065] In some embodiments, industrial images to be detected can be acquired through surveillance videos, cameras, or other means to determine whether any industrial anomalies exist. For example, surveillance videos can be used to obtain images of workers at a substation in an industrial setting, and the system can detect whether workers in the images are wearing helmets, i.e., whether there is an industrial anomaly where workers are not wearing helmets.

[0066] Step S2: Perform visual similarity retrieval based on the industrial image to be detected to obtain similar visual anomaly prompts.

[0067] In some embodiments, this application introduces a similar sample recall mechanism based on semantic embedding to perform visual similarity retrieval for industrial images to be detected and obtain similar visual anomaly prompts. Figure 2The diagram shown is a schematic representation of a process for visual similarity retrieval based on the industrial image to be detected, as provided in an embodiment of this application. Figure 2 As shown, the visual similarity retrieval based on the industrial image to be detected includes steps S21 to S23.

[0068] Step S21: Perform visual semantic encoding on the industrial image to be detected to obtain a visual semantic representation.

[0069] In some embodiments, the industrial image to be detected is visually semantically encoded using a multimodal model (such as CLIP-ViT), a high-dimensional semantic embedding vector is extracted, and a visual semantic representation e is constructed. Iq This representation can effectively capture the category, environmental context, and potential danger factors of abnormal objects in an image, laying the foundation for subsequent semantic alignment retrieval.

[0070] Step S22: Perform visual similarity retrieval on the visual semantic representation based on the visual experience database to obtain visual precedents.

[0071] In some embodiments, visual semantic representation is used based on a pre-built visual experience database. Iq Cosine similarity ranking retrieval is performed in the semantic space to filter out the top K historical samples that are most similar to the current industrial image to be detected, and then visual experience enhancer (VEE) retrieval is performed.

[0072] Furthermore, the specific similarity calculation and ranking retrieval formulas are as follows:

[0073]

[0074] in, DB represents the visual semantic representation of the industrial image to be detected. VEE Represents a visual experience database, where each visual precedent entry (e I ,gt s ) includes visual semantic experience representation e I Its corresponding semantic triple gt s This step enables efficient recall of precedent samples with high semantic consistency.

[0075] In some embodiments, semantic triple gt s Represented as:

[0076] gt s ={object, anomaly, location}

[0077] Here, "object" represents the abnormal target, "anomaly" represents the anomaly category, and "location" represents the target's location information. Furthermore, the target location information is labeled using a 3x3 grid mechanism, with positions marked according to labels such as "top left, top center, top right...".

[0078] Step S23: Construct similar visual anomaly prompts based on the aforementioned visual precedents.

[0079] In some embodiments, this application retrieves the set V of visual precedent triples. retrieved A set of refined semantic cue word templates is generated to guide subsequent tasks such as industrial anomaly localization, classification, description, and suggestion generation. This similar visual anomaly cue not only improves the accuracy of target area perception but also enhances the ability of subsequent large-scale model agents to discriminate ambiguous or minor anomalies by incorporating empirical cases, effectively improving the stability and interpretability of multimodal models in industrial tasks.

[0080] Step S3: Trigger knowledge retrieval based on the similar visual anomaly prompts to obtain dynamic knowledge prompts.

[0081] In some embodiments, after completing visual similarity retrieval and obtaining similar visual anomaly prompts, knowledge retrieval is triggered to obtain dynamic knowledge prompts. Figure 3 The diagram shown is a flowchart illustrating the process of obtaining dynamic knowledge prompts provided in an embodiment of this application. For example... Figure 3 As shown, obtaining dynamic knowledge prompts includes steps S31 to S33.

[0082] Step S31: Identify candidate industrial anomaly types based on the similar visual anomaly prompts.

[0083] Step S32: Perform knowledge retrieval on the candidate industrial anomaly types based on the concept knowledge base to obtain relevant knowledge entries.

[0084] Step S33: Construct dynamic knowledge prompts based on the relevant knowledge entries.

[0085] In the above embodiments, after completing the visual precedent retrieval, candidate industrial anomaly types and their associated targets (such as "damaged cable," "personnel not wearing safety helmets," etc.) can be automatically identified based on similar visual anomaly cues. Then, based on these identified candidate industrial anomaly types, data is sent to the concept knowledge base (DB). DKE Submitting a semantic query, that is, performing knowledge retrieval on the candidate industrial anomaly types based on the concept knowledge base (DB). DKEIt contains structured knowledge entries generated by a multimodal large language model, covering anomaly categories, involved objects or scenarios, semantic descriptions, potential risks, and professional advice. Through a semantic matching mechanism centered on anomaly categories, the system can efficiently recall knowledge entries highly related to anomalies in the current image, providing traceable and interpretable prior information.

[0086] In some embodiments, a knowledge entry is represented as:

[0087] DK={anomaly,description,suggestion}

[0088] In this context, anomaly is the same as in the semantic triple, representing the specific industrial anomaly category identified by the model (e.g., "workers not wearing helmets"); description is the anomaly description, used to characterize the anomaly with more fine-grained semantic features, reflecting its contextual risk or semantic meaning (e.g., "in a high-voltage area, there is a risk of electric shock"); and suggestion is the anomaly handling suggestion, used to provide operational suggestions or handling strategies related to the anomaly (e.g., "issue a warning and isolate the area").

[0089] Furthermore, the retrieved relevant knowledge entries are automatically integrated into a set of dynamic knowledge prompts, serving as the input context for multimodal reasoning. The prompt format includes, but is not limited to: "Anomaly: Insulator damage. Description: Visible cracks in the upper phase insulator. Risk and Recommendation: Risk of arc discharge exists. Isolate the affected line and dispatch a maintenance team." This dynamic knowledge prompt not only enhances the multimodal large model's understanding of the current scenario but also possesses causal guidance capabilities, helping the model to produce reasoning results that conform to safety rules.

[0090] This application effectively introduces external prior knowledge with domain knowledge and causal reasoning capabilities, enabling the system to possess context-awareness when determining anomaly types and their handling suggestions. For example, the same incident of "personnel not wearing safety helmets" carries completely different risk levels and response suggestions in a "high-voltage substation area" and an "office area." This mechanism significantly improves the generalization, interpretability, and reliability of the anomaly detection system in complex environments.

[0091] Step S4: Perform industrial anomaly detection on the industrial image to be detected based on the similar visual anomaly prompts and the dynamic knowledge prompts to obtain industrial anomaly detection results.

[0092] In some embodiments, after obtaining similar visual anomaly cues and dynamic knowledge cues, both are input into a multimodal large language model to perform industrial anomaly detection and output structured industrial anomaly detection results. Specifically, industrial anomaly detection of the industrial image to be detected based on the similar visual anomaly cues and the dynamic knowledge cues includes: perceiving the image to be processed and its visual semantic representation to obtain perceptual information; comparing the similar visual anomaly cues and the dynamic knowledge cues based on the perceptual information to obtain industrial anomalies in the image to be processed, performing inference, and outputting structured industrial anomaly detection results.

[0093] In some embodiments, similar visual anomaly cues and dynamic knowledge cues are jointly input into a multimodal large language model, prompting it to act as an intelligent agent playing the role of an "industrial safety inspector." The model first perceives the input image and its semantic embedding, understanding potential anomaly risk points, object types, and spatial distribution in the scene, thus establishing a semantic perception foundation for the subsequent reasoning stage.

[0094] The next stage is inference, where potential anomalies are inferred by comparing historical precedents with dynamic knowledge cues and similar visual anomaly cues. Specifically, after perception, the agent utilizes the model's multimodal association and logical capabilities, combined with a visual precedent library (DB). VEE Visual precedents in (e) I ,gt s Conceptual Knowledge Base (DB) DKE The knowledge entries (rules and suggestions) in the image are used to perform attribution analysis, risk assessment, and confidence judgment on anomalies in the current image. This stage fully embodies the multi-layered reasoning chain from "visual cues → experiential association → causal explanation," achieving deep semantic recognition and contextual awareness of anomalies.

[0095] Finally, in the decision-making phase, structured responses are output through task alignment to support evaluation and downstream use. Ultimately, the agent-guided model outputs standardized structured question-and-answer formats according to six predefined task types (e.g., anomaly warning, anomaly classification, anomaly localization, multiple anomaly identification, anomaly description generation, and handling suggestion generation), ensuring a unified format that facilitates evaluation and deployment. This output can not only be used for security incident analysis and reporting but also serve as input for downstream modules such as knowledge graph construction, multi-turn dialogue planning, and control command generation, effectively supporting the closed-loop operation of the complete agent system.

[0096] This application, through the introduction of an intelligent agent module, realizes a task-oriented multimodal intelligent agent closed loop from "perception-understanding-decision", which significantly improves the stability, interpretability and response consistency of the system in complex scenarios, and ensures the reliability and robustness of the AI ​​security inspection system in actual deployment.

[0097] Furthermore, the industrial anomaly detection method provided in this application also includes constructing a visual experience database and a conceptual knowledge base, thereby providing basic support for the reasoning in the above embodiments. Figure 4 The diagram shows a flowchart illustrating the process of constructing a visual experience database and a conceptual knowledge base, as provided in an embodiment of this application. Figure 4 As shown, constructing the visual experience database and concept knowledge base includes steps S5 to S9.

[0098] S5. Obtain several industrial image samples and label the industrial image samples with target and category tags to obtain an image dataset.

[0099] S6. Map the category labels marked in the industrial image samples to the corresponding target location information to obtain a spatial location and category mapping set.

[0100] S7. The abnormal targets and abnormal category labels marked in the industrial image samples are used as an abnormal label set.

[0101] S8. Use the image dataset, the spatial location and category mapping set, and the anomaly annotation set as an industrial dataset.

[0102] S9. Construct the visual experience database and the concept knowledge base based on the industrial dataset.

[0103] In some embodiments, to construct a knowledge layer with semantic awareness capabilities, a traditional object detection dataset covering typical industrial scenarios and anomaly categories is first introduced as the initial data source for factual knowledge extraction and conceptual knowledge construction. Then, the industrial image samples in the initial data source are labeled with object and category labels to obtain image dataset I. Image dataset I includes several industrial image samples, each labeled with multiple bounding boxes and their corresponding category labels. id .

[0104] Furthermore, regarding the category labels c in the industrial samples... id The spatial location and category mapping set L is obtained by mapping the corresponding target location information. This spatial location and category mapping set L records the component location and category information corresponding to each target in the image. For the target location information, a grid mechanism is used, dividing the entire image sample into an average of 3×3 grids, and labeling the location according to "top left, top center, top right..." to determine the location information of each target.

[0105] Furthermore, abnormal targets and their corresponding abnormal category labels in some images are used as an abnormal label set A. Anomaly label set A contains abnormal information attached to some images (such as equipment damage, personnel violations, etc.).

[0106] Subsequently, the image dataset I, the spatial location and category mapping set L, and the anomaly annotation set A are used as an industrial dataset. This dataset contains a large number of manually annotated image samples, which have high annotation quality and target distribution diversity, and can be used to construct the visual experience database and the concept knowledge base. Figure 5 The diagram shows a flowchart illustrating the process of constructing the visual experience database and the conceptual knowledge base based on the industrial dataset, as provided in an embodiment of this application. Figure 5 As shown, constructing the visual experience database and the concept knowledge base based on the industrial dataset includes steps S91 to S94.

[0107] S91. Perform visual semantic encoding on the industrial image samples to obtain visual semantic empirical representation.

[0108] S92. The abnormal target, the abnormal category, and the corresponding target location information labeled in the industrial image sample are used as semantic triples.

[0109] S93. The visual semantic experience representation and the corresponding semantic triple are stored as visual precedents to construct the visual experience database.

[0110] S94. Generate domain triples based on the industrial image and the corresponding semantic triples and external prior knowledge, and store the domain triples as knowledge entries to construct the concept knowledge base.

[0111] The process of generating a domain triple based on the industrial image and the corresponding semantic triple and external prior knowledge includes: obtaining an anomaly category based on the industrial image and the corresponding semantic triple, and generating an anomaly description; generating an anomaly handling suggestion based on the external prior knowledge; and using the anomaly category, the anomaly description, and the anomaly handling suggestion as the domain triple.

[0112] In some embodiments, the visual experience database is constructed based on the industrial dataset. Raw image samples from the prepared dataset are input into a pre-trained visual language encoding model (such as CLIP-ViT) to extract their visual semantic experience representations. I Simultaneously, the image annotation information (such as anomalous targets and their corresponding anomalous categories, and location information) is transformed into structured semantic triples:

[0113] gt s ={object, anomaly, location}

[0114] Here, object represents the anomaly target (the object involved in the anomaly or risk), anomaly represents the anomaly category (the type name of the anomaly), and location is the target location information (the location where the anomaly is located).

[0115] Subsequently, the visual semantic experience representation of the image is e I With the corresponding semantic facts gt s Combination, as a visual precedent (e I ,gt s The data is uniformly stored in the visual experience database DB based on the FAISS framework. VEE This process forms the foundation for visual similarity retrieval and class-case reasoning. It constructs an indexing mechanism that maps visual information to semantics, enabling subsequent models to retrieve relevant visual precedents and achieve analogical reasoning and anomaly attribution.

[0116] Furthermore, based on the completed visual experience database, a multimodal large language model (such as GPT-4o, LLaMA, etc.) is introduced to analyze industrial image samples and their corresponding semantic triples. s ={object, anomaly, location} for joint understanding and semantic dimensionality enhancement. In some embodiments, the model collaboratively processes the image and labeled semantics through multimodal input. s Generates higher-order structured knowledge items in the form of triples:

[0117] DK={anomaly,description,suggestion}

[0118] Among them, anomaly and semantic triple gt s The same in the middle indicates the specific industrial anomaly category identified by the model (e.g., "workers not wearing helmets"); description is the anomaly description, used to characterize the anomaly with more fine-grained semantic features, reflecting its contextual risk or semantic meaning (e.g., "in a high-voltage area, there is a risk of electric shock"); suggestion is the handling suggestion, which combines external prior knowledge to provide operational suggestions or handling strategies related to the anomaly (e.g., "issue a warning and isolate the area").

[0119] The knowledge entries DK consisting of the above triples will be uniformly stored in the concept knowledge base DB. DKE This provides rich structured knowledge support for semantic reasoning, multimodal cue construction, and anomaly response strategy generation in subsequent tasks. It effectively realizes the transformation from perceived facts to semantic knowledge, elevating traditional detection labels to a general knowledge representation that can be understood and utilized by large multimodal models, thus enhancing domain adaptability, interpretability, and task generalization ability.

[0120] Therefore, the industrial anomaly detection method provided in this application combines visual-language bimodal modeling with knowledge-enhanced reasoning mechanisms, making it suitable for anomaly detection tasks in critical industrial infrastructure such as substations. This method uses "perception-understanding-decision" as its core process, constructing a multi-agent collaborative perception framework to achieve high-precision detection, semantic understanding, and response generation of multi-source visual anomalies. Compared to existing methods that rely solely on single-modal perception or fixed-rule responses, this application effectively improves the model's ability to identify complex scenarios and multiple types of faults, reduces reliance on large-scale manual annotation, and possesses advantages such as strong generalization, flexible deployment, and sustainable knowledge updates.

[0121] This application demonstrates a significant improvement in overall scores across six sub-tasks (anomaly detection, classification, localization, multiple anomaly identification, anomaly description, and generation of impact and suggestions). Using the self-built SafeSubstation-Bench benchmark as the evaluation platform, compared to traditional zero-shot inference configurations without knowledge enhancement, this application achieves performance improvements on several representative multimodal large models (including GPT-4o, GPT-4o-mini, Qwen2.5-VL, and LLaMA3.2-Vision).

[0122] Experimental results based on GPT-4o show that this application improves the overall average accuracy by 6.52% on six tasks; by 8.56% on GPT-4o-mini; by 9.52% on Qwen2.5-VL (32B); and by 17.49% on LLaMA3.2-Vision (11B). These results fully demonstrate that by introducing multimodal knowledge enhancement and collaborative agent reasoning mechanisms, this application can effectively improve the robustness and generalization ability of multimodal large models of different sizes and architectures in complex industrial scenarios.

[0123] Furthermore, this application not only improves the accuracy of detection tasks (detection, classification, localization), but also achieves consistent improvement across models in description and suggestion generation tasks that require semantic understanding and reasoning, effectively alleviating the problems of model illusion and semantic bias.

[0124] In summary, this application demonstrates strong robustness and universality in industrial visual security inspection scenarios. Technically, it breaks through the dependence of existing methods on model fine-tuning, significantly improves the zero-shot detection and inference capabilities of multimodal large models, and has broad application prospects and industrial value.

[0125] The scope of protection of the industrial anomaly detection method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0126] This application also provides an industrial anomaly detection system, which can implement the industrial anomaly detection method described in this application. However, the implementation device of the industrial anomaly detection system described in this application includes, but is not limited to, the structure of the industrial anomaly detection system listed in this embodiment. All structural modifications and substitutions of the prior art made based on the principles of this application are included within the protection scope of this application.

[0127] Figure 6 The diagram shown is a structural schematic of an industrial anomaly detection system provided in an embodiment of this application. Figure 6 As shown, the industrial anomaly detection system of this application includes an acquisition module 31, a visual precedent retrieval module 32, a domain knowledge enhancement module 33, and an industrial anomaly detection module 34.

[0128] The acquisition module 31 is configured to acquire an industrial image to be detected;

[0129] The visual precedent retrieval module 32 is configured to perform visual similarity retrieval based on the industrial image to be detected in order to obtain similar visual anomaly prompts.

[0130] Domain knowledge enhancement module 33 is configured to trigger knowledge retrieval based on the similar visual anomaly prompts to obtain dynamic knowledge prompts;

[0131] The industrial anomaly detection module 34 is configured to perform industrial anomaly detection on the industrial image to be detected based on the similar visual anomaly prompts and the dynamic knowledge prompts, so as to obtain industrial anomaly detection results.

[0132] In some other embodiments, the industrial anomaly detection system further includes a knowledge extraction module configured to build a visual experience database and a conceptual knowledge base.

[0133] The structure and principle of each module in the industrial anomaly detection system correspond one-to-one with the steps in the above-mentioned industrial anomaly detection method, so they will not be repeated here.

[0134] The industrial anomaly detection system provided in this application utilizes a knowledge extraction module to construct a visual experience base and a domain concept base based on traditional detection data and semantic annotation, providing fundamental support for subsequent reasoning. A visual precedent retrieval module introduces a similar sample recall mechanism based on semantic embedding, providing visual references and prior hints for the industrial images to be inspected. Subsequently, a domain knowledge enhancement module triggers knowledge base retrieval and dynamically generates knowledge hints during the domain knowledge enhancement and semantic reasoning optimization stages, strengthening context awareness and anomaly interpretation capabilities. Finally, an industrial anomaly detection module integrates perception and knowledge hints during the anomaly detection and structured response output stages, guiding a multimodal large model to complete the entire process from image understanding to task output. The overall system design achieves the synergistic integration of visual perception and knowledge reasoning, effectively improving the accuracy and interpretability of anomaly detection and risk assessment in complex industrial scenarios, and is applicable to multiple complex industrial scenarios such as substations.

[0135] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0136] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0137] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the industrial anomaly detection method of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0139] This application also provides an electronic device. The electronic device includes a processor and a memory.

[0140] The memory is used to store computer programs.

[0141] The memory includes various media capable of storing program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card, or optical disk.

[0142] The processor is connected to the memory and is used to execute the computer program stored in the memory so that the electronic device performs the above-described industrial anomaly detection method.

[0143] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0144] like Figure 7 As shown, the electronic device of this application is embodied in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: one or more processors or processing units 41, memory 42, and bus 43 connecting different system components (including memory 42 and processing unit 41).

[0145] Bus 43 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0146] Electronic devices typically include a variety of computer-readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, and removable and non-removable media.

[0147] Memory 42 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 423 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 7 Not shown; usually referred to as a "hard drive"). Although Figure 7Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 43 via one or more data media interfaces. Memory 42 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.

[0148] A program / utility 424 having a set (at least one) of program modules 4241 may be stored, for example, in memory 42. Such program modules 4241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 4241 typically perform the functions and / or methods described in the embodiments of this application.

[0149] The electronic device can also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed through input / output (I / O) interface 44. Furthermore, the electronic device can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) through network adapter 45. Figure 7 As shown, network adapter 45 communicates with other modules of the electronic device via bus 43. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0150] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. An industrial anomaly detection method, characterized in that, The method includes: Acquire the industrial image to be inspected; Visual similarity retrieval is performed based on the industrial image to be detected to obtain similar visual anomaly alerts; Knowledge retrieval is triggered based on the similar visual anomaly prompts to obtain dynamic knowledge prompts; Based on the similar visual anomaly prompts and the dynamic knowledge prompts, industrial anomaly detection is performed on the industrial image to be detected to obtain industrial anomaly detection results.

2. The industrial anomaly detection method according to claim 1, characterized in that, Visual similarity retrieval based on the industrial image to be detected includes: The industrial image to be detected is subjected to visual semantic encoding to obtain a visual semantic representation; Visual similarity retrieval is performed on the visual semantic representation based on a visual experience database to obtain visual precedents; Based on the aforementioned visual precedents, construct similar visual anomaly prompts.

3. The industrial anomaly detection method according to claim 1, characterized in that, The dynamic knowledge tips include: Based on the similar visual anomaly cues, candidate industrial anomaly types are identified; Based on the conceptual knowledge base, knowledge retrieval is performed on the candidate industrial anomaly types to obtain relevant knowledge entries; Dynamic knowledge prompts are constructed based on the relevant knowledge entries.

4. The industrial anomaly detection method according to claim 1, characterized in that, Industrial anomaly detection of the industrial image to be detected based on the similar visual anomaly cues and the dynamic knowledge cues includes: The image to be processed and its visual semantic representation are perceived to obtain perceptual information; Based on the perceptual information, the similar visual anomaly prompts are compared with the dynamic knowledge prompts to obtain industrial anomalies in the image to be processed for inference, and a structured industrial anomaly detection result is output.

5. The industrial anomaly detection method according to claim 1, characterized in that, The method further includes constructing a visual experience database and a concept knowledge base; wherein, it includes Acquire several industrial image samples and label the industrial image samples with target and category tags to obtain an image dataset; The category labels marked in the industrial image samples are mapped to the corresponding target location information to obtain a spatial location-category mapping set; The abnormal targets and abnormal category labels marked in the industrial image samples are used as an abnormal label set; The image dataset, the spatial location and category mapping set, and the anomaly annotation set are used as the industrial dataset; The visual experience database and the conceptual knowledge base are constructed based on the industrial dataset.

6. The industrial anomaly detection method according to claim 4, characterized in that, The construction of the visual experience database and the concept knowledge base based on the industrial dataset includes: The industrial image samples are subjected to visual semantic encoding to obtain visual semantic empirical representations; The abnormal target, the abnormal category, and the corresponding target location information labeled in the industrial image sample are used as semantic triples; The visual semantic experience representation and the corresponding semantic triple are stored as visual precedents to construct the visual experience database. Based on the industrial image and the corresponding semantic triples and external prior knowledge, domain triples are generated, and the domain triples are stored as knowledge entries to construct the concept knowledge base.

7. The industrial anomaly detection method according to claim 6, characterized in that, Based on the industrial image and the corresponding semantic triples, the domain triples generated from external prior knowledge include: Based on the industrial image and the corresponding semantic triple, the anomaly category is obtained, and an anomaly description is generated; Anomaly handling suggestions are generated based on the aforementioned external prior knowledge; The anomaly category, the anomaly description, and the anomaly handling suggestion are used as the domain triple.

8. An industrial anomaly detection system, characterized in that, The system includes: The acquisition module is configured to acquire the industrial image to be inspected; The visual precedent retrieval module is configured to perform visual similarity retrieval based on the industrial image to be detected in order to obtain similar visual anomaly prompts. The domain knowledge enhancement module is configured to trigger knowledge retrieval based on the similar visual anomaly prompts to obtain dynamic knowledge prompts; The industrial anomaly detection module is configured to perform industrial anomaly detection on the industrial image to be detected based on the similar visual anomaly prompts and the dynamic knowledge prompts, so as to obtain industrial anomaly detection results.

9. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the industrial anomaly detection method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory to cause the electronic device to perform the industrial anomaly detection method according to any one of claims 1 to 7.

Images