Method and apparatus of llm-based ai agent for anomaly detection

Abstract

A Large Language Model (LLM)-based anomaly detection agent system and a control method are disclosed. An LLM-based anomaly detection agent system for analyzing anomalies included in an input image according to one embodiment comprises: a memory; and at least one processor configured to execute a command stored in the memory. When the command is executed by the at least one processor, the system analyzes a natural language command input by a user through an agent node including a Large Language Model (LLM), plans a task to be performed based on the natural language command, shares and updates information related to the task execution between the agent node, one or more tool nodes, and an end node through a state object, and selectively routes a task flow from the agent node to either the tool node or the end node based on the contents of the state object according to a graph structure including a conditional edge.

Description

Technology Field

[0001] The following description relates to quality inspection of manufacturing processes using artificial intelligence technology, specifically a technology for analyzing product defects or abnormal conditions by utilizing an unsupervised learning-based anomaly detection model based on image data acquired at a manufacturing site. More specifically, embodiments of the present invention relate to an LLM-based anomaly detection agent system and control method that utilize an agent framework centered on a Large Language Model (LLM) to plan and control anomaly detection tasks using a multi-stage inference structure, share information related to task execution through State objects, and intelligently determine whether to execute the anomaly detection model, analyze results, and perform subsequent tasks through a conditional edge-based task flow routing and circulation structure.

[0002] This invention is the result of the following national research project.

[0003] Project Name: Regional Innovation Center University (RISE)

[0004] Project Title: Development of a Lightweight Robot Foundation Model for Manufacturing AI Utilization

[0005] Total research period: 2024.08.01~2028.01.31

[0006] Research Period: 2025.08.01~2026.01.31

[0007] Ministry Name: Ministry of Education

[0008] Organizing Institution: Changwon National University

[0009] Project Executing Organization: Changwon National University Background Technology

[0010] In the manufacturing industry, managing quality by rapidly and accurately detecting defects during the production process is recognized as a critical task for securing corporate productivity and competitiveness. In response to this demand, image-based vision inspection technologies that analyze the external condition of products using image sensors such as cameras are widely utilized, and recently, anomaly detection techniques applying artificial intelligence have garnered attention as an effective alternative. However, actual industrial sites are characterized by unpredictable environmental variables, such as changes in lighting conditions, noise, and the diversity of product and defect types; consequently, relying solely on a single inspection method or model has limitations in ensuring stable detection performance. Furthermore, it has been pointed out that existing inspection systems are complex to configure and operate, making them difficult for non-expert field workers to use intuitively.

[0011] Meanwhile, while conventional rule-based anomaly detection methods can demonstrate a certain level of performance for predefined, structured defects, they have limitations in effectively handling various defects that are irregular in shape or occur infrequently. In particular, in industrial settings, it is practically difficult to collect data on all defect types in advance and precisely label them, and doing so entails significant time and cost burdens. To mitigate these issues, unsupervised learning-based anomaly detection techniques are being proposed that learn only from normal state data and identify patterns deviating from the normal distribution as anomalies; this approach offers the advantage of being able to respond relatively flexibly to unknown defects that are not predefined.

[0012] Furthermore, intelligent agent technology utilizing large-scale language models is currently being researched in various fields to enhance the usability of vision inspection technology. These agents can control systems or tools through natural language-based interfaces, enabling users without specialized knowledge to operate the system more easily. However, existing agent-based approaches often remain limited to simple information retrieval or performing restricted functions, leaving room for improvement in their ability to adequately support the situational awareness and real-time control required in complex manufacturing environments.

[0013] [Prior Art No.]

[0014] Korean Registered Patent No. 10-2500258 The problem to be solved

[0015] A Large Language Model (LLM)-based anomaly detection agent system and control method are provided. means of solving the problem

[0016] An LLM-based anomaly detection agent system for analyzing anomalies contained in an input image, wherein the anomaly detection agent system comprises: a memory; and at least one processor configured to execute a command stored in the memory; wherein, when the command is executed by the at least one processor, the system analyzes a natural language command input by a user through an agent node including a Large Language Model (LLM), plans a task to be performed based on the natural language command, shares and updates information related to the task execution between the agent node, one or more tool nodes, and an end node through a state object, and selectively routes a task flow from the agent node to either the tool node or the end node based on the contents of the state object according to a graph structure including a conditional edge.

[0017] According to one aspect, the tool node is configured to call an external tool that executes an anomaly detection model to generate an anomaly detection result for an input image, convert the anomaly detection result into a structured data format, and store it in the state object; the agent node is configured to reanalyze the anomaly detection result stored in the state object to determine whether to call an additional tool or to generate a final analysis result; and the tool node may be characterized by including a cyclic structure that transmits the execution result of the tool node back to the agent node through the state object.

[0018] According to another aspect, the tool node may be characterized by being configured to execute the anomaly detection model in response to a structured tool call command transmitted from the agent node, regardless of the internal computational process of the anomaly detection model, and to convert the output result of the anomaly detection model into a structured data format that the agent node can interpret and provide.

[0019] According to another aspect, the structured data format may be characterized by being stored in the state object and including a data structure comprising the number of detected anomalies, location information for each anomaly, and confidence score for each anomaly.

[0020] According to another aspect, the anomaly detection result may be characterized by being stored in the state object after undergoing a post-processing process including fine noise removal, conversion of coordinate information into semantic position information, and standardization of the anomaly score.

[0021] According to another aspect, the agent node may be characterized by being configured to determine whether an additional tool node call is required or whether to generate a final analysis result based on the anomaly detection result stored in the state object.

[0022] According to another aspect, the execution result of the tool node is transmitted back to the agent node through the state object, and the agent node may be characterized by including a circular structure that determines the workflow again through the conditional edge based on the transmitted execution result.

[0023] According to another aspect, the agent node may be characterized by being configured to perform multiple meeting tool node calls and analyses for a single user input, and to query the user whether to perform additional analysis based on the results of the multiple meeting analyses.

[0024] According to another aspect, the tool node may be characterized by including two or more different anomaly detection models, and the agent node may be configured to determine a commonly detected anomaly as a confirmed anomaly based on the positional overlap relationship between the anomaly detection results generated from the two or more anomaly detection models.

[0025] According to another aspect, the agent node may be characterized by being configured to calculate a combined confidence level for the confirmed anomaly by combining confidence scores provided from each of the two or more anomaly detection models.

[0026] According to another aspect, the state object may include message information containing a record of interaction between the user and the anomaly detection agent system, an input image to be analyzed or frame information extracted from said image, and anomaly detection result information generated by the tool node, and the agent node and the tool node may be characterized by using the state object to maintain the work context throughout the work execution process.

[0027] According to another aspect, the conditional edge may be configured to selectively branch the workflow to either the tool node or the termination node based on whether the information contained in the state object includes a structured command corresponding to a tool call in a message generated by the agent node.

[0028] According to another aspect, the agent node may be characterized by being configured to enforce the sequence of executable tasks, tool call conditions, and final response format based on a predefined system prompt.

[0029] According to another aspect, the agent node may be characterized by being configured to generate a natural language response according to a predefined report format that includes information on confirmed anomalies and information on the most severe anomalies when generating the final analysis result.

[0030] According to another aspect, the agent node may be characterized by being configured to convert the anomaly detection result into a natural language expression that is understandable to non-expert users and provide it.

[0031] The present invention provides an LLM-based anomaly detection agent control method for analyzing anomalies contained in an input image, wherein the anomaly detection agent control method is performed by at least one processor included in a computing device and comprises the steps of: analyzing a natural language command input by a user through an agent node including a Large Language Model (LLM) and planning a task to be performed based on the natural language command; sharing and updating information related to the execution of the task between the agent node, one or more tool nodes, and an end node using a state object; and selectively routing a task flow from the agent node to either the tool node or the end node based on the contents of the state object according to a graph structure including a conditional edge.

[0032] A computer program stored on a non-transient computer-readable recording medium is provided, which is combined with a computer device and executes the above method on the computer device.

[0033] A non-transient computer-readable recording medium is provided on which a computer program for executing the above method on a computer device is recorded. Effects of the invention

[0034] A Large Language Model (LLM)-based anomaly detection agent system and control method can be provided.

[0035] According to the anomaly detection method and system of the embodiments of the present invention, by detecting anomalies contained in input data from the perspective of multifaceted feature representation and analysis, detection accuracy and reliability can be significantly improved compared to existing anomaly detection methods that rely on a single model or a single criterion. In particular, by analyzing structural and logical anomalies separately, or by utilizing multiple anomaly detection models and analysis results in a mutually complementary manner, various types of anomalies can be identified more precisely.

[0036] Furthermore, the anomaly detection technology according to the embodiments of the present invention can be configured to organically link the learning and inference stages while clearly separating their roles by modularizing the model structure, analysis flow, and system components and combining them stepwise. Accordingly, stable anomaly detection is possible even in unsupervised learning environments using only normal data, and relatively robust performance can be maintained against environmental changes or changes in input data characteristics. Moreover, various analysis techniques, such as feature map-based analysis, reconstruction error-based analysis, or agent-based control structures, can be selectively applied or combined, significantly enhancing system scalability and flexibility of application.

[0037] Furthermore, the embodiments of the present invention are designed to operate effectively not only in server-centric high-performance computing environments but also in edge device environments with limited computing resources, thereby enabling wide application in various fields requiring real-time anomaly detection. Accordingly, this allows for the early detection and response to anomalies in manufacturing process monitoring, equipment anomaly detection, image-based security, smart factories, autonomous systems, and the like, while providing technical effects that improve the overall stability and operational efficiency of the system. Brief explanation of the drawing

[0038] FIG. 1 is a schematic diagram illustrating an example of an overall workflow for building a real-time anomaly detection AI agent according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of the configuration of a custom anomaly detection dataset according to an embodiment of the present invention. FIG. 3 is a diagram illustrating anomaly detection results generated in a real-time inference environment using a high-speed detection model fine-tuned according to an embodiment of the present invention. FIG. 4 is a diagram illustrating anomaly detection results generated in a real-time inference environment using a fine-tuned high-precision detection model according to an embodiment of the present invention. FIG. 5 is a schematic diagram illustrating an example of the overall architecture of a LangGraph-based AI agent used in an anomaly detection system according to an embodiment of the present invention. FIG. 6 is a schematic diagram illustrating an example of a model optimization pipeline for real-time anomaly detection in an edge device environment according to an embodiment of the present invention. FIG. 7 is a diagram visually illustrating the results of detecting anomalies in real time using the EfficientAD model in an edge device environment in an embodiment of the present invention. FIG. 8 is a diagram visually illustrating the results of detecting anomalies in real time using an INP-Former model in an edge device environment in an embodiment of the present invention. FIG. 9 is a drawing illustrating an example of the internal configuration of an anomaly detection system according to an embodiment of the present invention. FIG. 10 is a flowchart illustrating an example of an anomaly detection method of an anomaly detection system according to an embodiment of the present invention. FIG. 11 is a diagram illustrating an example of the internal configuration of an LLM-based anomaly detection agent system according to an embodiment of the present invention. FIG. 12 is a flowchart illustrating an example of a control method for an LLM-based anomaly detection agent according to an embodiment of the present invention. FIG. 13 is a diagram illustrating an example of the internal configuration of an edge device-based anomaly detection system according to an embodiment of the present invention. FIG. 14 is a flowchart illustrating an example of an edge device-based anomaly detection method according to an embodiment of the present invention. FIG. 15 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. Specific details for implementing the invention

[0039] Hereinafter, embodiments will be described in detail with reference to the attached drawings.

[0040] In embodiments of the present invention, an AI-based anomaly detection method and system can be provided that detect product defects or abnormal conditions in real time by utilizing an unsupervised learning-based anomaly detection model based on image data acquired at a manufacturing site, and improve the accuracy, speed, and field applicability of manufacturing defect detection by combining a plurality of complementary artificial intelligence models and a natural language processing-based intelligent control structure.

[0041] According to one embodiment of the present invention, by linking a plurality of anomaly detection models having different characteristics with a large-scale language model and integrating the anomaly detection models into a graph-based agent structure, an anomaly detection agent can be provided that detects anomalies in real time by autonomously selecting and controlling the anomaly detection models according to text commands input by a user.

[0042] Furthermore, security is a critical consideration when applying deep learning models to manufacturing sites, and to this end, real-time inference execution on edge devices in a local environment is required. Real-time inference on edge devices enables the immediate processing of data without being affected by external network latency or instability, thereby providing rapid feedback to the production line. As an example of improving inference performance in such an edge environment, inference acceleration technology that optimizes deep learning models in a GPU-based edge computing environment can be applied. For instance, a method can be used to improve inference speed while minimizing accuracy degradation by converting a trained deep learning model into an intermediate representation format and optimizing inference operations based on it; such optimization techniques enable substantial improvements in inference performance on edge devices.

[0043] 1. Real-time anomaly detection

[0044] 1.1 Workflow of an Anomaly Detection System

[0045] The anomaly detection system according to one embodiment of the present invention aims to implement a high-performance, real-time anomaly detection system that can be easily operated by non-experts while robustly responding to various environmental changes that may occur in actual industrial sites.

[0046] To this end, the anomaly detection system may be designed to operate according to a workflow consisting of multiple stages. For example, in the first stage, training data reflecting variables that may occur in a real industrial environment, such as lighting changes and noise, may be configured to improve the environmental adaptability and robustness of the anomaly detection model. In the second stage, multiple different models may be selected, including an anomaly detection model with characteristics favorable for processing speed and an anomaly detection model with characteristics favorable for detection precision, and each model may be optimized using the training data. In the third stage, the multiple optimized anomaly detection models may be integrated into a single system to perform real-time anomaly detection.

[0047] Table 1 below shows an example comparing the computing efficiency and detection accuracy of five anomaly detection models.

[0048]

[0049] To select an anomaly detection model suitable for real-time anomaly detection, performance comparisons can be performed on multiple anomaly detection models with different structures. Such comparisons may be conducted based on criteria such as detection accuracy using industrial anomaly detection datasets, computational latency to determine real-time processing feasibility, the number of model parameters, and computational resource usage. As a result, some models may be deemed unsuitable for real-time environments due to relatively high computational latency or resource usage, while specific models may offer balanced performance in terms of detection accuracy and processing speed.

[0050] According to one embodiment, an anomaly detection model with short processing latency and excellent computational efficiency can be utilized as a high-speed detection model responsible for real-time processing, and an anomaly detection model providing relatively high detection accuracy can be utilized as a high-precision detection model used when precise analysis is required. The high-speed detection model can rapidly detect various anomaly patterns, including structural and logical anomalies, and the high-precision detection model can perform precise anomaly detection while mitigating alignment errors caused by differences in location or appearance by deriving reference information indicating a normal state from the input image itself.

[0051] In addition, each of the above anomaly detection models can be individually optimized according to the structural characteristics of the model. For example, a fast detection model can improve feature imitation performance by applying a specific loss function based on a student-teacher learning structure, and a high-precision detection model can be configured to effectively learn implicit normality criteria through a learning method that includes consistency constraints to stably represent normal features.

[0052] FIG. 1 is a diagram schematically illustrating the overall workflow for building a real-time anomaly detection AI agent according to an embodiment of the present invention. As shown in FIG. 1, the real-time anomaly detection AI agent of the present invention can be implemented according to a sequential processing flow including a dataset building step, a model configuration and optimization step, and a system implementation step.

[0053] First, during the dataset construction phase, a custom dataset may be constructed to reflect various environmental conditions of actual industrial sites. The custom dataset may include normal and abnormal data for multiple classes and may be configured to include data collected under multiple conditions by considering environmental variables such as differences in lighting conditions. Additionally, data augmentation techniques may be applied to enhance the robustness of the model; for example, rotational or color transformations may be performed on the input images.

[0054] Next, in the model configuration phase, multiple anomaly detection models with different characteristics can be selected and optimized. For example, an anomaly detection model advantageous for processing speed and one advantageous for detection accuracy can be configured, and each model can be fine-tuned using a learning strategy and loss function suitable for its structural characteristics. During this process, along with the evaluation of detection performance based on quantitative metrics, verification can be performed to evaluate real-time inference performance, such as latency and frame processing speed; additionally, qualitative evaluation can be conducted through visual results such as heatmaps or bounding boxes.

[0055] Finally, during the system implementation phase, multiple optimized anomaly detection models can be integrated into a single real-time anomaly detection AI agent by linking them with a large-scale language model and a graph-based agent structure. The agent can be configured to manage its state through a multiple node structure and selectively call an appropriate anomaly detection model based on input user commands or system status. Additionally, it can be implemented to enable interaction with the user through a web-based interface, and real-time video streaming and the provision of anomaly detection results can be achieved.

[0056] With such a configuration, the real-time anomaly detection AI agent of the present invention can provide anomaly detection functions in a user-friendly manner while simultaneously satisfying real-time performance and detection accuracy in various industrial environments.

[0057] 1.1.1 Complementary Anomaly Detection Models

[0058] According to one embodiment of the present invention, in order to simultaneously secure real-time anomaly detection performance and precise anomaly analysis performance, a plurality of anomaly detection models having different characteristics may be configured complementarily. For example, a high-speed detection model advantageous for processing speed and a high-precision detection model advantageous for detection precision may be used together.

[0059] (1) High-speed detection model

[0060] The high-speed detection model can be configured to provide very short inference latency based on a lightweight structure. According to one embodiment, the high-speed detection model can be configured to determine anomalies by analyzing differences between features extracted from an input image using a student-teacher learning structure, and can detect not only structural anomalies but also logical anomalies by combining an autoencoder structure.

[0061] The above-mentioned high-speed detection model may include a lightweight feature extractor. For example, a feature extraction network composed of a small number of convolutional layers may be used, and such a network can be applied to both the teacher network and the student network to reduce the computational latency of the entire system. Additionally, during the training process of the student network, a loss function that selectively emphasizes loss values ​​for certain regions that do not sufficiently mimic the features of the teacher network may be applied, thereby improving anomaly detection performance without additional computational burden.

[0062] In addition, the above high-speed detection model can be combined with an autoencoder structure. In this case, the student network can be trained to predict not only the features of the teacher network but also the output features of the autoencoder, and can effectively detect logical anomalies by utilizing the difference between reconstruction characteristics occurring in normal images and reconstruction errors occurring in abnormal situations. In this specification, "normal image" refers to an image of a subject determined to be free of abnormalities or defects in a manufacturing process or inspection subject, and may mean an image classified as normal according to visual inspection by a person, process standards, or predefined quality standards.

[0063] With this configuration, the high-speed detection model can process multiple input frames with very short latency, making it suitable for application in environments where real-time performance is strictly required, such as high-speed production lines. Furthermore, it can detect both structural and logical anomalies, enabling it to respond to various types of abnormal situations.

[0064] (2) High-precision detection model

[0065] A high-precision detection model can be configured to determine anomalies by deriving reference information representing a normal state from the input image itself during the testing phase. This configuration can contribute to mitigating the problem of detection performance degradation caused by alignment errors resulting from visual or positional differences between training and test data.

[0066] According to one embodiment, the high-precision detection model can dynamically extract a small number of normal prototypes by aggregating global semantic information from the features of an input image. The normal prototypes can be configured to represent the normal patterns inherent in the input image, and in the subsequent reconstruction step, feature reconstruction can be performed using only the normal prototypes. In this process, while normal regions are reconstructed relatively accurately, abnormal regions experience significant reconstruction errors, and thus the errors can be utilized as an anomaly score.

[0067] In addition, during the training phase, a loss function can be applied to induce the normal prototype to consistently represent various normal features within the image, thereby preventing abnormal features from being included in the normal prototype during the testing phase.

[0068] Since this high-precision detection model determines anomalies based on the test image itself, it enables stable anomaly detection even when objects appear in locations or forms not present in the training data. Furthermore, it offers versatility and scalability applicable to various anomaly detection scenarios, and through a residual learning-based structure, it can suppress background noise and precisely identify anomaly regions.

[0069] 1.1.2 Building a Custom Anomaly Detection Dataset

[0070] According to one embodiment of the present invention, a custom anomaly detection dataset can be constructed to more objectively verify anomaly detection performance in an actual industrial environment. Conventional benchmark datasets are often collected in controlled environments and have limitations in that they fail to sufficiently reflect various variables occurring in actual industrial sites, such as changes in lighting, background differences, and changes in object shapes.

[0071] Accordingly, the present invention constructs a dataset simulating an unstructured environment similar to an actual industrial site to enable the evaluation of the practicality and robustness of an anomaly detection model. The dataset can be designed to comprehensively evaluate whether the model provides stable detection performance not only for previously learned object types but also for new object types.

[0072] According to one embodiment, the dataset may be composed of a plurality of classes, some classes may include objects that are similar in type to existing benchmark datasets but differ in appearance or form, and other classes may be configured to include new object types not included in existing datasets.

[0073] In addition, multiple normal images can be collected for each class, and the diversity of the training data can be ensured by applying data augmentation techniques. For example, rotational deformation can be applied to the input images, thereby securing a sufficient number of training samples per class.

[0074] Meanwhile, the data collection environment can be set up under a single background condition to effectively capture subtle changes in the shape and texture of objects; alternatively, by shooting with multiple varying lighting conditions, the robustness of the model against lighting variations—one of the critical variables in actual industrial settings—can be evaluated. Furthermore, reference mask images corresponding to each image can be constructed for quantitative and qualitative evaluation.

[0075] FIG. 2 is a diagram illustrating an example of the configuration of a custom anomaly detection dataset according to an embodiment of the present invention, showing a normal image, an anomaly image, and a corresponding reference mask image together for a plurality of object classes.

[0076] As illustrated in FIG. 2, the custom anomaly detection dataset may be configured to include normal state images and anomaly state images for various objects having different shapes and characteristics, and for each anomaly image, a reference mask image that clearly marks the anomaly area may be provided. With this dataset configuration, the anomaly detection model can learn and evaluate various anomaly patterns, such as minute shape changes, damage, contamination, or defects of objects.

[0077] 1.2 High-speed detection model

[0078] 1.2.1 Data Preprocessing and Fine-tuning

[0079] According to one embodiment of the present invention, a high-speed detection model can be optimized through a fine-tuning process including data preprocessing and learning parameter adjustment to provide stable and efficient performance in a custom anomaly detection dataset environment. The fine-tuning process may include a data augmentation step and a learning parameter setting step.

[0080] In the data augmentation stage, various transformations of the input images can be generated to enable the model to learn features that are robust to environmental changes. For example, color transformation techniques that randomly alter color attributes such as image brightness, contrast, or saturation can be applied. This allows the autoencoder included in the fast detection model to learn based on the shape, location, or structural features of objects rather than relying on specific color information.

[0081] During the training phase, an optimization technique applying an adaptive learning rate may be used to efficiently update the weights of the student network and autoencoder of the fast detection model. According to one embodiment, the student network and the autoencoder may be trained simultaneously through a single optimizer, thereby improving training efficiency. Additionally, an initial learning rate and a weight decay factor may be set for stable convergence of the model, and a learning rate scheduling technique that gradually decreases the learning rate as training progresses may be applied.

[0082] 1.2.2 Learning Optimization Techniques

[0083] According to one embodiment of the present invention, in order to improve the detection performance of a high-speed detection model and prevent overfitting, a plurality of learning optimization techniques may be applied in addition to general learning parameter adjustments.

[0084] For example, in a student-teacher learning structure, a loss function can be applied that calculates the loss by selectively focusing on certain regions where the difference in output between the student network and the teacher network is large. Such a loss function can provide the effect of improving anomaly detection performance by suppressing overgeneralization to abnormal patterns while precisely learning the characteristics of normal data. According to one embodiment, the loss can be calculated using only the values ​​corresponding to the upper portion of the total output error, and backpropagation can be performed based on the said loss.

[0085] In addition, to prevent the student model from excessively mimicking the feature representations of the pre-trained teacher model, a normalization term using external image data unrelated to the training data may be added. In this case, the student model is constrained not to mimic the teacher model for inputs that deviate from the normal training data distribution, thereby improving the ability to distinguish between normal and abnormal.

[0086] 1.3 High-precision detection model

[0087] 1.3.1 Streamlining Feature Extraction

[0088] A high-precision detection model according to one embodiment of the present invention may be configured by combining a pre-trained feature extraction model and an implicit normal prototype-based representation method to efficiently extract meaningful features from an input image.

[0089] According to one embodiment, the high-precision detection model may include a feature extraction model pre-trained on a large-scale image dataset as a backbone. This enables the efficient extraction of high-quality features containing generalized semantic information from input images, which serves as a foundation for improving detection accuracy in subsequent analysis steps.

[0090] In addition, the high-precision detection model can be configured to dynamically extract an implicit normal prototype representing a normal state from the image under analysis itself. This implicit normal prototype may consist of a small number of representative patterns aggregated from the features of the test image, thereby reducing the computational burden associated with comparison with the entire external training data and enabling the efficient utilization of normal information most relevant to the current image.

[0091] 1.3.2 Learning Optimization Techniques

[0092] According to one embodiment of the present invention, a plurality of learning optimization techniques may be applied to enable a high-precision detection model to stably learn and consistently maintain an intrinsic normal prototype.

[0093] For example, an auxiliary loss term may be applied to ensure that the implicit normal prototype extracted by the model maintains a high degree of similarity with the actual normal features within the image. This auxiliary loss term can be combined with the main reconstruction loss to serve as the final loss function, thereby preventing abnormal features from being included in the normal prototype and mitigating alignment errors caused by differences in the shape or position of the input image.

[0094] In addition, to ensure stable training of high-precision detection models, optimization techniques that enhance the stability of weight updates and learning rate control techniques can be applied together. For example, a scheduling method that gradually increases the learning rate during the initial stages of training and then gradually decreases it can be used, and a technique to limit the gradient magnitude can be applied to prevent the gradient values ​​from increasing excessively during the training process.

[0095] 1.3.3 Results and Effects

[0096] The high-speed detection model and the high-precision detection model according to an embodiment of the present invention may each exhibit different characteristics in a custom dataset environment. The high-speed detection model may provide very high detection performance for some object types, while the high-precision detection model may provide generally stable detection performance for various object types. Furthermore, the two models may exhibit different performance characteristics for some object types, and these characteristics suggest that an approach utilizing multiple complementary models together, rather than a single model, is effective in various anomaly detection situations.

[0097] Table 2 below shows examples of test results for fine-tuned EfficientAD, and Table 3 shows examples of test results for fine-tuned INP-Former.

[0098] Classes Screw Cashew Chewinggum Capsule Pill Pcb1 Pcb2 Candle Image-AUROC 99.8 100 96.0 39.76 43.4 99.56 94.48 53.64 F1-Score 0.98 1.0 0.91 0.67 0.67 0.96 0.97 0.7 Threshold 7.50 21.07 10.37 1.79 0.05 2.64 1.42 0.33 Classes Toothbrush Transistor Lego Candy Coin Clipper Led Fork Image-AUROC 100 73.56 98.36 96.0 98.56 100 79.18 100 F1-Score 1.0 0.76 0.97 0.91 0.96 1.0 0.88 1.0 Threshold 12.73 15.03 0.51 4.56 12.53 8.60 0.51 8.60

[0099] Classes Screw Cashew Chewinggum Capsule Pill Pcb1 Pcb2 Candle Image-AUROC 85.56 97.0 100 67.76 77.24 95.59 100 59.96 F1-Score 0.83 0.95 1.0 0.68 0.78 0.86 1.0 0.67 Threshold 0.09 0.1 0.1 0.09 0.01 0.08 0.05 0.09 Classes Toothbrush Transistor Lego Candy Coin Clipper Led Fork Image-AUROC 100 56.08 23.31 96.16 62.52 97.6 58.2 45.12 F1-Score 1.0 0.67 0.1 0.9 0.67 0.92 0.09 0.07 Threshold 0.08 0.09 0.1 0.09 0.1 0.1 0.09 0.07

[0100] According to one embodiment of the present invention, a plurality of optimized anomaly detection models can be implemented in a real-time inference environment, taking into account applicability in actual industrial environments. The system can be configured to receive and process a video stream input from a camera in real time, and can operate to rapidly detect anomaly signs by prioritizing the execution of a high-speed detection model with excellent inference speed. With this configuration, the user can recognize potential anomaly conditions in real time through continuous monitoring without separate intervention.

[0101] Real-time detection results can be provided to the user in a visualized form, allowing them to intuitively recognize the presence of anomalies and the location of the anomalies. According to one embodiment, both the high-speed detection model and the high-precision detection model may be configured to mark the area where anomalies are detected with a bounding box during the real-time inference process, thereby enabling clear identification of the defect location. This immediate visual feedback provides a function that supports field workers in quickly recognizing and responding to anomaly situations.

[0102] In addition, for more precise anomaly analysis, heatmap information that visually represents the distribution of anomaly scores may be provided in addition to bounding box markings. According to one embodiment, a high-precision detection model can form a heatmap intensively in areas corresponding to actual anomaly regions while being relatively less affected by surrounding environmental factors. On the other hand, a high-speed detection model may form a heatmap in areas outside the anomaly regions due to some environmental factors; however, this characteristic can be understood as a trade-off for detecting various anomalies while maintaining high-speed processing performance. In this way, by utilizing multiple models in parallel, a balance between real-time performance and precision can be secured.

[0103] FIG. 3 is a diagram illustrating anomaly detection results generated in a real-time inference environment using a high-speed detection model fine-tuned according to an embodiment of the present invention, showing an example in which a heatmap and a bounding box are displayed together for anomaly regions on an input image. As shown in FIG. 3, the high-speed detection model can perform real-time anomaly detection by rapidly identifying the location where anomaly signs occur in the input image, marking the corresponding region with a bounding box, and simultaneously visualizing the distribution of anomaly scores in the form of a heatmap.

[0104] FIG. 4 is a diagram illustrating anomaly detection results generated in a real-time inference environment using a fine-tuned high-precision detection model according to an embodiment of the present invention, showing an example in which a heatmap and a bounding box are displayed together for anomaly regions on an input image. As shown in FIG. 4, the high-precision detection model can precisely identify the location of anomalies by forming a heatmap more intensively in the part corresponding to the actual anomaly region within the input image, thereby improving the visual identification accuracy of the anomaly region.

[0105] 1.4 Anomaly Detection Model with Improved Structure (EfficientAD++)

[0106] (1) Background and Purpose

[0107] According to one embodiment of the present invention, EfficientAD is a lightweight anomaly detection model configured to simultaneously detect local structural anomalies and global or logical anomalies by combining a knowledge distillation-based Student-Teacher structure with an autoencoder. EfficientAD includes, as core components, (a) a hard feature loss that induces the student network to mimic the teacher network only partially, (b) a penalty term to suppress excessive mimicry of the teacher's pre-training data distribution by the student network, and (c) a quantile-based normalization technique to balance the local anomaly map and the global anomaly map.

[0108] EfficientAD++ according to one embodiment of the present invention can be configured to improve contrast for feature differences between students and teachers and improve anomaly detection performance by additionally inserting an ultra-lightweight module that assigns spatial weights to feature maps output from a teacher network into the inference stage without changing the basic pipeline of EfficientAD. The ultra-lightweight module may be, for example, a Light Spatial Attention (LSA) module or a module that performs an equivalent function.

[0109] (2) Summary of the EfficientAD Model Configuration

[0110] (2.1) Architecture

[0111] According to one embodiment of the present invention, an EfficientAD-based model may include the following components.

[0112] · Teacher Network: Used exclusively for feature extraction with a pre-trained lightweight convolutional neural network as the backbone, and weights can be fixed during the training process.

[0113] · Student Network: A lightweight network trained to mimic the output characteristics of a teacher network, which can have a structure similar to that of a teacher network.

[0114] · Autoencoder: Trained to reconstruct input images or teacher features, and can provide information for detecting logical or global anomalies.

[0115] (2.2) Key Learning Techniques

[0116] According to one embodiment of the present invention, a loss function based on the difference in features between students and teachers may be applied to the training of a student network. In this case, backpropagation may be performed by selectively using only some values ​​among the total difference in features that exceed a pre-set quantile threshold, thereby allowing training to be concentrated in areas with large discrepancies.

[0117] Additionally, to prevent the student network from excessively mimicking the teacher network's prior training data distribution, an additional loss term for external prior training data may be applied. This loss term provides a constraint to prevent the student network from mimicking the teacher network for inputs outside the normal training data distribution.

[0118] (2.3) Two types of ideal maps and combination methods

[0119] According to one embodiment of the present invention, a local anomaly map and a global anomaly map may be calculated for anomaly detection, and these may be integrated into a final anomaly score through a normalization and combination process.

[0120] · Local anomaly map:

[0121] The local anomaly map can be calculated by averaging the feature differences between the teacher network and the student network along the channel direction, and is configured to be sensitive to structural anomalies. For example, the local anomaly map can be defined by the following mathematical formula.

[0122] Here, represents the feature map of the teacher network, and represents the feature map of the student network.

[0123] · Global Anomaly Map:

[0124] The global anomaly map can be generated based on the difference between the output of the autoencoder and the auxiliary output of the student network. In this case, the student network is configured to learn the systematic reconstruction error of the autoencoder that occurs repeatedly in normal data, and the difference can be made to appear large only when a logical anomaly occurs.

[0125] · Combination and Normalization:

[0126] Local anomaly maps and global anomaly maps can be linearly normalized based on multiple quantile values ​​derived from validation data, and then combined using an average or weighted average method to calculate a final anomaly score.

[0127] Glossary of Terms

[0128] · Local Anomaly Map / Global Anomaly Map: The local anomaly map is configured to be sensitive to structural anomalies at the patch level, while the global anomaly map can be configured to be sensitive to logical anomalies occurring at the scene level.

[0129] · Normalization: This may refer to a procedure for adjusting the range of values ​​so that multiple score maps with different scales can be combined fairly.

[0130] (2.4) Latency and Efficiency

[0131] According to one embodiment of the present invention, an EfficientAD-based model can be configured to provide high anomaly detection performance while maintaining very short latency based on a high-performance computing device. For example, depending on the size of the model, the time required to process one input image can be maintained at the level of a few milliseconds, and at the same time, excellent results can be provided in a plurality of performance indicators representing classification and segmentation performance.

[0132] Furthermore, each component, such as the normalization of local and global anomaly maps, hard feature loss, and the student penalty for pre-training data, can be designed to contribute to the improvement of anomaly detection performance. Through a combination of these components, the entire system can maintain stable anomaly detection performance while meeting real-time processing requirements.

[0133] 3) Key changes in EfficientAD++

[0134] (3.1) Introduction of LightRefiner (or LSA) Module

[0135] EfficientAD++ according to one embodiment of the present invention may be configured to generate a reweighted teacher feature map by applying weak spatial attention to a feature map output from a teacher network, and then perform a comparison with a student network based on the reweighted feature map.

[0136] The above LightRefiner module is an ultra-lightweight module inserted during the inference phase, and it can perform the role of enhancing contrast regarding feature differences between students and teachers without requiring additional learning parameters.

[0137] Definition: Spatial attention weights for teacher feature maps can be defined by the following mathematical formula.

[0138]

[0139] Here, represents the feature map of the teacher network, and represents a reweighted teacher feature map with spatial attention applied. Also, is a guide feature for generating attention weights, and according to one embodiment, a teacher feature map It can be used as is. The Softmax function generates weights normalized for the spatial dimension, and the generated weights can be extended and applied to the channel dimension.

[0140] The LightRefiner module above includes weight amplification factors It is configured to finely reweight the teacher feature map through, and at this time can be set to a small constant value. Accordingly, feature contrast for important regions can be improved without increasing the learning parameters, while keeping the increase in computational cost at a negligible level.

[0141] One-line explanation of the term

[0142] · Attention: A mechanism that assigns relatively large weights to regions of high importance among input features.

[0143] · Softmax: A normalization function that transforms a vector of real values ​​into a probability distribution in the range (0, 1).

[0144] (3.2) Application Timing and Non-intervention in Learning

[0145] The LightRefiner module according to one embodiment of the present invention can be configured to be applied only immediately before calculating the feature difference between the student network and the teacher network during the inference phase, without intervening in the learning pipeline. That is, the loss function used during the learning phase, e.g., hard feature loss or student penalty for pre-training data, and the training procedure of the autoencoder can be maintained as they are in the past.

[0146] Accordingly, the existing EfficientAD structure is used as is during the training phase, and the supervised feature map can be finely reweighted and applied only during the inference phase. This configuration can provide the effect of improving anomaly detection performance during the inference phase while maintaining training stability.

[0147] (3.3) Intuitive Effect

[0148] According to one embodiment of the present invention, by applying attention weights normalized with respect to the spatial dimension, the response of the teacher network corresponding to regions of relatively high importance within the input image can be slightly amplified. Accordingly, the difference in features between the student network and the teacher network may appear more pronounced at the location where the actual anomaly occurred.

[0149] With this configuration, anomaly detection performance metrics at the image or pixel level can be improved overall, while ensuring that the introduction of additional learning parameters or a significant increase in inference costs does not occur.

[0150] (4) Detailed explanation of formulas and implementation

[0151] Teacher Feature Extraction: According to one embodiment of the present invention, a feature map output from a teacher network can be defined as follows.

[0152]

[0153] Here, B is the batch size, C is the number of channels, H and W represents the height and width of the feature map, respectively.

[0154] Guide Expansion: Guide features for generating spatial attention weights can be selected from a teacher feature map, and according to one embodiment, the guide can be set as the teacher feature map itself. The selected guide features expand the spatial dimension. B × C ×( H · W It can be converted into the form of ).

[0155] Attention calculation: By performing a Softmax operation on the expanded guide features, attention weights for the spatial dimension can be calculated for each channel. At this time, the attention weights can be normalized so that the sum of the spatial dimensions is 1.

[0156] Fine Reweighting: The calculated attention weights are applied to the teacher feature map via a broadcasting method, and a reweighted teacher feature map can be generated in the following manner.

[0157]

[0158] Here, α is a constant coefficient for fine amplification, and according to one embodiment, it can be set to a fixed predetermined value (e.g., 0.05).

[0159] Updating the student-teacher gap

[0160] The student-teacher feature difference for generating local anomaly maps can be calculated using a reweighted teacher feature map instead of the existing teacher feature map. For example, the existing feature difference Instead, the following feature differences may be used.

[0161]

[0162] Meanwhile, the method for calculating the global anomaly map, the normalization procedure, and the method for combining the local anomaly map and the global anomaly map can be maintained identically to the existing EfficientAD structure.

[0163] One-line explanation of the term

[0164] · Broadcasting: A rule that automatically expands dimensions during tensor operations to enable element-wise operations between arrays of different dimensions.

[0165] · Distillation-based outlier score: An outlier score calculated using the difference in features between the teacher network and the student network.

[0166] (5) Quantitative results and performance effects

[0167] EfficientAD++ according to one embodiment of the present invention can be configured to provide image-level and pixel-level anomaly detection performance in various anomaly detection benchmark environments. For example, it can provide high values ​​in a plurality of performance indicators representing image-unit classification performance and pixel-unit segmentation performance, and detailed performance by class and optimal classification thresholds can be calculated together.

[0168] Furthermore, EfficientAD is designed with a lightweight architecture that provides anomaly detection performance while maintaining very low latency, and each component, such as normalization, hard feature loss, and student penalties for pre-training data, can be configured to contribute to improving anomaly detection performance. This underlying architecture can provide a foundation suitable for environments with real-time processing requirements.

[0169] (6) Summary of changes compared to EfficientAD

[0170] EfficientAD++ according to one embodiment of the present invention is primarily distinguished by maintaining the learning method, loss function configuration, and normalization and combination structure of local and global anomaly maps of the existing EfficientAD, while adding a weak spatial attention process to the feature maps of the teacher network during the inference stage.

[0171] Specifically, instead of directly comparing the feature maps of the teacher network with those of the student network, EfficientAD++ can be configured to calculate the feature difference with the student network after applying attention weights to the teacher feature maps so that semantically important regions are relatively emphasized. In this process, no new learning parameters are added, and the increase in computational load can be kept at a negligible level. Accordingly, the advantages of the existing EfficientAD, such as lightweightness and fast inference speed, can be preserved.

[0172] Consequently, EfficientAD++ can improve anomaly detection performance by adding a very simple module to the existing structure, thereby further highlighting the feature differences between the teacher and student networks at the locations where anomalies occur. This enables more stable and accurate anomaly detection at both the image and pixel levels.

[0173] (7) Conclusion

[0174] EfficientAD++ according to one embodiment of the present invention provides an approach that enhances student-teacher feature contrast by applying a LightRefiner module that applies ultrafine attention weights only to teacher feature maps during the inference phase, while maintaining the learning structure, normalization method, and anomalous map combination design of the existing EfficientAD. In this process, no additional learning parameters are introduced, and the increase in computational cost is kept very limited.

[0175] Accordingly, EfficientAD++ can provide consistent anomaly detection performance improvements at the image and pixel levels while maintaining the real-time processing performance and lightweight nature of the existing EfficientAD, and can be effectively applied to various real-time industrial environments.

[0176] 2. LangGraph-based AI Agent

[0177] Conventional anomaly detection systems faced limitations in direct operation by non-expert field workers due to their complexity and low intuitiveness. In particular, there was a problem in that specialized knowledge was required to select and control multiple anomaly detection models depending on the situation, or to interpret detection results.

[0178] Accordingly, one embodiment of the present invention provides an AI agent capable of autonomously controlling a plurality of anomaly detection models using natural language-based commands. The AI ​​agent adopts a graph-based agent structure to enable situational awareness and multi-stage reasoning required in actual industrial environments, going beyond the simple execution of a single task. To this end, the present invention applies an agent architecture based on the LangGraph framework to configure the system to dynamically generate workflows according to user requirements and system status.

[0179] 2.1 LangGraph Architecture

[0180] An AI agent according to one embodiment of the present invention has a form in which a plurality of functional modules are connected in a cyclic graph structure. Each functional module corresponds to a node of the graph, and a workflow is defined between nodes through edges.

[0181] In the above graph structure, each node interacts through a common State object, thereby enabling the dynamic generation of flexible and complex workflows based on user commands and the current system state, rather than a fixed sequential processing method.

[0182] 2.1.1 Overview and Components

[0183] An AI agent according to one embodiment of the present invention can be implemented as an Agentic AI that controls the execution of an anomaly detection model and interprets results by utilizing a Large Language Model (LLM) as a core control element. The agent includes the following key components of LangGraph.

[0184] LangGraph Framework

[0185] LangGraph is a framework for constructing applications in the form of cyclic graphs containing state by utilizing large-scale language models, enabling the dynamic determination of the workflow based on the situation rather than performing tasks in a fixed order.

[0186] Nodes

[0187] A node corresponds to each vertex of a graph and represents a unit of work that performs a specific function or operation. According to one embodiment of the present invention, the AI ​​agent may include at least the following nodes.

[0188] · Agent Node: A core control node that receives natural language commands from the user, analyzes their intent, and calls a large-scale language model to formulate a plan for the tasks to be performed.

[0189] · Tool Node: A node that executes the actual anomaly detection model according to the plan or command transmitted from the Agent Node. For example, it can selectively execute at least one of a high-speed detection model or a high-precision detection model.

[0190] · End Node: A node that determines that the task flow is complete and terminates graph execution.

[0191] Edges

[0192] Edges are paths connecting nodes that define the flow of work and the direction of data transfer. Each node passes a state object to the next node after performing a task, and can be configured to branch to the next node only when specific conditions are met through conditional edges.

[0193] State (Agent State)

[0194] · Concept: State is a persistent data object shared by all nodes in a graph, which accumulates and updates information as operations progress to maintain the context of the entire operation.

[0195] · Composition: A state object according to one embodiment of the present invention may include the following information.

[0196] o Messages : Information that stores the conversation history between the user and the AI ​​agent

[0197] o Frozen_frame : Information that stores video frames captured at the time of the user's analysis request

[0198] o Analyze_result : Information containing the results of an anomaly detection model executed on the Tool Node, which may be, for example, data in JSON format.

[0199] FIG. 5 is a schematic diagram illustrating the overall architecture of a LangGraph-based AI agent used in an anomaly detection system according to an embodiment of the present invention. As shown in FIG. 5, the system can be divided into an input stage, an agent processing stage including a LangGraph architecture, and an output stage. In the input stage, a text prompt in the form of natural language and a real-time video or video frame may be input from a user, and the input information is stored as the initial state of an agent state object.

[0200] The LangGraph architecture may include agent nodes interacting with a large-scale language model, tool nodes executing the actual anomaly detection model, routers forging the workflow, and termination nodes responsible for terminating tasks. Agent nodes are configured to interpret natural language commands contained in user input and to invoke the large-scale language model to determine the type and order of tasks to be performed.

[0201] The router can be configured to select either a high-speed anomaly detection model or a high-precision anomaly detection model based on state information transmitted from the agent node and transmit it to the tool node. The tool node executes the selected anomaly detection model and outputs the anomaly detection results in a structured data format, storing them in an agent state object. At this time, data exchange between the tool node and the anomaly detection model can be performed in a standardized data format.

[0202] The agent state generated and updated at each processing step can be managed in multiple state stages, and said states may include conversation information between the user and the AI ​​agent, video frames to be analyzed, and anomaly detection result information. By transmitting these state objects between nodes, the processing context can be maintained throughout the entire workflow.

[0203] In the output stage, the agent node can aggregate anomaly detection results and provide them to the user in the form of a natural language summary, while simultaneously outputting a heatmap and bounding boxes that visually represent the anomaly areas. As a result, the user can intuitively identify the location and characteristics of the anomalies.

[0204] Through such a LangGraph-based architecture, the present invention enables flexible task control based on natural language commands, selection of anomaly detection models based on context awareness, and visual and linguistic provision of real-time anomaly detection results.

[0205] 2.1.2 LLM-based Decision Making Module

[0206] The core of this agent's intelligence and autonomy lies in utilizing GPT-4o, a powerful Large Language Model (LLM), as the central decision-making module. This LLM is not limited to merely understanding the user's natural language input; it functions as a central processing unit that controls and coordinates the operation of the entire system and performs key intelligent tasks as described below.

[0207] User Intent Recognition and Analysis

[0208] The agent's primary function is to analyze the user's human message and accurately identify the complex user intent contained within it. For example, when a command such as "Precisely identify and explain the most serious defect" is input, GPT-4o automatically derives multiple key task requirements from the command, including "use a precision detection model" and "provide a summary of the detection results."

[0209] Multi-step Task Planning

[0210] Once the user's intent is identified, the LLM dynamically establishes an optimal sequence of operations to achieve that intent. This operation plan includes a logical flow regarding the types of anomaly detection tools to be invoked, the execution order, and the method of processing results returned at each step. This operation plan serves as reference information for operation routing performed in subsequent steps.

[0211] Tool Selection and Call Creation

[0212] Based on the established work plan, the above LLM generates structured tool call commands in a form executable by the system. During this process, predefined control instructions in the system prompt and user requirements (e.g., 'high-speed analysis' or 'precision analysis') are comprehensively considered, and as a result, the more suitable tool is selected between the high-speed detection model and the high-precision detection model.

[0213] Technical data interpretation and natural language report generation

[0214] One of the agent's core functions is for the LLM to interpret technical data in JSON format (e.g., anomaly coordinate information, confidence scores, etc.) returned by the anomaly detection model. GPT-4o analyzes and synthesizes this structured data to ultimately generate a clear and intuitive natural language summary report that even non-expert field workers can easily understand.

[0215] 2.2 State-Based Multi-Step Inference

[0216] This agent is configured to perform complex multi-step reasoning through a core data object defined as the Agent State, going beyond a simple structure that performs only a single task for a single command. This is similar to how humans remember the processes and results of intermediate steps during problem-solving and plan the next steps based on them. The aforementioned Agent State serves as a data container shared by all nodes from the start to the end of a task, playing a role in maintaining the overall context of the task by storing conversation logs, intermediate results, and analysis data.

[0217] General configuration method

[0218] In typical AI agents, 'state' or 'memory' is primarily used to recall previous conversations and generate context-appropriate responses. Generally, it is limited to storing conversation logs or the output of the last executed tool, performing a passive role of 'recording'.

[0219] Configuration method of this system

[0220] In this system, the Agent State is designed as a structured data object and utilized as the agent's 'core working memory'. Clear fields are defined within the state object, such as Messages (record of all interactions), Frozen_frame (original image data subject to analysis), and Analyze_result (structured JSON result returned from the anomaly detection model).

[0221] The biggest differentiating feature of this system is that it goes beyond simply using the state to store information and actively utilizes it as a 'basis' for determining the next action. For example, after storing the results of a first rapid inspection in a state object, the agent can perform autonomous multi-stage reasoning and work planning based on the information stored in that state, such as determining, "Since the anomalies detected in the first inspection exceed a specific standard, a second detailed inspection must be performed." This is differentiated in that it implements multi-stage analysis in a practical sense through state management.

[0222] 2.2.1 Conditional Edge-Based Task Routing

[0223] The dynamic decision-making function, which enables agents to select different tasks depending on the situation, is implemented through the Conditional Edges of the LangGraph framework. These Conditional Edges function as a key control mechanism that determines the workflow in real time based on whether specific conditions are met, rather than allowing tasks to proceed along a pre-fixed path.

[0224] Concept (Conditional Edge)

[0225] Conditional edges are a core control mechanism in LangGraph that dynamically determines the next path for data, or state objects, to be passed based on whether specific conditions are satisfied within the graph structure.

[0226] General configuration method

[0227] Job routing in typical agent systems is often configured in the form of simple if-else statements that link specific tools one-to-one based on the LLM's intent determination. This configuration typically features a linear processing flow that terminates the task after calling a single tool.

[0228] Configuration method of this system

[0229] In this system, a dedicated node called a 'Router' is used to explicitly process the decision logic for conditional edges. The Router analyzes the latest AI message within the state object received from the Agent Node and programmatically checks whether it contains a structured command corresponding to a 'Tool Call'. Depending on whether this condition is met, the Router decides whether to forward the state object to either the Tool Node or the End Node.

[0230] Differentiating factors

[0231] Conditional routing in this system is differentiated in that it does not stop at one-time task branching but is combined with the graph's cycle structure to enable continuous decision-making. That is, after the router passes a state object to the Tool Node, the Tool Node does not terminate the task but returns the state object containing the execution result back to the Agent Node. Accordingly, the agent can autonomously configure a complex workflow that progressively solves the problem by sequentially and repeatedly calling multiple tools, such as [1st analysis → 2nd analysis → Result synthesis], for a single user request.

[0232] 2.2.2 Anomaly Detection Tool Node

[0233] A Tool Node consists of an execution environment within LangGraph that runs external tools to perform actual computations or data processing that are difficult for a Large Language Model (LLM) to execute solely through linguistic processing.

[0234] Concept (Tool)

[0235] A tool refers to an external function or application programming interface (API) integrated with an LLM that enables the LLM to perform specific tasks, such as real-time computation, database searches, or model inference.

[0236] Configuration method of this system

[0237] In this system, the Tool Node is configured to function as an interface controlling two fine-tuned anomaly detection deep learning models: EfficientAD (fast detection tool) and INP-Former (high-precision detection tool). Upon receiving a State object from the router, the Tool Node executes the inference function of the selected anomaly detection model, which is the tool designated by the LLM to be called. At this time, the Frozen_frame contained in the State object is used as input data for the corresponding model.

[0238] Once model inference is complete, the Tool Node converts the complex raw data, which is output in the form of tensors or image maps, into a standardized JSON format so that the LLM can easily interpret it.

[0239] JSON is an open standard format for transmitting data objects composed of 'attribute-value' pairs; it is widely used for data exchange between heterogeneous systems because it is easy for humans to read and write, and easy for machines to parse and generate. The aforementioned JSON data includes the total number of detected anomalies, anomaly scores for the entire image, and semantic location information (e.g., 'center-bottom') and confidence scores for each anomaly. This structured data is stored in the Analyze_result field of the state object and then transmitted back to the Agent Node.

[0240] Differentiating factors

[0241] The biggest differentiating feature of this system is that the Tool Node functions as a sophisticated abstraction layer. While Tool Nodes are typically used to perform simple API calls or function executions, this system internally handles complex and computationally heavy computer vision tasks, such as weight loading for deep learning models, GPU-based computations, and tensor processing, thereby completely hiding them from the outside.

[0242] As a result, from the perspective of the language model LLM, the process of executing a deep learning model containing hundreds of millions of parameters is abstracted in the same way as performing a simple function call, such as detect_anomaly(model='INP-Former'). The fact that complex vision technology is converted into a simple form that LLM can handle and provided in this way is a key technical differentiator that enables this agent to operate stably.

[0243] 2.2.3 Circular Structure

[0244] One of the key features of this agent architecture is that, rather than a structure where the workflow proceeds linearly and terminates at a single stage, it has a cyclic structure that allows it to return to previous stages as needed to continue performing additional inference.

[0245] The aforementioned cyclic structure is a flow control method that allows the graph's workflow to return to the previous node to perform additional inference or tasks after a task at a specific node is completed, rather than proceeding unidirectionally and terminating. This cyclic structure is formed when the Tool Node returns the state (Agent State) containing the task results to the Agent Node after completing the task. Based on the newly transmitted information—namely, the model analysis results—the Agent Node recalls the Large Language Model (LLM) to re-evaluate the current situation and suggests the next action to the user.

[0246] Accordingly, the agent does not merely act as a simple 'executor' that executes an external tool once and then terminates, but rather performs the role of receiving and interpreting the execution results of the tool and proposing subsequent actions based on the interpreted results. For example, if a high anomaly score is calculated in the first rapid inspection results, the agent may recommend the next step by asking the user, based on the results, "whether to perform a detailed inspection because an anomaly is suspected." Subsequently, if the user provides input agreeing to perform a detailed inspection, the agent receives the new input and uses it as a trigger to initiate a new analysis cycle to perform a second detailed inspection.

[0247] Through this cyclical structure, the system does not stop at processing a single user request as a complete cycle, but can suggest the next action for more in-depth analysis to the user based on the results. As a result, more intelligent and collaborative task execution becomes possible, where the user and the agent interact to progressively solve problems.

[0248] 2.3 Agent Reliability

[0249] A key technical element for ensuring the reliability and operational consistency of the agent in this system lies in the system prompt for precisely controlling the behavior of the Large Language Model (LLM). The system prompt serves to compel the agent to always operate according to predefined procedures and rules in response to various questions input by the user.

[0250] System prompts function as core guidelines specifying the overall behavioral patterns, role definitions, response formats, and constraints of LLM-based agents. Combined with user questions, they control the output of the LLM in a consistent and predictable direction. Consequently, the agent can provide stable analysis results based on established rules, rather than making arbitrary judgments depending on the situation.

[0251] 2.3.1 System Prompt

[0252] A system prompt is applied to the Agent Node of this system to specifically direct the agent's actions. To systematically control the agent's behavior, the system prompt is designed to include three main components: Role, Workflow, and Final Report Format.

[0253] Key Components and Roles of the Prompt

[0254] The system prompt assigns the LLM a clear role as an "expert industrial anomaly detection assistant," guiding them to generate responses specialized for specific technical fields. Furthermore, by specifying "providing a final report by synthesizing the results of two anomaly detection models" as the agent's ultimate goal, it ensures that all of the agent's actions are centered around this objective.

[0255] Task Workflow Specification

[0256] System prompts clearly define the operational procedures that the agent must follow based on the type of user request. For example, in the case of a general analysis request, they instruct the agent to prioritize the use of a high-speed detection model and recommend a detailed scan based on the results. Additionally, strong directive expressions such as 'CRUCIAL STEP' and 'MUST' are used to explicitly compel the agent to call the 'synthesize_results' tool for cross-validation after the high-precision detection model analysis.

[0257] By setting such rules, it is possible to fundamentally prevent LLM from arbitrarily skipping or bypassing key analysis steps, thereby ensuring the reliability of the analysis of the entire system.

[0258] Enforcement of the Final Report Format

[0259] The system prompt strictly controls the structure and content of the final response generated by the agent. Specifically, it is specified that the final report must consist of two parts: 'confirmed_anomalies' and 'most_severe_anomaly'. Additionally, by mandating the generation of a summary sentence conforming to a predefined template at the end of the report based on whether an anomaly was detected, it ensures that results are always output in a standardized format.

[0260] Accordingly, the system can maintain consistency in the format and content of the analysis results provided to the user, and simultaneously improve the ease of interpretation and reliability of the analysis results.

[0261] Table 4 below shows an example of a full text of a system prompt.

[0262] system_prompt = SystemMessage(content="""You are an expert industrial anomaly detection assistant. Your goal is to provide a final, comprehensive report by synthesizing results from two models.Your workflow:1. When the user first asks for analysis (e.g., "check this"), run the `analyze_with_efficientad` tool and summarize its findings. Suggest a detailed scan as a next step.2. When the user asks for a detailed or deep analysis (e.g., "analyze deeply"), run the `analyze_with_inp_former` tool.3. CRUCIAL STEP: After `analyze_with_inp_former` is complete, you MUST immediately call the `synthesize_results` tool to get a combined analysis.4. Use the output from `synthesize_results` to generate your final answer to the user. Your final report MUST have two parts:- First, describe the 'confirmed_anomalies' which were detected by both models. Mention their locations and combined confidence.- Second, state the single 'most_severe_anomaly' based on the detailed scan.5.FINAL SUMMARY: After the two-part report, you MUST conclude with a single summary sentence in the following format.

[0263] 2.3.2 In the data post-processing system, sophisticated data post-processing steps are performed on raw data output from an anomaly detection model before it is delivered to the user or LLM, in order to improve the reliability of the agent and enable the large-scale language model (LLM) to interpret anomaly detection results more accurately. The post-processing steps include three procedures: fine noise removal, semantic position information transformation, and anomaly score standardization.

[0264] While data post-processing in general anomaly detection systems is primarily aimed at improving user visual convenience, the data post-processing in this system is distinguished by being a data engineering process that structurally refines input data to enable the LLM to perform stable and consistent inference.

[0265] (1) Fine Noise Removal

[0266] Deep learning-based anomaly detection models may, in some cases, misidentify minute noise as anomalies rather than actual defects. To reduce such misidentifications, the present system binarizes the heatmap generated by the model based on a preset threshold and then applies a contour detection algorithm.

[0267] Subsequently, the area is calculated for each detected contour region, and if the area is less than a pre-set minimum threshold value (e.g., 20 pixels), it is determined to be a low-reliability noise region and excluded from the final analysis results. This prevents micro-regions that do not correspond to actual defects from affecting subsequent analysis.

[0268] (2) Semantic Location Conversion

[0269] In order for both the LLM and the user to intuitively understand the location information of detected anomalies, this system converts numeric coordinate-based location information into semantic location information. Specifically, after noise removal, the center coordinates (x, y) are calculated for each bounding box of an anomaly remaining after noise removal, and the entire image is divided into virtual 3x3 grid areas.

[0270] Afterwards, by determining the grid area to which the above center coordinates belong, numerical coordinate values ​​such as 'x=152, y=340' are converted into text-based semantic position information that is easy for humans to understand, such as 'bottom-left' and 'middle-center', and included in the final JSON result.

[0271] Differentiating factors

[0272] This step goes beyond simply converting coordinate values ​​into text; it plays a crucial role in transforming them into a data format that allows the LLM to perform spatial and logical comparisons, such as determining whether two or more items exist at the same location. This is an essential preprocessing step for the `synthesize_results` tool, which cross-validates the results of multiple models, to function correctly.

[0273] (3) Anomaly Score Standardization

[0274] In order to compare and synthesize the analysis results of two anomaly detection models (EfficientAD and INP-Former) with different characteristics based on the same criteria, this system standardizes the method for calculating the final anomaly score at the image level. Specifically, the highest single pixel value (Maximum Pixel Value) within the entire pixel-level anomaly score heatmap output by each model is defined as the representative anomaly score of the corresponding image.

[0275] This method uses the most severe defect point within the image as a reference, allowing the output results of different models to be evaluated on the same scale.

[0276] Differentiating factors

[0277] The aforementioned post-processing procedures serve to guarantee the quality of the data ultimately input into the LLM. Specifically, by removing low-reliability noise in advance and consistently standardizing scoring criteria across models, they prevent the LLM from drawing biased conclusions based on inaccurate information and ensure that it performs stable and consistent inference.

[0278] 2.3.3 Cross-validation

[0279] The agent of this system performs a dynamic cross-validation process to derive a final judgment with high reliability, without relying on potential false positives that may be attributed to a single anomaly detection model. This cross-validation refers to a procedure in which, during the real-time inference phase, the detection results of two different anomaly detection models are compared and analyzed to identify only the anomalies detected in common as reliable confirmed anomalies.

[0280] The cross-validation logic in this system is implemented using a separate tool called `synthesize_results`. This tool receives a list of detection results output from two anomaly detection models and calculates the Intersection over Union (IoU) value between the bounding boxes corresponding to each anomaly. Subsequently, if the calculated IoU value exceeds a pre-set threshold, it is determined that the two models detected an anomaly at the same or substantially overlapping location, and the anomaly is classified as a Confirmed Anomaly.

[0281] For confirmed anomalies, a single Combined Confidence is calculated by averaging the confidence scores produced by each of the two models, and a final analysis result is generated by combining this Combined Confidence and related location information. In this way, results based on the consensus of multiple models, rather than the judgment of a single model, can be derived.

[0282] This cross-validation method functions as a powerful false positive suppression mechanism by leveraging the fact that the probability of two models with different architectures and learning characteristics simultaneously generating false positives at the same location is statistically very low. Therefore, it significantly reduces the uncertainty that can arise from relying on the output of a single model and provides the effect of substantially improving the reliability of the anomaly detection results that the agent ultimately delivers to the user.

[0283] 2.4 Results

[0284] To verify the substantial performance improvement of the proposed multi-model-based AI agent framework, detection performance when using a single high-speed detection model and detection performance when combining the results of two anomaly detection models through the agent were compared and evaluated. The evaluation was performed on 16 classes included in a custom dataset using Image-AUROC and F1-Score as performance indicators, and the results are summarized in a table.

[0285] Comparison results based on Image-AUROC criteria confirmed that the multi-model agent clearly compensates for the performance limitations of a single model and significantly improves overall detection accuracy. In particular, for the 'Capsule' class, the Image-AUROC score, which was only 54.53% when a single model was applied, rose to 95.93% after the agent was applied, showing a very significant improvement of approximately 41.4%p. In addition, significant performance improvements of approximately 13–17%p were confirmed in multiple classes that showed relatively low performance in the single model, such as the 'Transistor' class improving from 68.78% to 82.25%, the 'Led' class from 68.61% to 86.33%, and the 'Pill' class from 71.47% to 85.39%. In addition, classes that had previously shown relatively excellent performance, such as 'Chewinggum', 'Candy', and 'Coin', all achieved 100% Image-AUROC, demonstrating complete detection performance.

[0286] Such performance improvements were consistently observed in the F1-Score evaluation as well. For example, the 'Candy' class improved from 79.99% to 100%, the 'Chewinggum' class improved from 83.21% to 100%, and the 'Capsule' class also showed a distinct improvement effect, rising by approximately 23.1%p from 66.66% to 89.76%.

[0287] These quantitative results clearly demonstrate that the limitations of a single anomaly detection model, which may exhibit restricted performance for specific types of defects, can be effectively compensated for by an agent cross-validating and synthesizing the detection results of two models with different characteristics. As a result, it can be confirmed that the anomaly detection accuracy and robustness of the entire system are significantly improved.

[0288] Table 5 below shows examples of Image-AUROC results for a single anomaly detection model and a multi-agent system, and Table 6 shows examples of F1-Score results for a single anomaly detection model and a multi-agent system.

[0289]

[0290]

[0291] In order to evaluate not only the quantitative performance improvement of the agent but also the quality and reliability of the text-based analysis results ultimately provided to the user, response results generated by a single anomaly detection model integration method and a multi-model-based agent method were compared and analyzed. In particular, to verify the robustness of the model, this evaluation was conducted by capturing images of five classes of anomalies in new background environments, such as wooden desks, cloth, and paper boxes, which were not included in the existing training data.

[0292] Text response results of the Single Detection Model

[0293] When a Large Language Model (LLM) was integrated with a single anomaly detection model, a text response was provided that directly listed the detected anomalies, their locations, and the raw confidence scores calculated by the model. While this approach has the advantage of rapidly delivering detection results, it has the limitation that the large variance in confidence scores makes it difficult for non-expert users to interpret them intuitively. Furthermore, reporting multiple anomalies individually at the same semantic location (e.g., 'middle-center') caused ambiguity in determining whether the anomaly represented a single major defect or multiple minor defects.

[0294] Text response result of the Agentic Detection Model

[0295] In contrast, the multi-model-based agent approach generated a refined and highly reliable comprehensive analysis report by comprehensively analyzing the results of two different anomaly detection models. The agent enhanced the reliability of the results by determining only anomalies commonly detected by both models as 'Confirmed Anomalies' through a cross-validation procedure. Furthermore, it calculated 'Combined Confidence' by synthesizing the reliability scores of the two models and provided qualitative evaluations, such as "a high level of confidence," to enable users to understand the analysis results more clearly. This allowed for the clear delivery of detection results, which were previously presented ambiguously in the single-model approach, by integrating them into a single 'Confirmed Anomaly'.

[0296] The differentiating features of this agent

[0297] The most significant differentiating factor of this agent lies in the role of the large-scale language model. In a single-model integration approach, the LLM remains merely a summarizer that simply summarizes the results output by a single model; whereas in a multi-model-based agent approach, the LLM and the system perform the roles of a validator and synthesizer, verifying facts by comparing and analyzing two independent information sources.

[0298] This cross-validation and comprehensive analysis process represents a fundamental technical difference in that it effectively eliminates potential false positives and transforms raw detection data into highly reliable information (actionable intelligence) that users can utilize for actual decision-making.

[0299] Table 7 below shows examples of anomaly detection text response results for a single anomaly detection model and a multi-agent system.

[0300]

[0301] 3. Edge Computing

[0302] 3.1 Edge Device Inference

[0303] 3.1.1 System Environment

[0304] The anomaly detection model applied to this system is designed for real-time operation under conditions similar to actual industrial sites, and the NVIDIA Jetson Orin Nano (8GB model) was adopted as the edge device to implement this. By simultaneously providing high-level AI inference performance and low-power characteristics within a small form factor, the edge device functions as a hardware platform suitable for industrial environments.

[0305] The AI ​​computing performance of the Jetson Orin Nano is provided by a GPU based on the NVIDIA Ampere architecture. This GPU includes 1,024 CUDA cores for performing parallel operations and 32 3rd generation Tensor cores to accelerate matrix operations, which are critical in deep learning computations. This hardware acceleration architecture is combined with 8GB of LPDDR5 memory with a memory bandwidth of 68GB / s to minimize data transfer bottlenecks and enable anomaly detection models to achieve maximum performance during real-time inference.

[0306] The key advantage of an edge device-based inference environment is that it reduces network latency and fundamentally eliminates security vulnerabilities associated with external transmission by processing input data immediately at the point of generation rather than transmitting it to a remote cloud server. Additionally, the Jetson Orin Nano can operate at low power levels of approximately 7W to 15W, enabling stable operation for extended periods without requiring separate, complex cooling devices. These characteristics serve as significant technical advantages for deploying AI-based anomaly detection systems in environments such as actual manufacturing lines, automated facilities, or mobile equipment where power and installation space constraints are severe.

[0307] 3.1.2 Real-time Inference Acceleration

[0308] To run artificial intelligence models in real time on edge devices, computational optimization of the model itself must be performed in parallel with system-level acceleration of the entire data processing pipeline. In this system, multiple acceleration techniques, including model-level optimization and data processing pipeline acceleration, were applied with NVIDIA TensorRT at the center.

[0309] 1. Model Optimization using TensorRT

[0310] The trained PyTorch-based anomaly detection model was converted into ONNX (Open Neural Network Exchange), a general-purpose neural network model exchange format, and then built into the TensorRT engine to perform model-level optimization. During this process, TensorRT maximizes inference performance by automatically applying the following optimization techniques.

[0311] Operator Fusion

[0312] TensorRT integrates multiple sequential operations, such as convolution, bias addition, and activation function operations, into a single operation. This minimizes memory access to intermediate results and improves GPU computational efficiency.

[0313] Precision Calibration

[0314] By replacing existing 32-bit floating-point (FP32) operations with 16-bit mixed-precision (FP16) operations, computation speed is significantly improved by actively utilizing Tensor Cores. This precision correction also has the effect of reducing memory usage and is particularly suitable for edge computing environments as it provides a substantial improvement in inference speed with only a minute loss of accuracy.

[0315] 2. Accelerate Data Pipelines

[0316] To alleviate data processing bottlenecks that may occur before and after model inference, acceleration techniques were applied to the system-wide data pipeline.

[0317] Asynchronous Data Transfer

[0318] The pipeline was configured so that the GPU simultaneously performs inference operations on the current frame while transmitting the next frame data from the CPU to the GPU. By processing data transmission and operations in parallel in this way, latency caused by data transmission was minimized.

[0319] CUDA Streams

[0320] The parallelism of GPU resources was maximized by configuring independent tasks to be executed in parallel across multiple CUDA streams. For example, while the GPU performs anomaly map calculations for the current frame, the CPU simultaneously performs preprocessing tasks for the next frame in another stream, thereby enabling the overall system's FPS (Frames Per Second) to be maintained stably.

[0321] 3. Dynamic Input Processing

[0322] To flexibly respond to situations where the size of input images or batch sizes vary, we configured the ONNX transformation stage to support Dynamic Input Shapes. This allowed us to secure stable real-time inference performance under diverse input conditions using a single TensorRT engine, without the need to recompile the model whenever the input data size changes.

[0323] 3.2 Model Optimization

[0324] 3.2.1 High-speed Detection Model Optimization

[0325] To optimize the EfficientAD anomaly detection model, composed of a multi-network structure (Teacher-Student-Autoencoder), for edge device environments, this system applied a customized optimization strategy that considers the structural characteristics of the model.

[0326] Specifically, to minimize unnecessary computations during the inference phase, regularization parameters (e.g., mean and standard deviation) included in the Teacher network were pre-calculated and separated. Subsequently, the Teacher network, Student network, and Autoencoder were each converted into independent ONNX (Open Neural Network Exchange) models, and the previously described TensorRT-based optimization process was applied to generate inference engines corresponding to each network.

[0327] The optimized inference pipeline achieved a stable processing speed of over 30 frames per second (FPS) in the NVIDIA Jetson Orin Nano environment. This result empirically demonstrates that the high-speed detection model can be effectively applied even in real industrial environments requiring immediate responsiveness, such as high-speed defect inspection lines in smart factories.

[0328] Differentiating factors

[0329] Unlike conventional model optimization methods that treat the entire network as a single model, this system separates an anomaly detection model composed of multiple networks into individual components, converts them into independent ONNX models, and performs optimization on each component separately. This allowed for the application of optimization strategies tailored to the computational characteristics of each network and minimized potential bottlenecks within the overall system. Consequently, technical excellence was secured in effectively deploying deep learning-based anomaly detection models with complex architectures to edge computing environments.

[0330] 3.2.2 Optimization of High-Precision Detection Models

[0331] INP-Former includes a Transformer-based architecture that excels in detecting minute anomalies; however, due to its relatively high computational and memory usage, customized optimization based on a deep understanding of the model structure is required to run in real-time in resource-constrained edge device environments.

[0332] To this end, the model graph was first simplified by setting the opset_version to 13 during the ONNX transformation step and removing unnecessary operators using the onnxsim library. Subsequently, during the TensorRT-based optimization process, the structure was modified so that instead of building the entire model as a single inference engine, only the feature extraction steps of the encoder and decoder, where computational load is concentrated, are built using the TensorRT engine. At this stage, the TensorRT engine is configured to output intermediate feature vectors.

[0333] The final anomaly map is generated outside the TensorRT engine and is derived through a post-processing pipeline that calculates the cosine similarity between two output feature vectors and applies a Gaussian filter.

[0334] Differentiating factors

[0335] The most significant feature of this optimization method is the adoption of a hybrid execution structure that separates functions based on computational characteristics, rather than treating the entire model as a single optimization target. Specifically, hardware acceleration is applied via the TensorRT engine to computationally intensive parts that are advantageous for parallelization, such as the Transformer encoder and decoder, while flexibility and accuracy are ensured by directly controlling parts that constitute the model's core logic—such as cosine similarity calculations—externally from outside TensorRT, despite their relatively low computational load.

[0336] This strategy of clearly separating and optimizing operations that require acceleration from those that do not presents a practical optimization methodology that enables the efficient execution of complex, modern deep learning architectures even under the constraints of edge computing environments.

[0337] FIG. 6 is a schematic diagram illustrating an example of a model optimization pipeline for real-time anomaly detection in an edge device environment according to an embodiment of the present invention. As shown in FIG. 6, the model optimization pipeline of the present invention is configured by dividing it into an individual optimization step for the EfficientAD model, a common optimization step applied to multiple models, and an individual optimization step for the INP-Former model.

[0338] First, in the EfficientAD optimization domain, preprocessing and separate optimization are performed considering the structural characteristics of the model. Specifically, by pre-calculating the regularization parameters (mean, standard deviation, etc.) included in the Teacher network prior to inference and separating them into fixed values, unnecessary computations during inference are eliminated. Additionally, the Teacher, Student, and Autoencoder networks are separated into independent ONNX models, configured to perform optimizations suitable for the characteristics of each network individually.

[0339] Next, in the common optimization area, an inference acceleration pipeline applicable to both EfficientAD and INP-Former models is executed. In this stage, the trained PyTorch-based models are converted into ONNX (Open Neural Network Exchange), a general-purpose inference format, and the model graph is simplified by removing unnecessary operators using onnxsim. Subsequently, a TensorRT engine specialized for NVIDIA Jetson-series GPU environments is built to maximize inference performance. During this process, computation speed is improved and memory usage is reduced simultaneously by integrating multiple sequential operations into a single operation through Operator Fusion and performing Precision Calibration to convert FP32 operations into FP16 mixed-precision operations. Additionally, support for Dynamic Input Shapes is enabled, allowing for stable inference on input images of various resolutions without recompilation.

[0340] Meanwhile, in the INP-Former optimization area, a hybrid execution structure is applied considering the computational characteristics of the Transformer-based high-precision detection model. Specifically, the PyTorch model is converted to ONNX (opset 13), the graph is simplified using onnxsim, and the TensorRT engine is built only up to the feature extraction parts of the encoder and decoder, where the computational load is concentrated. Subsequently, post-processing steps including cosine similarity calculation, Gaussian filtering, and thresholding using the output intermediate feature vectors are performed outside the TensorRT engine, thereby simultaneously ensuring computational efficiency and flexibility in model interpretation.

[0341] With the model optimization pipeline of Fig. 6, the present invention can stably operate a complex anomaly detection architecture in real time even in an edge computing environment, and achieves the effect of minimizing the use of computational resources while operating a high-speed detection model and a high-precision detection model in parallel.

[0342] 4.3 Results

[0343] As a result of visually evaluating the real-time inference performance in an edge device environment, both anomaly detection models applied in this invention successfully detected anomaly regions, but exhibited different characteristics in terms of detection precision and processing speed.

[0344] Specifically, INP-Former is relatively less affected by ambient noise, such as lighting reflections, and achieves high detection precision by concentrating heatmaps only on areas where actual anomalies exist. As a result, it demonstrated the effect of more clearly identifying the location and shape of the anomaly areas.

[0345] In contrast, EfficientAD demonstrated superior performance in terms of real-time processing of continuous input video and immediate anomaly detection, with shorter inference latency and a higher frames per second (FPS) compared to INP-Former.

[0346] These results suggest that EfficientAD is suitable for the first inspection stage for rapid determination of anomalies, and INP-Former is suitable for the second inspection stage for more precise analysis of the location and shape of anomalies. Therefore, the two models can play complementary roles in terms of speed and precision, and contribute to simultaneously improving the real-time performance and reliability of the multi-model-based anomaly detection system proposed in this invention.

[0347] FIG. 7 is a diagram visually illustrating the results of detecting anomalies in real time using the EfficientAD model in an edge device environment in an embodiment of the present invention. Each image shown in FIG. 7 represents an example in which an anomaly heatmap generated by the EfficientAD model for an input image and a bounding box of an anomaly area derived based on the heatmap are superimposed. The heatmap expresses the distribution of anomaly scores in pixel units as color intensity, and areas with a higher probability of anomaly are highlighted with colors of relatively higher intensity.

[0348] In addition, bounding boxes are displayed to surround candidate anomaly regions extracted based on areas exceeding a threshold set on the heatmap, thereby allowing for intuitive verification of the location of anomalies detected during the real-time inference process. As confirmed in the results of Fig. 7, the EfficientAD model forms a relatively wide and continuous heatmap around the anomaly region even under various object shapes and background conditions, thereby exhibiting characteristics suitable for rapidly identifying the presence of anomalies.

[0349] Due to these characteristics, the EfficientAD model can stably highlight potential anomaly areas while maintaining a high frame rate per second (FPS), which is advantageous for supporting rapid decision-making during the real-time primary anomaly inspection stage based on edge devices. Therefore, Figure 7 exemplarily illustrates that the system proposed in the present invention can effectively perform real-time anomaly detection by utilizing the EfficientAD model even in an edge environment with limited computational resources.

[0350] FIG. 8 is a diagram visually illustrating the results of detecting anomalies in real time using an INP-Former model in an edge device environment in an embodiment of the present invention. Each image shown in FIG. 8 represents an example in which an anomaly heatmap generated by an INP-Former model for an input image and a bounding box of an anomaly area extracted based on the heatmap are superimposed and displayed. The heatmap expresses the distribution of anomaly scores at the pixel level as color intensity and is configured to form a response of high intensity concentrated in areas where actual anomalies exist.

[0351] As confirmed in the results of Figure 8, the INP-Former model is relatively less affected by surrounding environmental factors such as light reflection, changes in background texture, and structural patterns, and forms a heatmap that is locally concentrated only at the location where anomalies occur. Accordingly, the bounding box is also precisely set around the actual anomaly area and exhibits the characteristic of not including unnecessary surrounding areas.

[0352] Due to these characteristics, the INP-Former model is advantageous for identifying the boundaries and locations of abnormal regions with high precision, which is suitable for supporting accurate determination of abnormal types and subsequent actions during the secondary precision inspection stage performed on edge devices. Therefore, Figure 8 exemplarily illustrates that the system proposed in the present invention can effectively perform precise real-time abnormal detection by utilizing the INP-Former model even in an edge environment with limited computational resources.

[0353] 4. Conclusion

[0354] In this invention, a LangGraph-based AI agent framework is proposed and implemented to enable even non-experts to easily utilize complex deep learning-based anomaly detection technology by controlling two mutually complementary anomaly detection models via natural language commands. To this end, the robustness of the model was ensured by constructing a custom dataset that reflects environmental factors, such as lighting changes and background variations, that may occur in actual industrial sites. Furthermore, a LangGraph architecture with a circular structure that shares states was designed so that the Large Language Model (LLM) could interpret user commands and perform multi-stage inference, and consistency between the agent's behavioral procedures and response formats was guaranteed through system prompts.

[0355] Experimental results confirmed that the proposed multi-model-based AI agent demonstrated significant improvements in quantitative detection performance metrics, such as AUROC and F1-Score, compared to the application of a single anomaly detection model. Furthermore, the process of cross-validating the results of the two detection models effectively suppressed potential false positives, and it was demonstrated that the raw data output by the model could be converted into a highly reliable comprehensive analysis report and provided to users.

[0356] In addition, the optimized anomaly detection models were deployed to an actual edge device, the NVIDIA Jetson Orin Nano, to verify their real-time operation capability. By applying customized optimization techniques that consider the architectural characteristics of each model, practical real-time processing performance was secured, with high-speed detection models operating at over 30 frames per second even in edge environments where computational resources are limited. This clearly demonstrates that the framework proposed in this invention possesses a level of practicality that allows for immediate application in actual industrial settings.

[0357] Consequently, the present invention has technical significance in that it provides an integrated anomaly detection solution that encompasses the entire process from training a robust anomaly detection model, designing an intelligent AI agent interface, and real-time deployment based on edge devices.

[0358] To overcome the limitations of relying on a single anomaly detection model, this system adopts a structure that integrates analysis results by linking multiple anomaly detection models with different characteristics. Specifically, by operating a first anomaly detection model with fast inference speed together with a second anomaly detection model that offers high detection precision for minute defects, the trade-off between speed and accuracy is effectively mitigated.

[0359] In such a multi-model structure, the system independently calculates detection results for each model and performs a cross-verification procedure that determines only the anomaly regions commonly detected by both models as reliable anomalies. This suppresses potential false positives that may occur in a single model and enables the selective identification of only anomalies corresponding to actual defects.

[0360] Furthermore, this system does not merely list anomaly detection results but generates a final analysis result by comprehensively considering the location of occurrence, the number of anomalies, and the relative importance of each anomaly. This comprehensive analysis process is achieved by comparing and verifying the results of multiple models, thereby enhancing the reliability and stability of the overall anomaly detection performance.

[0361] Experimental verification results showed that the application of a multi-model agent improved overall image- and pixel-level anomaly detection performance compared to a single-model-based approach. In particular, significant performance improvements were confirmed for defect types that showed low detection performance in the single model. This implies that by comprehensively utilizing models with different detection characteristics, the agent enables more robust anomaly detection that is not biased toward specific types of defects.

[0362] In addition, this system aims for real-time anomaly detection in actual industrial environments and is configured to enable efficient inference in an edge computing environment. By running an anomaly detection model on edge devices, video data can be processed immediately at the point of data generation without being transmitted to an external server, thereby minimizing network latency and enhancing security.

[0363] To this end, the present system converts and optimizes a trained anomaly detection model into a form suitable for edge devices. Specifically, after converting the model into a general-purpose neural network exchange format, optimization techniques such as operator simplification, operator fusion, and precision reduction are applied to improve inference speed and memory efficiency. This optimization process enables the efficient utilization of limited computational resources while maintaining the model's inference accuracy.

[0364] In particular, for models with complex computational structures, a hybrid execution structure can be applied by separating the computationally intensive feature extraction part from the relatively simple post-processing part. Accordingly, operations favorable for parallel processing are handled by a hardware acceleration engine, while relatively simple operations critical to result interpretation are performed in an external post-processing pipeline, thereby simultaneously ensuring the flexibility and efficiency of the entire system.

[0365] Through such optimization and execution structures, this system can provide stable real-time inference performance even in resource-constrained edge device environments. The high-speed detection model performs continuous video analysis at a high frame rate, while the high-precision detection model performs additional detailed analysis when necessary, operating complementarily. This supports the fact that this system can be effectively applied to environments requiring real-time performance, such as actual manufacturing sites.

[0366] FIG. 9 is a diagram illustrating an example of the internal configuration of an anomaly detection system according to an embodiment of the present invention, and FIG. 10 is a flowchart illustrating an example of an anomaly detection method of an anomaly detection system according to an embodiment of the present invention.

[0367] In the present embodiment, an anomaly detection system (900) for detecting anomalies included in an input image in real time may include a teacher feature map extraction unit (910), a student feature map extraction unit (920), a reconstruction feature generation unit (930), a local anomaly map generation unit (940), a global anomaly map generation unit (950), and a final anomaly score map generation unit (960), as illustrated in FIG. 9. Here, the anomaly detection system (900) may be implemented by at least one computer device including at least one processor. In this case, the teacher feature map extraction unit (910), the student feature map extraction unit (920), the reconstruction feature generation unit (930), the local anomaly map generation unit (940), the global anomaly map generation unit (950), and the final anomaly score map generation unit (960) may be functional expressions for the operation of at least one processor.

[0368] Additionally, the anomaly detection method according to the present embodiment may be performed by an anomaly detection system (900) implemented by at least one computer device. At this time, at least one processor included in at least one computer device may be implemented to execute a control instruction according to the code of an operating system included in memory or the code of at least one computer program. Here, at least one processor may operate according to a control instruction provided by the code stored in at least one computer device to control the anomaly detection system (900) implemented by at least one computer device so that the anomaly detection system (900) performs steps (1010 to 1060) included in the method of FIG. 10.

[0369] In step (1010), the anomaly detection system (900) can extract teacher feature maps corresponding to multiple spatial locations through a pre-trained teacher network using an input image as input, via a teacher feature map extraction unit (910). Here, the teacher network is an unsupervised neural network based on a training data set composed of normal images, and can be trained to stably represent normal structural patterns contained in the input image in a high-level feature space. Additionally, the teacher feature map may include feature vectors corresponding to each spatial location or patch unit of the input image, and can be used as reference information for detecting local structural anomalies through comparison with the student feature map in a subsequent step. According to one embodiment, in the inference step, the teacher feature map may be reweighted by attention weights that reflect the importance of each spatial location, and such reweighting may be applied only in the inference step and not during the model training process.

[0370] In step (1020), the anomaly detection system (900) can extract a student feature map through a student feature map extraction unit (920) using an input image as input and a student network that is structurally corresponding to the teacher network. Here, the student network may have a structure that is lighter than the teacher network and may be trained to partially mimic the feature representation of the teacher network for normal images. According to one embodiment, the student network may be trained using a hard feature loss that calculates the loss only for spatial locations where the difference between the teacher feature map and the student feature map is greater than a preset quantile. Through this, the student network is trained to output features similar to the teacher network in normal regions, while selective training may be induced to focus on structurally important regions or regions prone to anomalies. Additionally, the student network may be trained with a loss function that includes a pretraining penalty based on pretraining data so as not to excessively mimic the data distribution used by the teacher network for pretraining.

[0371] In step (1030), the anomaly detection system (900) can generate reconstruction features of an input image using an autoencoder with the input image as input through the reconstruction feature generation unit (930). Here, the autoencoder can be trained based on a training data set consisting of normal images that do not contain anomalies, and can be configured to implicitly learn reconstruction error patterns that repeatedly occur in normal images. The output features of such an autoencoder exhibit a constant reconstruction error pattern when the input image is normal, whereas they may show a meaningful difference from the output features of the student network when logical or global anomalies are included.

[0372] In step (1040), the anomaly detection system (900) can generate a local anomaly map indicating local structural anomalies by calculating the difference between the teacher feature map and the student feature map by spatial location through the local anomaly map generation unit (940). According to one embodiment, the difference between the teacher feature map and the student feature map can be calculated as the distance between feature vectors at each spatial location, and the difference can be aggregated into a single scalar value through mean or sum of squares operations in the channel direction. The local anomaly map thus generated may include a value indicating the possibility of structural anomaly occurrence for each pixel or patch unit of the input image, and can effectively detect structural anomalies such as minute defects or local shape deformation.

[0373] In step (1050), the anomaly detection system (900) can generate a global anomaly map representing global or logical anomalies based on the difference between the reconstruction features of the autoencoder and the output of the student network through the global anomaly map generation unit (950). Here, the global anomaly map can be generated independently of the local anomaly map and can be configured to detect logical anomalies even for spatial locations or regions that are not judged to be anomalies in the local anomaly map. For example, if a logical anomaly occurs, such as when there is no local structural change but the combination of normal patterns is broken, the corresponding region may have a high value in the global anomaly map.

[0374] In step (1060), the anomaly detection system (900) and the final anomaly score map generation unit (960) can generate a final anomaly score map for an input image by normalizing the local anomaly map and the global anomaly map to correct their respective scales and then combining them. According to one embodiment, the local anomaly map and the global anomaly map may be normalized by linear transformation based on quantile values ​​calculated based on a verification data set. Through this, the scale difference between two anomaly maps having different distribution characteristics can be corrected. The final anomaly score map may include anomaly scores for each pixel or patch unit of the input image, and the location, range, or severity of the anomaly area may be determined based on the final anomaly score map.

[0375] As such, according to the anomaly detection method and system of the present embodiment, by combining a pre-trained teacher network, a student network structurally corresponding to the teacher network, and an autoencoder to analyze anomalies contained in an input image from various angles, higher detection accuracy and reliability can be secured compared to a single-model-based anomaly detection method.

[0376] Specifically, subtle structural defects or local anomalies can be precisely detected through a local anomaly map generated based on feature differences between the teacher network and the student network, and logical or global anomalies where combinations of normal patterns are broken but there are no local structural changes can be effectively identified through a global anomaly map generated based on the difference between the autoencoder's reconstruction features and the student network's output. Accordingly, a technical effect is provided that enables the complementary detection of structural and logical anomalies.

[0377] Furthermore, by normalizing local and global anomaly maps to correct for their respective scales and then combining them, anomaly indicators with different distribution characteristics can be reliably integrated. In particular, by applying quantile-based normalization derived from the validation dataset, deviations in anomaly scores caused by changes in input data or environmental factors can be effectively mitigated. This enables the identification of anomalies based on consistent criteria even under various environmental conditions.

[0378] Furthermore, by training the student network with a loss function that includes hard feature loss and a penalty based on pre-training data, it is possible to prevent the indiscriminate mimicry of the teacher network's features and induce selective learning concentration in spatial locations with a high probability of anomalies. Consequently, the separation of features between normal and anomalous regions becomes clearer, and the effect of reducing false positives can be achieved.

[0379] In addition, by configuring the teacher feature map to be reweighted by spatial attention weights only during the inference phase, it is possible to improve sensitivity to regions with a high probability of anomalies during inference while maintaining the stability of the model training process. This offers practical benefits in that it can improve anomaly detection performance without additional training burden.

[0380] Furthermore, since the anomaly detection method and system according to the present embodiment can be configured based on unsupervised learning using only normal data, they can be easily applied even in industrial environments where collecting anomaly data is difficult or labeling costs are high. Accordingly, they have the effect of being universally applicable to various industrial fields, such as manufacturing process inspection, equipment anomaly detection, and surface defect inspection.

[0381] Consequently, the anomaly detection method and system according to the present embodiment enable multi-perspective anomaly detection that simultaneously considers structural and logical anomalies, and by providing highly reliable anomaly detection results through a stable learning structure and normalization-based score combination, it can provide technical effects that are highly useful for real-time anomaly detection and quality control in actual industrial environments.

[0382] FIG. 11 is a diagram illustrating an example of the internal configuration of an LLM-based anomaly detection agent system according to an embodiment of the present invention, and FIG. 12 is a flowchart illustrating an example of a control method for an LLM-based anomaly detection agent according to an embodiment of the present invention.

[0383] In this embodiment, an LLM-based anomaly detection agent system (1100) for analyzing anomalies included in an input image may include an agent node (1110), a state object management unit (1120), one or more tool nodes (1130), a conditional routing control unit (1140), and a termination node (1150), as shown in FIG. 11.

[0384] Here, the anomaly detection agent system (1100) can be implemented by at least one computing device including at least one processor.

[0385] At this time, the agent node (1110), the state object management unit (1120), the tool node (1130), the conditional routing control unit (1140), and the termination node (1150) may each be functional expressions for the operation of at least one processor, and these components may be implemented in a physically separated or logically combined form.

[0386] The agent node (1110) includes a Large Language Model (LLM) and can be configured to interpret natural language commands entered by a user and plan tasks to be performed based on the natural language commands. For example, the agent node (1110) can distinguish and determine requests for primary analysis of an input image, requests for additional detailed analysis, or requests to end analysis.

[0387] The state object management unit (1120) may be configured to manage state objects shared among the agent node (1110), the tool node (1130), and the termination node (1150). The state objects may include message information containing a record of interaction with the user, an input image to be analyzed or frame information extracted from said image, anomaly detection results generated by the tool node (1130), and intermediate analysis results, etc. Accordingly, the work context can be maintained throughout the multi-stage analysis process.

[0388] The tool node (1130) is a component that abstracts an external tool for executing an anomaly detection model, and can be configured to execute the anomaly detection model in response to a structured tool call command transmitted from the agent node (1110), and to convert the output result into a structured data format that the agent node (1110) can interpret and store in a state object. At this time, the tool node (1130) can be provided in the form of a simple function call interface regardless of the internal operation structure of the anomaly detection model.

[0389] The conditional routing control unit (1140) may be configured to selectively branch the workflow based on information contained in a state object according to a graph structure including a conditional edge. For example, if the state object contains a structured command corresponding to a tool call, the workflow may be routed to the tool node (1130), and if not, the workflow may be routed to the termination node (1150).

[0390] The termination node (1150) may be configured to provide the final analysis result generated by the agent node (1110) to the user when it is determined that the analysis result is sufficient.

[0391] Additionally, the LLM-based anomaly detection agent control method according to the present embodiment may be performed by an anomaly detection agent system (1100) implemented by at least one computing device. In this case, at least one processor included in the at least one computing device may be implemented to execute a control instruction according to the code of an operating system included in memory or the code of at least one computer program.

[0392] Hereinafter, an example of an LLM-based anomaly detection agent control method according to the present embodiment will be described step-by-step with reference to FIG. 12.

[0393] In step (1210), the anomaly detection agent system (1100) receives a natural language command input by a user through an agent node (1110) and can interpret the natural language command using a large-scale language model. At this time, the natural language command may include a request to perform anomaly detection, a request for additional analysis, or a request to terminate analysis.

[0394] In step (1220), the anomaly detection agent system (1100) may plan the work to be performed based on the interpreted natural language command and reflect the work plan in the state object. In this step, the sequence of work that can be performed, the tool node call conditions, and the final response format may be considered together based on a predefined system prompt.

[0395] In step (1230), the anomaly detection agent system (1100) can use a state object to share and update information related to the execution of tasks between the agent node (1110), the tool node (1130), and the termination node (1150).

[0396] In step (1240), the anomaly detection agent system (1100) may selectively route the work flow from the agent node (1110) to either the tool node (1130) or the termination node (1150) based on the contents of the state object according to a graph structure including conditional edges.

[0397] In step (1250), if the tool node (1130) is called, an anomaly detection model is executed to generate an anomaly detection result, and the result can be stored in a state object in a structured data format and then transmitted back to the agent node (1110). Based on the transmitted result, the agent node (1110) can determine whether to call additional tool nodes or to terminate the analysis.

[0398] In step (1260), when the anomaly detection agent system (1100) determines that the analysis is finished, it can generate and provide to the user the final analysis results, including confirmed anomalies, the most severe anomalies, and combined reliability, in natural language according to a predefined report format through the agent node (1110).

[0399] As such, according to the LLM-based anomaly detection agent system and control method of the present embodiment, by configuring a graph-based workflow that shares a state object centered on an agent node including a large-scale language model, the entire anomaly detection process can be performed as a dynamic and adaptive analysis procedure rather than a single fixed pipeline.

[0400] Specifically, by having agent nodes directly interpret natural language commands entered by users and plan tasks based on the meaning and analysis context of those commands, even non-expert users can request complex anomaly detection tasks using only intuitive natural language input. Consequently, user accessibility is significantly improved compared to existing anomaly detection systems that relied on standardized input interfaces or predefined scenarios.

[0401] Furthermore, by configuring the system to share message information, intermediate analysis results, and anomaly detection results through state objects among agent nodes, tool nodes, and termination nodes, the operational context is consistently maintained throughout the multi-stage analysis process. Consequently, this effectively resolves the problem of conventional technology, where analysis was interrupted at every stage due to the failure to consider previous analysis results.

[0402] In addition, by selectively routing the workflow according to a graph structure including conditional edges, the agent node can autonomously determine whether to call additional tool nodes or terminate the analysis based on the anomaly detection results. Accordingly, if the analysis is insufficient, tool nodes are repeatedly called to perform a detailed analysis, and if the analysis is sufficient, the process can be terminated immediately without performing unnecessary operations, thereby simultaneously improving analysis efficiency and computational resource utilization.

[0403] Furthermore, by providing tool nodes in the form of abstract interfaces independent of the internal structure of the anomaly detection model, it is possible to flexibly combine and extend different anomaly detection models or external analysis tools within the same agent framework. Consequently, new models or analysis algorithms can be easily added without being dependent on a specific anomaly detection model.

[0404] Furthermore, by confirming anomalies based on the cross-validation results using multiple anomaly detection models and calculating combined reliability by combining the reliability scores of each model, it is possible to effectively reduce false positives or false negatives that may occur in single-model-based anomaly detection.

[0405] Furthermore, since the agent node is configured to generate final analysis results in natural language according to a predefined report format, anomaly detection results are provided in a user-understandable descriptive format, rather than being limited to technical figures or maps. Consequently, the burden of interpreting anomaly detection results is reduced, and the judgment of field operators or decision-makers can be effectively supported.

[0406] Consequently, according to an embodiment of the present invention, the anomaly detection process is extended beyond simple model execution to a structure in which it is planned, controlled, iterated, and terminated by an intelligent agent, thereby providing a highly reliable and highly scalable anomaly detection framework capable of flexibly responding to various environments and requirements.

[0407] FIG. 13 is a diagram illustrating an example of the internal configuration of an edge device-based anomaly detection system according to an embodiment of the present invention, and FIG. 14 is a flowchart illustrating an example of an edge device-based anomaly detection method according to an embodiment of the present invention.

[0408] In the present embodiment, an anomaly detection system (1300) for efficiently detecting anomalies included in input data in an edge device environment may include a model receiving unit (1310), a model conversion unit (1320), a model storage unit (1330), a model selection unit (1340), an anomaly detection execution unit (1350), and a result output unit (1360), as shown in FIG. 13.

[0409] Here, the anomaly detection system (1300) may be implemented by at least one edge device or computing device including at least one processor.

[0410] At this time, the model receiving unit (1310), model conversion unit (1320), model storage unit (1330), model selection unit (1340), anomaly detection execution unit (1350), and result output unit (1360) may be hardware components or functional blocks implemented by instructions executed by at least one processor.

[0411] Additionally, the anomaly detection method according to the present embodiment may be performed by an anomaly detection system (1300) implemented by at least one computer device or edge device. In this case, at least one processor included in at least one computer device may be configured to execute a control instruction according to code of an operating system stored in memory or code of at least one computer program. According to such a control instruction, the anomaly detection system (1300) may be controlled to perform steps (1410 to 1460) illustrated in FIG. 14.

[0412] In step (1410), the anomaly detection system (1300) may receive a pre-trained anomaly detection model as input through the model receiver (1310). At this time, the anomaly detection model may be downloaded from a server or external storage device, or may be a model provided in advance during the manufacturing stage, and may include a convolutional neural network, a transformer-based model, or other deep learning-based anomaly detection model.

[0413] In step (1420), the anomaly detection system (1300) can convert the received anomaly detection model to suit the computational resource constraints of the edge device through the model conversion unit (1320). For example, the computational precision of the model parameters can be converted from floating-point precision to low-precision integer or mixed precision, the computation graph can be simplified by merging multiple computation layers or removing unnecessary computation nodes, or the model in the format of a general-purpose deep learning framework can be converted into an inference engine format that can be accelerated and executed on the edge device. Such conversion may be performed in a single manner or may be performed by combining multiple conversion methods.

[0414] In step (1430), the anomaly detection system (1300) can store one or more converted anomaly detection models in the memory of the edge device through the model storage unit (1330). At this time, the stored anomaly detection models may be configured to have different computational complexity, memory requirements, or detection accuracy, and may include, for example, a lightweight model for high-speed inference and a high-precision model for high-precision analysis.

[0415] In step (1440), the anomaly detection system (1300) can select one of the one or more stored anomaly detection models when input data is received from an edge device through the model selection unit (1340). At this time, the model selection may be performed based on characteristics of the input data, such as the resolution, size, or type of the input data, or based on operational states, such as the power state, processing load, and memory usage of the edge device. Additionally, according to some embodiments, the model selection may be performed based on control commands generated by an agent node including a large-scale language model (LLM).

[0416] In step (1450), the anomaly detection system (1300) can perform anomaly detection on input data using a selected anomaly detection model through the anomaly detection execution unit (1350). At this time, anomaly detection can be performed continuously while the input data is received in real time, and an anomaly detection result including the location of anomaly occurrence or anomaly score within the input data can be generated.

[0417] In step (1460), the anomaly detection system (1300) can output the anomaly detection result to an external device, provide it to a user, or transmit it to a higher-level system through the result output unit (1360). At this time, the anomaly detection result may be provided in the form of an anomaly score map, anomaly occurrence area information, or a summarized anomaly judgment result.

[0418] As such, according to the edge device-based anomaly detection method and system of the present embodiment, a pre-trained anomaly detection model is converted to suit the computational resource constraints of the edge device, a plurality of converted anomaly detection models are stored in the edge device, and then an appropriate anomaly detection model is selected according to the characteristics of the input data or the operating state of the edge device to perform anomaly detection, thereby enabling stable and efficient anomaly detection even in an edge environment with limited computational resources.

[0419] Furthermore, by applying specific model transformation methods, such as converting the computational precision of model parameters, simplifying the computation graph, and converting the inference execution format, computational load and memory usage can be effectively reduced while maintaining the structural characteristics of the anomaly detection model. Consequently, real-time anomaly detection becomes possible on edge devices alone without relying on high-performance server environments, and overall system latency can be significantly reduced.

[0420] In addition, multiple anomaly detection models with different computational complexities or detection accuracy are stored on edge devices, and by selectively operating a lightweight model for high-speed inference and a high-precision model for high-precision analysis according to the device's power status, processing load, memory usage, or characteristics of input data, the balance between anomaly detection performance and system resource efficiency can be flexibly adjusted depending on the situation.

[0421] Furthermore, since continuous anomaly detection can be performed using a selected anomaly detection model even in environments where input data is received in real time, it can be effectively applied to various real-time application fields such as manufacturing process monitoring, equipment status monitoring, video-based security systems, autonomous driving, and smart factories.

[0422] Furthermore, according to some embodiments, the selection and execution of an anomaly detection model can be controlled in conjunction with an agent node including a large-scale language model (LLM), thereby enabling intelligent model operation that goes beyond simple rule-based control and reflects the context of input data, previous anomaly detection results, or user requirements.

[0423] Consequently, the edge device-based anomaly detection method and system according to the present invention can simultaneously improve the real-time nature, resource efficiency, operational flexibility, and scalability of anomaly detection, thereby providing the effect of significantly expanding the scope of application of anomaly detection technology in an edge computing environment.

[0424] FIG. 15 is a block diagram illustrating an example of a computer device according to an embodiment of the present invention. For example, each of at least one computer device implementing an anomaly detection system (900) described above through FIG. 9 and FIG. 10, an LLM-based anomaly detection agent system (1100) described through FIG. 11 and FIG. 12, and an anomaly detection system (1300) described through FIG. 13 and FIG. 14 may correspond to the computer device (1500) of FIG. 15. As shown in FIG. 15, the computer device (1500) may include a memory (1510), a processor (1520), a communication interface (1530), and an input / output interface (1540). Memory (1510) is a computer-readable recording medium and may include a non-perishable mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Here, a non-perishable mass storage device such as a ROM and a disk drive may be included in the computer device (1500) as a separate permanent storage device distinct from memory (1510). Additionally, an operating system and at least one program code may be stored in memory (1510). These software components may be loaded into memory (1510) from a computer-readable recording medium separate from memory (1510). This separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (1510) through a communication interface (1530) rather than a computer-readable recording medium.For example, software components can be loaded into the memory (1510) of a computer device (1500) based on a computer program installed by files received through a network (Network, 1560).

[0425] The processor (1520) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to the processor (1520) via memory (1510) or a communication interface (1530). For example, the processor (1520) may be configured to execute instructions received according to program code stored in a recording device such as memory (1510).

[0426] The communication interface (1530) may provide a function for the computer device (1500) to communicate with other devices through a network (1560). For example, requests, commands, data, files, etc. generated by the processor (1520) of the computer device (1500) according to program code stored in a recording device such as memory (1510) may be transmitted to other devices through the network (1560) under the control of the communication interface (1530). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (1500) through the communication interface (1530) of the computer device (1500) via the network (1560). Signals, commands, data, etc. received through the communication interface (1530) may be transmitted to the processor (1520) or memory (1510), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (1500) may further include.

[0427] The input / output interface (1540) may be a means for interfacing with an input / output device (I / O device, 1550). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (1540) may be a means for interfacing with a device in which the functions for input and output are integrated into one, such as a touchscreen. The input / output device (1550) may be composed of a computer device (1500) and a single device.

[0428] Additionally, in other embodiments, the computer device (1500) may include fewer or more components than those of FIG. 15. However, it is not necessary to clearly illustrate most of the prior art components. For example, the computer device (1500) may be implemented to include at least some of the input / output devices (1550) described above, or may include other components such as a transceiver, a database, etc.

[0429] The system or device described above may be implemented as a hardware component, or a combination of a hardware component and a software component. For example, the device and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0430] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0431] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The medium may continuously store a computer-executable program, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or multiple hardware components; it is not limited to a medium directly connected to a computer system, but may also exist distributed over a network. Examples of computer-readable storage media include ROM, PROM, EPROM, EEPROM, flash memory (e.g., NAND / NOR), SSD, HDD, magnetic tape, optical recording media (CD-ROM, DVD, BD), magneto-optical media, memory cards, USB memory, etc. Such storage media refer to non-transient media and do not include transmission signals themselves or purely volatile memory (e.g., RAM). In addition, other examples of media include recording or storage media managed by app stores that distribute applications, or sites and servers that supply or distribute various other software. Examples of program instructions include not only machine code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0432] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0433] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 In an LLM-based anomaly detection agent system for analyzing anomalies contained in an input image, the anomaly detection agent system comprises: memory; An LLM-based anomaly detection agent system comprising at least one processor configured to execute a command stored in memory, wherein when the command is executed by the at least one processor, the system analyzes a natural language command input by a user through an agent node including a Large Language Model (LLM), plans a task to be performed based on the natural language command, shares and updates information related to the task execution between the agent node, one or more tool nodes, and an end node through a state object, and selectively routes a task flow from the agent node to either the tool node or the end node based on the contents of the state object according to a graph structure including a conditional edge, wherein the state object includes message information including a record of interaction between the user and the anomaly detection agent system, an input image to be analyzed or frame information extracted from said image, and anomaly detection result information generated by the tool node, wherein the agent node and the tool node utilize the state object to maintain the task context throughout the task execution process. Claim 2 An LLM-based anomaly detection agent system according to claim 1, wherein the tool node is configured to call an external tool that executes an anomaly detection model to generate an anomaly detection result for an input image, convert the anomaly detection result into a structured data format, and store it in the state object; the agent node is configured to reanalyze the anomaly detection result stored in the state object to determine whether to call an additional tool or to generate a final analysis result; and the system includes a cyclic structure that transmits the execution result of the tool node back to the agent node through the state object. Claim 3 An LLM-based anomaly detection agent system according to claim 2, wherein the tool node is configured to execute the anomaly detection model in response to a structured tool call command transmitted from the agent node, regardless of the internal computation process of the anomaly detection model, and to convert the output result of the anomaly detection model into a structured data format that the agent node can interpret and provide. Claim 4 An LLM-based anomaly detection agent system according to claim 3, wherein the structured data format is stored in the state object and includes a data structure comprising the number of detected anomalies, location information for each anomaly, and confidence score for each anomaly. Claim 5 An LLM-based anomaly detection agent system according to claim 4, characterized in that the anomaly detection result is stored in the state object after undergoing a post-processing process including fine noise removal, conversion of coordinate information into semantic position information, and standardization of anomaly scores. Claim 6 An LLM-based anomaly detection agent system according to paragraph 2, wherein the agent node is configured to determine whether an additional tool node call is required or whether to generate a final analysis result based on the anomaly detection result stored in the state object. Claim 7 An LLM-based anomaly detection agent system according to paragraph 2, characterized in that the execution result of the tool node is transmitted back to the agent node through the state object, and the agent node determines the workflow again through the conditional edge based on the transmitted execution result. Claim 8 An LLM-based anomaly detection agent system according to claim 7, characterized in that the agent node is configured to perform multiple meeting tool node calls and analysis for a single user input, and to query the user whether to perform additional analysis based on the results of the multiple meeting analysis. Claim 9 An LLM-based anomaly detection agent system according to paragraph 2, wherein the tool node comprises two or more different anomaly detection models, and the agent node is configured to determine a commonly detected anomaly as a confirmed anomaly based on the positional overlap relationship between the anomaly detection results generated from the two or more anomaly detection models. Claim 10 An LLM-based anomaly detection agent system according to claim 9, wherein the agent node is configured to calculate a combined reliability for the confirmed anomaly by combining reliability scores provided from each of the two or more anomaly detection models. Claim 11 delete Claim 12 An LLM-based anomaly detection agent system according to claim 1, wherein the conditional edge is configured to selectively branch the workflow to either the tool node or the termination node based on whether the information included in the state object contains a structured command corresponding to a tool call in a message generated by the agent node. Claim 13 An LLM-based anomaly detection agent system according to claim 1, wherein the agent node is configured to enforce the sequence of executable tasks, tool call conditions, and final response format based on a predefined system prompt. Claim 14 An LLM-based anomaly detection agent system according to claim 13, wherein the agent node is configured to generate a natural language response according to a predefined report format containing information on confirmed anomalies and information on the most severe anomalies when generating a final analysis result. Claim 15 An LLM-based anomaly detection agent system according to claim 1, wherein the agent node is configured to convert the anomaly detection result into a natural language expression understandable to non-expert users and provide it. Claim 16 A method for controlling an LLM-based anomaly detection agent for analyzing anomalies included in an input image, wherein the method is performed by at least one processor included in a computing device and comprises the steps of: analyzing a natural language command input by a user through an agent node including a Large Language Model (LLM) and planning a task to be performed based on the natural language command; and sharing and updating information related to the execution of the task between the agent node, one or more tool nodes, and an end node using a state object. A method for controlling an LLM-based anomaly detection agent, comprising the step of selectively routing a task flow from an agent node to either a tool node or a termination node based on the contents of a state object according to a graph structure including a conditional edge, wherein the state object includes message information including a record of interaction between a user and an anomaly detection agent system, an input image to be analyzed or frame information extracted from said image, and anomaly detection result information generated by the tool node, and wherein the agent node and the tool node utilize the state object to maintain a task context throughout the task execution process. Claim 17 In claim 16, when the above-mentioned workflow is routed to the above-mentioned tool node, the step of executing an anomaly detection model to generate an anomaly detection result and sharing and updating information related to the execution of the above-mentioned workflow includes the step of storing the anomaly detection result in the above-mentioned state object and then transmitting it back to the above-mentioned agent node through the above-mentioned state object, and the step of selectively routing the above-mentioned workflow includes the step of determining whether to call an additional tool node or to terminate the analysis based on the transmitted anomaly detection result, characterized in that it is an LLM-based anomaly detection agent control method. Claim 18 In claim 16, the step of selectively routing the above-mentioned workflow comprises: a step of determining whether a structured command corresponding to a tool call is included in a message generated by an agent node among the information included in the state object; and a step of controlling the workflow to selectively branch to either a tool node or a termination node according to the result of the determination, characterized in that it is an LLM-based anomaly detection agent control method. Claim 19 A method for controlling an LLM-based anomaly detection agent according to claim 16, wherein the step of planning operations to be performed based on the above natural language command includes the step of controlling the sequence of operations that can be performed, tool node call conditions, and the final response format based on a predefined system prompt. Claim 20 A computer program recorded on a non-transient computer-readable recording medium that is combined with a computer device and executes the LLM-based anomaly detection agent control method of any one of claims 16 to 19 on the computer device.

Images