Device and method for predicting performance of ai model

Abstract

Disclosed are a device and method for predicting a degradation in the performance of an AI model. The present disclosure relates to a computer-implemented method for monitoring a first AI model distributed to perform a task in a smart factory.

Description

AI model performance prediction device and method

[0001] The present disclosure relates to a device and method for predicting AI model performance degradation.

[0002]

[0003] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0004] Smart factories, utilizing artificial intelligence and big data, are expanding across the manufacturing sector. Faced with rising labor costs and a shrinking workforce, more and more companies are embracing automation.

[0005] A smart factory is a factory that utilizes IoT technology and automation to optimize production processes and improve efficiency. AI models play a crucial role in smart factories.

[0006] AI models can collect and analyze diverse data generated during the manufacturing process. Real-time data is collected from IoT devices such as sensors, cameras, and robots and fed into AI models.

[0007] When AI models are deployed in the field, their performance is theoretically expected to degrade over time. However, in real-world industrial settings, performance degradation is not necessarily affected by time. Furthermore, performance degradation occurs irregularly, making it difficult to predict when AI models will retrain their training data. Therefore, continuous management and inspection are necessary to determine when AI models deployed in industrial settings will retrain their training data, resulting in additional time and financial costs.

[0008] It is time to build pipelines and systems that predict when AI models will retrain their training data and select training data while reducing financial and time costs.

[0009]

[0010] The present disclosure aims to provide users with the option of retraining an artificial intelligence model by early detecting performance degradation of an artificial intelligence model, visualizing the performance of the artificial intelligence model and selected data, and providing a notification when performance degradation is predicted.

[0011] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0012] According to one aspect of the present disclosure, a computer-implemented method for monitoring a first AI model deployed to perform a task in a smart factory is provided, the method comprising: inputting the task result of the first AI model for data acquired from the smart factory into a second AI model trained to detect an object of a predetermined type from input data and predict the reliability of the detection, thereby obtaining an uncertainty performance index for the acquired data; collecting the acquired data, in which the uncertainty performance index is greater than or equal to a preset threshold, as data for retraining the first AI model; inputting the acquired data into a third AI model trained to detect an object of a different type from the second AI model; collecting the acquired data, including an undetected object that the second AI model cannot detect, as data for the retraining; predicting whether the performance of the first AI model has deteriorated based on the uncertainty performance index and the collected data; and generating a notification to the user that retraining is necessary if the performance deterioration is predicted.

[0013] According to another aspect of the present disclosure, there is provided a device comprising: at least one memory; and at least one processor, wherein the at least one processor executes instructions to input the work result of the first AI model for data acquired from the smart factory into a second AI model trained to detect an object of a predetermined type from input data and predict the reliability of the detection, thereby obtaining an uncertainty performance index for the acquired data; collecting the acquired data, in which the uncertainty performance index is greater than or equal to a preset threshold, as data for retraining the first AI model; inputting the acquired data into a third AI model trained to detect an object of a different type from the second AI model; collecting the acquired data, including an undetected object that the second AI model cannot detect, as data for retraining, based on the detection result of the third AI model; and predicting whether the performance of the first AI model deteriorates based on the uncertainty performance index and the collected data; and generating a notification to the user that retraining is necessary if the performance deterioration is predicted.

[0014]

[0015] According to one embodiment of the present disclosure, a function capable of predicting performance degradation of an artificial intelligence model in real time is provided, thereby significantly improving the stability and reliability of the artificial intelligence model.

[0016] According to one embodiment of the present disclosure, by automatically selecting data similar to learning data and continuously improving the learning data set of an artificial intelligence model, the prediction accuracy of the artificial intelligence model is increased and a quick response to performance degradation is enabled.

[0017] According to one embodiment of the present disclosure, the monitoring system visualizes the performance status of an artificial intelligence model to a user in real time, thereby improving management efficiency through interaction between the user and the system.

[0018] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0019] FIG. 1 is a schematic block diagram of an AI model performance prediction device according to one embodiment of the present disclosure.

[0020] FIG. 2 is a block diagram illustrating a first detection unit according to one embodiment of the present disclosure.

[0021] FIG. 3 is a block diagram illustrating a control unit according to one embodiment of the present disclosure.

[0022] FIG. 4 is a block diagram illustrating a second detection unit according to one embodiment of the present disclosure.

[0023] FIG. 5 is a flowchart illustrating a process for predicting AI model performance according to one embodiment of the present disclosure.

[0024] FIG. 6 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0025]

[0026] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0027] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0028] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0029]

[0030] FIG. 1 is a schematic block diagram of an AI model performance prediction device (10) according to one embodiment of the present disclosure.

[0031] An AI model performance prediction device (10) according to one embodiment of the present disclosure may include all or part of a first detection unit (110), a control unit (120), a second detection unit (130), and a monitoring system (140). The components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in an integrated form in an actual physical environment.

[0032] The first detection unit (110) performs a task using an artificial intelligence model on the acquired data, and detects data required for learning and data other than learning using the detection AI model on the acquired data.

[0033] The control unit (120) determines the performance of the AI ​​model based on the collected data, and when a performance degradation is detected, displays the performance degradation on the monitoring system (140) and notifies the user of the need for relearning.

[0034] The second detection unit (130) detects and collects data similar to the data collected from the first detection unit (110).

[0035] The monitoring system (140) can predict whether the performance of the first AI model (112) will deteriorate based on uncertainty-based performance evaluation indicators and data collected by the data collection unit (118). The monitoring system (140) visualizes the uncertainty-based performance evaluation indicators in real time, allowing the user to check them. Furthermore, if the monitoring system (140) predicts that the AI ​​model's performance will deteriorate, it generates a performance degradation notification to the user.

[0036] FIG. 2 is a block diagram for explaining a first detection unit (110) according to one embodiment of the present disclosure.

[0037] The first detection unit (110) may include all or part of the first AI model (112), the second AI model (114), the third model (116), and the data collection unit (118).

[0038] Smart factories can generate real-time data through the installation of various IoT devices. This data can be related to production line performance, product quality, energy usage, and other factors, as detected by sensors. AI models can collect and analyze relevant data.

[0039] The first AI model (112) is an artificial intelligence model, and may be any one of an artificial neural network (ANN), a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer, a generative adversarial network (GAN), and a reinforcement learning model.

[0040] Artificial neural networks (ANNs) are models inspired by the workings of the biological brain. They process information and learn patterns by mimicking the behavior of neurons, the brain's nerve cells. Convolutional neural networks (CNNs) are a type of ANN specialized in image processing and pattern recognition. They are effective for tasks such as image recognition, object detection, and segmentation, and are designed to understand and process the spatial structure of images. Recurrent neural networks (RENNs) are a type of ANN specialized in processing sequential data. Sequential data refers to a series of data arranged in order. Transformers are one of the deep learning model architectures used in natural language processing. These models are used in various natural language understanding tasks such as machine translation, sentence generation, summarization, and question answering. Generative adversarial networks (GANs) are used to generate new data that resembles reality. These networks consist of a generator and a discriminator. In GANs, the generator and discriminator compete against each other to learn. The generator generates data that appears real to fool the discriminator. The discriminator distinguishes between fake data generated by the generator and real data. A reinforcement learning model is a machine learning model in which an agent learns optimal actions to maximize rewards while interacting with the environment.

[0041] The first AI model (112) performs tasks using data acquired from the smart factory. Here, performing tasks using the acquired data using the AI ​​model means performing prediction, classification, or inference tasks using the data acquired from the smart factory. For example, the first AI model (112) may monitor the operating status of machines within the smart factory and perform tasks to predict the possibility of failure, or the AI ​​model may analyze images, sounds, data logs, etc. and classify them into specific categories. Furthermore, the AI ​​model may use the acquired data to draw conclusions or make judgments about the future.

[0042] The second AI model (114) is a detection AI model that detects data using an uncertainty-based performance evaluation index. Specifically, the second AI model (114) is an artificial intelligence model trained to detect predefined types of objects in input data and predict the reliability of the detection. The second AI model (114) inputs the results of the first AI model (112) on data acquired from a smart factory, thereby obtaining an uncertainty performance index for the acquired data.

[0043] Uncertainty refers to the degree of confidence in the predicted results of an AI model in use. When AI models make predictions, it's often impossible to be completely certain about all scenarios, so considering uncertainty is crucial for assessing the accuracy and reliability of predictions. Uncertainty can be divided into two types: epistemic uncertainty and aleatoric uncertainty.

[0044] Epistemic uncertainty refers to uncertainty arising from the incompleteness of the model itself or unknown information. Epistemic uncertainty can arise from a limited understanding of the model's training data or from insufficient data. Specifically, epistemic uncertainty arises from factors such as data insufficiency, model complexity, and model uncertainty. For example, if the data used for model training is insufficient or unrepresentative, the model may not sufficiently learn the diversity of real-world data. This increases the uncertainty of predictions in new data or in different environments. Furthermore, if the model is too simple or complex, it can underfit or overfit to the training data. An underfitted model can increase the uncertainty of predictions for new data. Uncertainty about the model's parameters or structure also constitutes epistemic uncertainty.

[0045] Alletoric uncertainty refers to uncertainty arising from the inherent uncertainty of the data itself. Alletoric uncertainty refers to situations where a model cannot predict with complete confidence due to the inherent uncertainty of the data. Specifically, alletoric uncertainty arises from factors such as data variability, measurement error, and data limitations. For example, natural variability in data is a major source of alletoric uncertainty. Natural phenomena may not have consistent patterns or regularities and can be influenced by random factors. For example, weather data and the timing of earthquake occurrences are affected by natural variability. Furthermore, measurement errors occurring during data collection increase alletoric uncertainty. Furthermore, alletoric uncertainty can arise depending on the characteristics of the data.

[0046] Therefore, to reduce uncertainty, additional data collection or further refinement of the model is necessary. Furthermore, to reduce uncertainty, additional data collection or further refinement of the model is necessary.

[0047] Detection AI models are used to detect and identify specific objects, patterns, or anomalies. Detection AI models are utilized in a variety of fields, including computer vision, speech recognition, and natural language processing. Detection AI models can process data and provide results in real time. For example, they are useful when applied to real-time detection, monitoring, or interaction systems. Detection AI models can be used in both supervised and unsupervised learning. Various algorithms and architectures can be used in detection AI models. For example, you can design and implement detection AI models by selecting algorithms such as Convolutional Neural Networks (CNN), You Only Look Once (YOLO), Region-based Convolutional Neural Networks (RCNN), Single Shot Multibox Detector (SSD), Faster R-CNN, Mask R-CNN, and Histogram of Oriented Gradients (HOG).

[0048] Therefore, the selected algorithm affects the model's accuracy, speed, memory usage, etc., so an appropriate algorithm must be selected considering the given requirements and environment.

[0049] The second AI model (114) evaluates the reliability of the results predicted, classified, or inferred by the first AI model (112). The second AI model (114) inputs the work results of the first AI model (112) for data acquired from a smart factory, and acquires an uncertainty performance index for the acquired data. The second AI model (114) detects data with high uncertainty using the acquired uncertainty performance index. Here, data with high uncertainty refers to data for which the prediction accuracy or reliability is expected to be low. In other words, the data detected by the second AI model (114) corresponds to the training data required for the first AI model (112). If the uncertainty performance index of the acquired data is higher than a preset threshold, the second AI model (114) detects data for retraining the first AI model (112). Therefore, the data and uncertainty performance index detected by the second AI model (114) are transmitted to the data collection unit (118).

[0050] The third AI model (116) corresponds to an AI model that detects data outside of learning. The second AI model (114) detects data based on learned patterns and cannot detect data for unlearned patterns. In other words, the third AI model (116) is an artificial intelligence model trained to detect objects of a different type than the second AI model (114).

[0051] The third AI model (116) inputs data acquired from the smart factory. The third AI model (116) detects data that the second AI model (114) cannot detect. For example, the data detected by the third AI model (116) has different characteristics from the data learned by the second AI model (114). The data detected by the third AI model (116) may correspond to the training data required for the first AI model (112). The data detected by the third AI model (116) is transmitted to the data collection unit (118).

[0052] The data collection unit (118) collects data by receiving data from the second AI model (114) and the third AI model (116). The data collection unit (118) can collect data acquired from a smart factory whose uncertainty performance index is higher than a preset threshold as data for retraining the first AI model (112). Here, the preset threshold can be set by the user. The data collection unit (118) can collect data acquired from a smart factory that includes an undetected object that the second AI model (114) cannot detect as data for retraining the first AI model (112) based on the detection result of the third AI model (116). The data collection unit (118) may include a task of generating or collecting new data as needed. In addition, the data collection unit (118) manages and verifies the quality of the data. The data collection unit (118) can structure the data using a database or a data warehouse, etc.

[0053] FIG. 3 is a block diagram illustrating a control unit (120) according to one embodiment of the present disclosure. To explain FIG. 3, FIGS. 1 and 2 may be referred to together.

[0054] The control unit may include all or part of the performance prediction unit (122) and the communication unit (124).

[0055] The performance prediction unit (122) can predict whether the performance of the first AI model (112) will deteriorate based on the uncertainty-based performance evaluation index and the data collected by the data collection unit (118). The performance prediction unit (122) can quantify the uncertainty-based performance index and use it as an index for predicting performance deterioration. That is, the performance prediction unit (122) records data that records uncertainty exceeding a set value, and if uncertainty data exceeding a preset threshold is generated, it can determine that there is a possibility of performance deterioration. If a deterioration in the performance of the first AI model (112) is predicted, the performance prediction unit (122) notifies the user of the need for retraining through the monitoring system (140).

[0056] The uncertainty-based performance evaluation indicator can be at least one of an uncertainty graph, an uncertainty histogram, an uncertainty distribution analysis, and an uncertainty performance curve.

[0057] Uncertainty graphs visualize model performance by graphically representing the model's prediction accuracy and the uncertainty of that prediction. Uncertainty histograms evaluate model performance by visualizing the accuracy of predicted results according to uncertainty. Uncertainty distribution analysis analyzes the model's prediction accuracy and uncertainty distribution to evaluate the model's overall performance. Uncertainty performance curves are graphs that visually represent the relationship between model uncertainty and performance. Using uncertainty performance curves, you can identify areas where performance drops sharply when uncertainty exceeds a certain level. By identifying these areas in advance, you can predict performance degradation or assess model reliability.

[0058] The communication unit (124) can use wireless communication and transmit and receive status information and control signals. The communication unit (124) can transmit and receive data between external systems (e.g., a monitoring system, a user interface).

[0059] FIG. 4 is a block diagram illustrating a second detection unit (130) according to one embodiment of the present disclosure. To explain FIG. 4, FIG. 2 may also be referred to.

[0060] The second detection unit (130) may include all or part of the fourth AI model (132), data selection unit (134), labeling unit (136), and relearning unit (138).

[0061] The fourth AI model (132) is a detection AI that selects data having a feature vector similar to the learning data.

[0062] Among the data selected by the data selection unit (134), there may be data that is detected regardless of the learning data and is not related to improving or maintaining the performance of the AI ​​model.

[0063] Accordingly, the fourth AI model (132) selects data having a feature vector similar to the learning data used for learning the first AI model (112) from among the data collected by the data collection unit (118). The data selected by the fourth AI model (132) is transmitted to the data selection unit (134). To this end, the fourth AI model (132) may include a feature extractor that extracts a feature vector from the given input data. The fourth AI model (132) may select data from which a feature vector similar to the feature vector extracted from the learning data is extracted from among the data that requires relearning detected by the first detection unit (110), as data that is helpful for learning.

[0064] The data selection unit (134) may include tasks for generating or collecting new data as needed. In addition, the data selection unit (134) may manage and verify the quality of data.

[0065] The labeling unit (136) labels the data selected by the fourth model (132) so that the first AI model (112) can use the selected data for re-learning.

[0066] Additionally, the labeling unit (136) may request the user to label the selected data before the first AI model (112) retrains the selected data.

[0067] Labeling can be performed in a variety of ways, either manually by humans or using automated tools. Manual labeling involves a human reviewing the given data and assigning accurate labels. For example, in image classification tasks, a labeler manually determines the category of each image and assigns a label. Batch labeling tools provide an interface that displays a dataset in bulk and allows users to enter or select labels for each piece of data. Automated labeling technologies utilize computer vision or natural language processing to automatically analyze and assign labels to data.

[0068] When performance degradation is predicted, the retraining unit (138) retrains the first AI model (112) using data selected by the fourth AI model (132). That is, the retraining unit (138) trains the first AI model (112) using labeled data and existing training data.

[0069] Meanwhile, the process by which an artificial intelligence model learns using learning data can be divided into the stages of data preparation, model building, learning, and evaluation.

[0070] First, data for the AI ​​model to learn from is collected. The collected data may be relevant to the problem the AI ​​model will solve. The collected data must be preprocessed to make it easier for the AI ​​model to understand. Preprocessing may include data cleansing, noise removal, image resizing, normalization, and vectorization.

[0071] Next, an appropriate model is selected based on the problem to be learned by the AI ​​model, and the selected model's structure is designed. This may include the composition of the input layer, hidden layer, and output layer, the number of neurons in each layer, and the activation function. Furthermore, before the AI ​​model learns with training data, its weights and biases are initialized. Initializing the weights and bias values ​​means setting the initial state when the model begins learning.

[0072] Next, a loss function is selected, representing the goal the AI ​​model seeks to optimize. The loss function measures the difference between the AI ​​model's output and the actual value.

[0073] Next, the preprocessed data is used to train an AI model. A portion of the training data is used as input, and the model's output is compared with the correct answer. The model's weights are adjusted to calculate and minimize the error. The backpropagation algorithm is used to minimize the error.

[0074] Next, the performance of the AI ​​model is evaluated using validation data.

[0075] Finally, the model after learning is finally evaluated using test data.

[0076] FIG. 5 is a flowchart illustrating a process for predicting AI model performance according to one embodiment of the present disclosure.

[0077] The first AI model (112) and the third AI model (116) input data acquired from the smart factory (S502).

[0078] The first AI model (112) performs tasks using data acquired from the smart factory. Here, performing tasks using the acquired data using the AI ​​model means performing prediction, classification, or inference tasks using the data acquired from the smart factory. For example, the AI ​​model may monitor the operating status of machines within the smart factory and predict the likelihood of failure, or the AI ​​model may analyze images, sounds, data logs, etc. and classify them into specific categories. Furthermore, the AI ​​model may use the acquired data to draw conclusions or make judgments about the future.

[0079] The second AI model (114) is a detection AI model that detects data using a performance evaluation index based on uncertainty. That is, the second AI model (114) is an artificial intelligence model trained to detect a predetermined type of object from input data and predict the reliability of the detection. The second AI model (114) inputs the work results of the first AI model (112) for data acquired from a smart factory and acquires an uncertainty performance index for the acquired data. The second AI model (114) detects data with high uncertainty using the acquired uncertainty performance index (S504). Here, data with high uncertainty refers to data that is expected to have low prediction accuracy or reliability. That is, the data detected by the second AI model (114) corresponds to the training data required for the first AI model (112). If the uncertainty performance index of the acquired data is higher than a preset threshold, the second AI model (114) detects data for retraining the first AI model (112). Accordingly, the data and uncertainty performance indicators detected from the second AI model (114) are transmitted to the data collection unit (118).

[0080] The third AI model (116) is an artificial intelligence model trained to detect different types of objects than the second AI model (114). The third AI model (116) inputs data acquired from a smart factory. The third AI model (116) detects data that the second AI model (114) cannot detect. For example, the data detected by the third AI model (116) detects data with different characteristics from the data learned by the second AI model (114) (S506). The data detected by the third AI model (116) may correspond to the training data required for the first AI model (112). The data detected by the third AI model (116) is transmitted to the data collection unit (118).

[0081] The data collection unit (118) receives data from the second AI model (114) and the third AI model (116) and collects data (S508). The data collection unit (118) can collect data acquired from a smart factory whose uncertainty performance index is higher than a preset threshold as data for retraining the first AI model (112). Here, the preset threshold can be set by the user. The data collection unit (118) can collect data acquired from a smart factory that includes an undetected object that the second AI model (114) cannot detect as data for retraining the first AI model (112) based on the detection result of the third AI model (116).

[0082] The performance prediction unit (122) can predict whether the performance of the first AI model (112) will deteriorate based on the uncertainty-based performance evaluation index and the data collected by the data collection unit (118) (S510). The performance prediction unit (122) can quantify the uncertainty-based performance index and use it as an index for predicting performance deterioration. That is, the performance prediction unit (122) counts the number of data that record uncertainty exceeding a set value, and if uncertainty data exceeding a preset threshold is generated, it can determine that there is a possibility of performance deterioration. If a performance deterioration of the first AI model (112) is predicted, the performance prediction unit (122) notifies the user of the need for retraining through the monitoring system (140).

[0083] The fourth AI model (132) is a detection AI that selects data with feature vectors similar to the training data. The fourth AI model (132) selects data with feature vectors similar to the training data used in the training of the first AI model (112) from among the data collected by the data collection unit (118) (S512). The data selected by the fourth AI model (132) is transmitted to the data selection unit (134).

[0084] The labeling unit (136) labels the data selected by the fourth model (132) so that the first AI model (112) can utilize the selected data for retraining. In addition, the labeling unit (136) may request the user to label the selected data before the first AI model (112) retrains the selected data.

[0085] When a performance degradation of the first AI model (112) is predicted, the retraining unit (138) retrains the first AI model (112) using the data selected by the fourth AI model (132) (S514).

[0086] That is, the retraining unit (138) trains the first AI model (112) using labeled data and existing training data.

[0087] FIG. 6 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0088] The computing device (60) may include some or all of a memory (600), a processor (620), storage (640), an input / output interface (660), and a communication interface (680). The computing device (60) may be a stationary computing device such as a desktop computer, a server, etc., as well as a mobile computing device such as a laptop computer, a smart phone, etc. The computing device (60) may include any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (60) may include a graphic processing unit (GPU), a tensor processing unit (TPU), or a neural processing unit (NPU).

[0089] The memory (600) may store a program that causes the processor (620) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (620), and the above-described method or operation may be performed by executing the plurality of instructions by the processor (620). The memory (600) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in a plurality of memories. When the memory (600) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (600) may include at least one of a volatile memory and a non-volatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the non-volatile memory includes a flash memory.

[0090] The processor (620) may include at least one core capable of executing at least one instruction. The processor (620) may execute instructions stored in the memory (600). The processor (620) may be a single processor or multiple processors.

[0091] Storage (640) maintains stored data even when power supplied to the computing device (60) is cut off. For example, storage (640) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk. A program stored in storage (640) may be loaded into memory (600) before being executed by processor (620). Storage (640) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (600). Storage (640) may store data to be processed by processor (620) and / or data processed by processor (620).

[0092] The input / output interface (660) may provide an interface with an input device such as a keyboard, mouse, etc. and / or an output device such as a display device, printer, etc. A user may trigger the execution of a program by the processor (620) through the input device and / or check the processing result of the processor (620) through the output device.

[0093] The communication interface (680) may provide access to an external network. The computing device (60) may communicate with other devices via the communication interface (680).

[0094] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0095] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0096] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0097] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0098] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0099]

[0100] CROSS-REFERENCE TO RELATED APPLICATION

[0101] This patent application claims priority to Korean Patent Application No. 10-2023-0158992, filed in Korea on November 16, 2023, and Korean Patent Application No. 10-2024-0140266, filed in Korea on October 15, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A computer implementation method for monitoring a first AI model deployed to perform work in a smart factory, A process of obtaining an uncertainty performance index for the acquired data by inputting the work result of the first AI model for the data acquired from the smart factory into a second AI model trained to detect a predetermined type of object from input data and predict the reliability of the detection; A process of collecting the acquired data, in which the above uncertainty performance indicator is greater than a preset threshold, as data for re-learning the first AI model; A process of inputting the acquired data into a third AI model trained to detect a different type of object from the second AI model; A process of collecting the acquired data including undetected objects that the second AI model cannot detect, as data for the re-learning, based on the detection results of the third AI model; A process of predicting whether the performance of the first AI model deteriorates based on the above uncertainty performance indicators and collected data; and A method comprising the step of generating a notification to a user that retraining is necessary if the above performance degradation is predicted.

2. In paragraph 1, A process of selecting data having a feature vector similar to the learning data used for learning the first AI model among the collected data; and A method further comprising a process of retraining the first AI model using selected data.

3. In paragraph 2, Before the above relearning process, A method further comprising the step of requesting a labeling of selected data from said user.

4. In paragraph 1, The above first AI model is, A method, comprising any one of an artificial neural network (ANN), a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer, a generative adversarial network (GAN), and a reinforcement learning model.

5. In paragraph 1, The above first AI model performs the task, A method comprising: monitoring the operating status of a machine within the smart factory, performing a task of predicting the possibility of a failure, or including a process in which an artificial intelligence model analyzes images, sounds, data logs, etc. and classifies them into a specific category.

6. In paragraph 1, The above uncertainty-based performance evaluation indicators are: A method, at least one of Uncertainty Graph, Uncertainty Histogram, Uncertainty Distribution Analysis and Uncertainty Performance Curve.

7. At least one memory; and Containing at least one processor, At least one of said processors executes instructions, An operation of inputting the work result of the first AI model for data acquired from the smart factory into a second AI model trained to detect a predetermined type of object from input data and predict the reliability of the detection, thereby obtaining an uncertainty performance index for the acquired data. The acquired data, in which the above uncertainty performance indicator is greater than or equal to a preset threshold, is collected as data for re-learning the first AI model, The acquired data is input into a third AI model trained to detect a different type of object than the second AI model, Based on the detection results of the third AI model, the acquired data including undetected objects that the second AI model cannot detect is collected as data for the re-learning. Based on the above uncertainty performance indicators and collected data, it is predicted whether the performance of the first AI model deteriorates, A device that performs an action to generate a notification to the user that relearning is necessary when the above performance degradation is predicted.

8. In paragraph 7, The above processor, by executing the above instructions, Among the collected data, data having a feature vector similar to the learning data used for learning the first AI model are selected, A device that further performs the operation of relearning the first AI model using the selected data.

9. In paragraph 8, The above processor, by executing the above instructions, Before the above relearning action, A device further performing an action of requesting labeling of selected data from said user.

10. In paragraph 7, The above first AI model is, A device, any one of an Artificial Neural Network (ANN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Transformer, a Generative Adversarial Network (GAN), and a Reinforcement Learning Model.

11. In paragraph 7, The above first AI model performs the task, A device that monitors the operating status of machines within the smart factory, performs tasks to predict the possibility of failure, or includes a process in which an artificial intelligence model analyzes images, sounds, data logs, etc. and classifies them into specific categories.

12. In paragraph 1, The above uncertainty-based performance evaluation indicators are: A device having at least one of an uncertainty graph, an uncertainty histogram, an uncertainty distribution analysis, and an uncertainty performance curve.

Images