Process abnormal state detection system of artificial intelligence-based data integration structure

Abstract

The present invention relates to a process abnormal state detection system of an artificial intelligence-based data integration structure. The process abnormal state system of an artificial intelligence-based data integration structure comprises: an image data acquisition module for acquiring image data; a sensing data acquisition module for acquiring process sensing data according to process parameters generated during equipment processing; an ensemble data model generation module for generating an artificial intelligence learning ensemble data model from the image data and the process sensing data; and an artificial intelligence-based process detection module for monitoring a process state on the basis of the artificial intelligence learning ensemble data model.

Description

AI-based data integration structure process anomaly detection system

[0001] The present invention relates to a process abnormal state detection system with an artificial intelligence-based data integration structure, and specifically, to a system that detects process abnormal states by integrally learning image data and process sensor data based on an artificial intelligence learning algorithm.

[0002]

[0003] Various types of abnormal conditions occurring at industrial sites can be monitored by vision means, such as cameras. Additionally, the operating status of various equipment deployed at industrial sites can be monitored by vision means or process sensors, such as vibration sensors, temperature sensors, or humidity sensors. Images regarding the operating status of the equipment are acquired by a camera, and the acquired image data can be transmitted to a monitoring computer for analysis. Additionally, time-series sensor data acquired by process sensors can be transmitted to a monitoring computer for analysis. Optionally, image data or process time-series data can be transmitted to an artificial intelligence learning server, and the server can monitor abnormal conditions by analyzing the image data or process time-series data based on a model learned according to an artificial intelligence learning algorithm. Regarding abnormal condition monitoring technology based on artificial intelligence learning algorithms, Patent Registration No. 10-2154854 discloses a switchboard monitoring system utilizing big data and artificial intelligence configured to enable a switchboard operator to recognize abnormal signs, such as failures, fires, or earthquakes, within the switchboard without opening the inside of the switchboard. In addition, Patent Registration No. 10-2531238 discloses an artificial intelligence-based multi-purpose safety monitoring integrated camera system. The disclosed prior art or known technology monitors abnormal conditions by independently analyzing image data acquired by a camera or sense data acquired by a sensor based on an artificial intelligence algorithm. However, when an abnormal condition occurs, image data and sense data may be related to each other, so integrated analysis is necessary for accurate monitoring of abnormal conditions, but the prior art or known technology does not disclose this.

[0004] The present invention aims to solve the problems of the prior art and has the following objectives.

[0005] [Prior Art Literature]

[0006] [Patent Literature]

[0007] (Patent Document 0001) Prior Art 1: Patent Registration No. 10-2154854 (Sejong Electric Industry Co., Ltd., Published Sep. 10, 2020) Switchgear Monitoring System Utilizing Big Data and Artificial Intelligence

[0008] (Patent Document 0002) Prior Art 2: Patent Registration No. 10-2531238 (Infostech Co., Ltd., Published May 11, 2023) AI-based Multi-purpose Safety Surveillance Integrated Camera System

[0009]

[0010] The objective of the present invention is to provide a process abnormal state detection system with an AI-based data integration structure that detects abnormal state of a process by generating an integrated AI-learned ensemble data model based on AI learning of image data and process sense data.

[0011]

[0012] According to a suitable embodiment of the present invention, a process abnormal state detection system of an artificial intelligence-based data integration structure comprises: an image data acquisition module for acquiring image data; a sense data acquisition module for acquiring process sense data according to process parameters generated during the process of equipment; an ensemble data model generation module for generating an artificial intelligence learning ensemble data model from image data and process sense data; and an artificial intelligence-based process detection module for monitoring the process state based on the artificial intelligence learning ensemble data model.

[0013] According to another suitable embodiment of the present invention, the image data acquisition module and the sense data acquisition module are formed integrally.

[0014] According to another suitable embodiment of the present invention, an artificial intelligence learning ensemble data model is created by an image learning model and a sense learning model generated from image learning data and sense learning data.

[0015] According to another suitable embodiment of the present invention, a process abnormal state system of an artificial intelligence-based data integration structure comprises: a camera module that receives image data and time-series sense data for a monitored object; a monitoring computer that receives image data and time-series sense data transmitted from the camera module and monitors an abnormal state; and an AI learning / detection server that receives image data and time-series sense data from the monitoring computer, learns by an artificial intelligence learning algorithm to generate an artificial intelligence learning ensemble data model, and determines whether an abnormal state exists.

[0016] According to another suitable embodiment of the present invention, time series sense data is at least one data selected from the group consisting of vibration data, temperature data, humidity data and noise data.

[0017] The process anomaly detection system of an AI-based data integration structure according to the present invention improves upon the shortcomings of known rule-based image anomaly detection methods and provides a stable anomaly detection environment by developing an AI anomaly detection model that learns from images generated within a manufacturing process. The anomaly detection system according to the present invention can improve the safety of a smart factory environment by enabling proactive responses to abnormal situations occurring in real-time within the manufacturing process through the development of an AI model capable of pattern recognition and anomaly detection for various time-series sense data that occur continuously over time. Furthermore, the process anomaly detection system according to the present invention provides advanced anomaly detection technology in the manufacturing environment by developing an AI-based intelligent inspection system that is flexible in response to real-time changes in the smart factory environment. The anomaly detection system according to the present invention improves upon the limitations of known PC-based inspection systems by integrating a sense-type high-resolution camera with an AI-based intelligent inspection system, thereby satisfying the new technology requirements of the smart factory industry. Additionally, the anomaly inspection system according to the present invention ensures user convenience by developing a Machine Learning Operations (MLOps)-based process anomaly detection platform capable of data management and the creation, management, and distribution of AI models. The abnormal state inspection system according to the present invention can be applied to the monitoring of various processes in industrial sites, and the present invention is not limited thereto. Additionally, in the inspection system according to the present invention, image data includes still image data and motion images, and detected process parameters include temperature, humidity, vibration, noise, or similar state parameters, and are not limited thereto.

[0018]

[0019] FIG. 1 illustrates an example of a process detection structure according to a process abnormal state detection system of an artificial intelligence-based data integration structure according to the present invention.

[0020] FIG. 2 illustrates an example of an abnormal state detection system according to the present invention.

[0021] FIG. 3 illustrates an example of a sensor-coupled camera for an abnormal state detection system according to the present invention.

[0022] FIG. 4 illustrates an embodiment to which an abnormal state detection system according to the present invention is applied.

[0023] FIG. 5 illustrates an example of a structure in which an ensemble data model is generated for an abnormal state detection system according to the present invention.

[0024] FIG. 6 illustrates an example of a process in which a process is detected based on an ensemble data model according to the present invention.

[0025] FIG. 7 illustrates an example of a Machining Learning Operations (MLOps) model learning and detection process applicable to an abnormal state detection system according to the present invention.

[0026]

[0027] The present invention is described in detail below with reference to embodiments shown in the attached drawings, but the embodiments are for a clear understanding of the invention and the invention is not limited thereto. In the description below, components having the same reference numeral in different drawings have similar functions and are not described repeatedly unless necessary for understanding the invention, and known components are described briefly or omitted, but should not be understood as being excluded from the embodiments of the present invention.

[0028] FIG. 1 illustrates an example of a process detection structure according to a process abnormal state detection system of an artificial intelligence-based data integration structure according to the present invention.

[0029] Referring to FIG. 1, the process abnormal state detection system of an artificial intelligence-based data integration structure includes: an image data acquisition module (11) for acquiring image data; a sense data acquisition module (12) for acquiring process sense data according to process parameters generated during the process of the equipment; an ensemble data model generation module (13) for generating an artificial intelligence learning ensemble data model from image data and process sense data; and an artificial intelligence-based process detection module (14) for monitoring the process state based on the artificial intelligence learning ensemble data model.

[0030]

[0031] The operating status of various facilities, equipment, machines, or devices installed at an industrial site can be monitored by a process abnormality status system, and the process is the operating status or operation process of the facilities, equipment, machines, or devices. The process abnormality status system may have the function of monitoring whether an abnormal state has occurred while monitoring the process of such a monitoring target in real time. For process monitoring, video data of the monitoring target may be acquired by a video data acquisition module (11), and the video data acquisition module (11) may be, for example, a camera or an IP camera. Time-series process data may be acquired by a time-series sense data acquisition module (12), and the time-series sense data acquisition module (12) may be, for example, a vibration sensor, a temperature sensor, a humidity sensor, a noise sensor, or various sensors for measuring process parameters. Time-series sense data may be data detected by such sensors for measuring process parameters, and time-series sense data may be data in which a change value over time is measured. Data acquired by the image data acquisition module (11) and the sense data acquisition module (12) can be made into an ensemble data model by the ensemble data model generation module. The ensemble data model can be a data model capable of detecting abnormal process conditions and can be created by an artificial intelligence learning algorithm. Specifically, the ensemble data model generation module (13) can learn image data and time-series sense data according to an artificial intelligence learning algorithm and generate an ensemble data model based on the learning results. The ensemble data model can be a combined form in which image data and time-series sense data have a correlation with each other. For example, if abnormal vibration occurs due to an abnormal process condition, the occurrence of the abnormal condition can be finally determined by referring to changes in the image data.In this way, the ensemble data model combines time-synchronized video data and time-series sense data to enable monitoring of whether an abnormal state occurs in the process. When an ensemble data model having such a function is created, the process can be monitored by the AI ​​basic process detection module (14) based on it. The AI ​​basic process detection module (14) acquires video data and time-series sense data regarding the monitoring target and monitors whether an abnormal state occurs based on the ensemble data model. The AI ​​basic process detection module (14) may be, for example, a personal computer, but is not limited thereto and may be various types of devices capable of processing and analyzing data.

[0032] FIG. 2 illustrates an example of an abnormal state detection system according to the present invention.

[0033] Referring to FIG. 2, a facility, machine, equipment, device, or means for a similar process may be a monitoring target (21), and a camera module (24) may be installed at an appropriate location to acquire image data (22) of the monitoring target (21). Additionally, a sensor for measuring vibration, temperature, humidity, or noise of the monitoring target (21) may be installed, and the sensor may be an IoT sensor. Each sensor may communicate with the camera module (24), and each process parameter may be measured in real time by each sensor. Each sensor may be installed integrally with the camera module (24), and process parameter values ​​measured in a time series by each sensor may be transmitted to the camera module (24). The camera module (24) may acquire images (22) and process parameters measured from each sensor. Specifically, image data (22) can be acquired by the camera module (24), and process parameter data (23) acquired by each sensor can be transmitted to the camera module (24). The camera module (24) can convert the image data (22) and process parameter data (23) into signals that can be transmitted and read by electronic devices while synchronizing the time. In this way, the image data (23) and process parameter data (23) processed by the camera module (24) can be transmitted to a monitoring computer (25), such as a personal computer. The monitoring computer (25) can analyze the image data (22) and process parameter data (23) to determine whether an abnormal state has occurred. In this monitoring process, the monitoring computer can determine whether an abnormal state has occurred by applying the ensemble data model described in relation to the embodiment of FIG. 1. The ensemble data model can be generated by an AI learning / detection server (26).To generate an ensemble data model, the monitoring computer (25) can transmit image data (22) and process parameter data (23) received from the camera module (25) to the AI ​​learning / detection server (26). The AI ​​learning / detection server (26) can store various image data (22) and process parameter data (23) regarding the monitored object and can learn the image data (22) and process parameter data (23) by an artificial intelligence learning algorithm. Through such learning, the AI ​​learning / detection server (26) can generate an ensemble data model for identifying abnormal and normal states. The ensemble data model generated in this way can be transmitted to the monitoring computer (25), and the monitoring computer (25) can apply the ensemble data model to analyze the image data (22) and process parameter data (23) to monitor the process of the monitored object (21). Additionally, the AI ​​learning / detection server (26) can return results based on the learning results to the sensor or camera module (24) of the monitoring target (21). The AI ​​learning / detection server (26) can generate ensemble model data based on various artificial intelligence learning algorithms, and the present invention is not limited by this.

[0034] FIG. 3 illustrates an example of a sensor-coupled camera for an abnormal state detection system according to the present invention.

[0035] Referring to FIG. 3, the monitoring target (21) may be equipment (31) installed at an industrial site, and the camera module (24) may be a sensor camera combined with a sensor and a camera. Image data (32) and time-series process data (33) can be acquired simultaneously by the camera module (24), and, for example, various environmental time-series sense data such as temperature, humidity, or vibration can be acquired along with image data (32). In this way, the sensor camera or camera module (24) can perform the function of simultaneously collecting time-series sense data and process images for various environments and synchronizing them. The camera module (24) can be connected to a monitoring computer (25) via wired or wireless communication, and the camera module (24) can transmit the process images and time-series sense data acquired in real time to the monitoring computer (25). The monitoring computer (25) may include a data processing module (34) that classifies, stores, and manages process images and time-series sensor data transmitted from the camera module (24), and may determine whether there is an abnormal state by analyzing the transmitted process images and time-series sensor data. Additionally, the monitoring computer (25) may display data or analysis results on a screen and transmit process images or time-series sensor data to an AI learning / detection server (26) as described above. The time-series process data (33) may include temperature, humidity, vibration, noise, or various similar environmental parameters, and the present invention is not limited by this.

[0036] FIG. 4 illustrates an embodiment to which an abnormal state detection system according to the present invention is applied.

[0037] Referring to FIG. 4, the process abnormal state detection system of an artificial intelligence-based data integration structure includes: a camera module (24) that collects image data and time-series sense data for a monitored object (21); a monitoring computer (25) that receives image data and time-series sense data transmitted from the camera module (24) and monitors the abnormal state; and an AI learning / detection server (26) that receives image data and time-series sense data from the monitoring computer (25), learns by an artificial intelligence learning algorithm to create an artificial intelligence learning ensemble data model, and determines whether there is an abnormal state.

[0038] The monitoring target (21) may be equipment at an industrial site, for example, a machine or device that performs rotational operation, but is not limited thereto. The camera module (24) may include, for example, an IP camera or a PZT camera, and image data and time-series sense data regarding the monitoring target (21) may be collected by the camera module (24). Image data may be collected directly by the camera module (24), and time-series sense data may be collected in various ways. For example, the camera module (24) may include a sensor for acquiring time-series sense data, and time-series sense data may be acquired by the sensor installed in the camera module (24). Alternatively, a sensor for acquiring time-series sense data may be installed separately from the camera module (24), and the separately installed sensor may transmit time-series sensor data to the camera module (24). The camera module (24) may collect time-series sense data in various ways, and the present invention is not limited by this. The camera module (24) can be connected to communicate with the monitoring computer (25), and the camera module (24) can transmit collected image data and time-series sense data to the monitoring computer (25). The monitoring computer (25) may have a data management function that classifies and stores data transmitted from the camera module (24). Additionally, the monitoring computer (25) may have a function that displays various data or information on a screen. The image data or time-series sense data managed by the monitoring computer (25) may be transmitted to the AI ​​learning / detection server (26), and the AI ​​learning / detection server (26) may learn the image data and time-series sense data transmitted from the monitoring computer (25) using an artificial intelligence learning algorithm.As described above, an ensemble data model can be created in the AI ​​learning / detection server based on the learning results according to the artificial intelligence learning algorithm, and whether an abnormal state has occurred in the monitored object (21) can be confirmed by the ensemble data model. The state analysis of the monitored object (21) by such an ensemble data model can be performed in the AI ​​learning / detection server (26). When the state analysis is performed by the AI ​​learning / detection server (26), the detection result can be returned to the monitoring computer (25) and displayed on the screen, and at the same time, the detection result can be returned to the monitored object (21) so that control of the equipment can be performed. Alternatively, the monitoring computer (25) can perform an analysis of the operating state of the monitored object (21), and for this purpose, the ensemble data model can be transmitted to the monitoring computer (25). The monitoring computer (25) can perform a state analysis of the monitored object (21) based on the ensemble data model. As such, the state analysis of the monitoring target (21) according to the ensemble data model can be performed in various ways, and the present invention is not limited by this.

[0039] In the system according to the present invention, the source of data generation and collection can be unified by a sensor-type camera, and the installation procedure can be simplified, while data referencing and merging are not required in the monitoring system. Additionally, an artificial intelligence image / time-series sense data integrated detection model enables simultaneous detection of abnormalities in the process environment and equipment. In order to detect abnormal states in the system according to the present invention, it is necessary to prepare for applying a pre-generated learning model to the detection process of the process abnormality detection system by performing model learning according to an artificial intelligence algorithm on the image data and time-series sense data generated within the process that were collected in advance. After such preparation is completed, a detection target such as an abnormal state within the process may occur during the operation of the sensor and camera module (24) that acquires data regarding the monitoring target (21), thereby triggering the event. After the image data and time-series sense data are aggregated by the camera module (24), they can be transmitted to a monitoring computer (25), and the camera may be an IP camera or a PTZ camera, but is not limited thereto. Additionally, the sensor may include, but is not limited to, temperature, vibration, and humidity sensors, and may be any various sensor capable of collecting environmental information. Image data and time-series sensor data collected can be transmitted to a monitoring computer (25), and after the data is classified and stored in the monitoring computer (25), the data to be detected can be transmitted to an AI learning / detection server (26). The AI ​​learning / detection server (26) can determine whether the data to be detected is in a normal state or an abnormal state based on an ensemble model. Since the learning and detection process is applied to data with different attributes, such as image data and time-series sensor data, it is necessary to create two models separately. In addition, since the results of the two models must be combined to derive a final result, an ensemble model needs to be prepared.Subsequently, the detection result may be returned to the monitoring computer (25) and the monitoring target (21), and the detection result may be visualized on the monitoring computer (25). Additionally, the detection result may be returned to the monitoring target so that the monitoring target, such as equipment, can be controlled. For example, if the detection result corresponds to an abnormal state, the equipment may be shut down. Various measures may be taken depending on the detection result, and the present invention is not limited by this.

[0040] FIG. 5 illustrates an example of a structure in which an ensemble data model is generated for an abnormal state detection system according to the present invention.

[0041] Referring to Fig. 5, the artificial intelligence learning ensemble data model is created by the image learning model and the sense learning model generated from the image learning data and the sense learning data.

[0042] To generate an artificial intelligence learning ensemble data model, image learning data may be acquired (P51), and the image learning data may be acquired via a camera (P51). An artificial intelligence learning algorithm may be applied to the acquired image learning data to create an image learning model (P52). Along with this, sense learning data may be acquired, and the sense learning data may be environmental parameters such as temperature, humidity, or vibration, and may be acquired from sensors capable of measuring each environmental parameter (P53). An artificial intelligence learning algorithm may be applied to the acquired time-series sense learning data to create a sense learning model (P54). An ensemble data module may be created by training the image learning model and sense learning model created in this way using an artificial intelligence learning algorithm (P55). Once an image / sense ensemble data model is created through this process, detection of a monitored target is performed based on it, and whether it is normal or abnormal can be detected. This process is described in detail below.

[0043] FIG. 6 illustrates an example of a process in which a process is detected based on an ensemble data model according to the present invention.

[0044] Referring to FIG. 6, process image learning data can be acquired (P61a), and the image may include a still image or a video. Along with this, process time series sense learning data can be acquired (P61b), and the process time series sense learning data may include temperature, humidity, vibration, noise, or similar environmental data. The features of the acquired image data and the features of the time series sense data can be analyzed (P62a, P62b), and the features can be analyzed based on the base image or base parameter values. Once such features are analyzed, the image data and time series sense data can be preprocessed (P63a, P63b). Preprocessing refers to the process of processing data into a model with features so that features can be learned by an artificial intelligence learning algorithm, and the preprocessed image detection model and time series sense detection model can be trained by an artificial intelligence learning algorithm to generate an ensemble data model (P65). The ensemble data model can be created by learning the association or correlation between each image detection model and the time series sense detection model.

[0045] Detection targets may occur within processes taking place in various industrial sites (P66), and process image detection data and process time-series sense detection data may be acquired, respectively, by the sensor-type camera described above (P67a, P67b). The acquired image data and process time-series sense data may be preprocessed, respectively (68a, 68b), and through preprocessing, the image data and process time-series sense data may be converted into a data form that allows for the extraction and comparison of various parts. An ensemble data model trained on the image data and process time-series sense data after such preprocessing may be applied (P69), thereby enabling the detection of whether the process state is normal or abnormal. Ensemble learning on image data and process time-series sense data is a form that combines various learning algorithms and is a machine learning method that can obtain superior prediction performance compared to the case where a single learning algorithm is applied. Through such ensemble learning, predictions based on models from each learning algorithm are integrated to enable more accurate and robust predictions. The MLOps model learning and detection process applied to the system according to the present invention is described below.

[0046] FIG. 7 illustrates an example of a Machining Learning Operations (MLOps) model learning and detection process applicable to an abnormal state detection system according to the present invention.

[0047] Referring to FIG. 7, the learning process can be carried out by the monitoring computer (25) and the AI ​​learning / detection server (26), and process detection can be performed. A learning root file can be generated by the monitoring computer (25) and transmitted to the AI ​​learning / detection server (26), and specifically, a Docker Container can be configured and a camera / sensor channel can be specified. Along with this, the type of algorithm for classification / anomaly detection / clustering can be specified. Learning data can be uploaded from the monitoring computer (25) to the AI ​​learning / detection server (26), and internal labeling data of the monitoring computer (25) can be uploaded to the AI ​​learning / detection server (26) after being refined. Basic learning can be configured, and, for example, an already generated root file that serves as the learning target can be uploaded. Additionally, a path for the already uploaded learning target data can be specified, and a detailed algorithm for learning / detection can be specified. Subsequently, learning parameters can be tuned, and hyper-parameters such as the learning rate or weight decay for the specified algorithm can be tuned. Once this process is completed, model training and verification / evaluation are performed on the AI ​​learning / detection server (26), and an AI model file and a configuration file can be generated. Specifically, the model that has been trained / verified can be saved with an extension of h5 or pth, and the configuration file for the hyper-parameters set during the training process can be saved with an extension of json or txt. Once training is completed through this process, a model to be used in real-time can be designated and distributed, and then the detection process can proceed. The detection process can be initiated by receiving data to be detected from an external source.Specifically, data such as image data and time-series process sense data acquired from the back-end of the monitoring computer (25) can be preprocessed and transmitted to the AI ​​learning / detection server (26) in a form detectable by the model. When the preprocessed data is transmitted, the AI ​​learning / detection server (26) can detect the model based on the learning results, and the detection results can be transmitted to the monitoring computer (25). The monitoring computer (25) can visualize the transmitted detection results, such as by displaying them on a screen, and transmit them to the equipment feedback. The MLOps (Machining Learning Operations) model learning and detection process can be applied in various ways for abnormal state detection in the system according to the present invention, and the present invention is not limited by this.

[0048]

[0049] Although the present invention has been described in detail above with reference to the presented embodiments, those skilled in the art may make various modifications and variations without departing from the technical spirit of the invention by referring to the presented embodiments. The present invention is not limited by such modifications and variations, but is limited only by the claims appended below.

[0050]

[0051] [Explanation of the symbol]

[0052] 11: Image data acquisition module 12: Sense data acquisition module

[0053] 13: Ensemble Data Model Creation Module 14: AI Basic Process Detection Module

[0054] 21: Surveillance Target 24: Camera Module

[0055] 25: Monitoring computer 26: AI learning detection sensor

Claims

1. Image data acquisition module for acquiring image data; A sense data acquisition module that acquires process sense data according to process parameters generated during the process of the equipment; An ensemble data model generation module that generates an artificial intelligence learning ensemble data model from image data and process sense data; and A process abnormal state detection system of an AI-based data integration structure including an AI-based process detection module that monitors process status based on an AI learning ensemble data model.

2. In Claim 1, An artificial intelligence-based data integration structure process abnormal state detection system characterized by the image data acquisition module and the sense data acquisition module being formed as a single unit.

3. In Claim 1, An AI-based data integration structure process abnormal state detection system characterized by an AI learning ensemble data model being created by an image learning model and a sense learning model generated from image learning data and sense learning data.

4. A camera module that receives video data and time-series sense data regarding a monitored target; A monitoring computer that receives image data and time-series sense data transmitted from a camera module and monitors abnormal conditions; and A process abnormal state detection system with an AI-based data integration structure, comprising an AI learning / detection server that receives video data and time-series sense data from a monitoring computer, learns through an AI learning algorithm to generate an AI learning ensemble data model, and determines whether an abnormal state exists.

5. In Claim 4, A process abnormal state detection system of an artificial intelligence-based data integration structure characterized in that time series sense data is at least one data selected from a group consisting of vibration data, temperature data, humidity data, and noise data.

Images