Method and apparatus for collecting data through HMI screen recognition and image analysis by using camera

Abstract

A method for collecting data through human-machine interface (HMI) screen recognition and image analysis by using a camera is disclosed. The method for collecting data through HMI screen recognition and image analysis by using a camera, according to one embodiment of the present invention, may comprise the steps of: receiving, from an image camera, a first image obtained by capturing an HMI screen; generating an image analysis result by analyzing the first image on the basis of an artificial intelligence learning model that analyzes an image and generates an analysis result; and, on the basis of the image analysis result, storing collected data corresponding to the first image.

Description

Method and device for collecting data through HMI screen recognition and image analysis using a camera

[0001] The present invention relates to a method and device for collecting data through HMI screen recognition and image analysis using a camera.

[0002] The content described below merely provides background information related to one embodiment of the present invention and does not constitute prior art.

[0003] In factories and power plants, SCADA (Supervisory Control and Data Acquisition) systems communicate with PLCs (Programmable Logic Controllers) and input / output sensors to obtain information about device operation. This information is then displayed on a Human-Machine Interface (HMI). The HMI presents this information in graphs, charts, or other easily readable and understandable visual formats.

[0004] However, since the S / W installation and system modification for data collection of these SCADA or PLC systems are very limited, it is very difficult to collect data for the integrated management system.

[0005] Therefore, a method is required to collect data by recognizing and analyzing charts and numbers from the HMI screen using image recognition, rather than the conventional method of directly modifying data from the system.

[0006] The purpose of the present invention is to collect data through HMI screen recognition and image analysis using a camera.

[0007] In addition, the present invention aims to train an artificial intelligence learning model using collected data.

[0008] A method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention for achieving the above-described purpose may include the steps of receiving a first image captured by a video camera of an HMI (Human-Machine Interface) screen, generating an image analysis result by analyzing the first image based on an artificial intelligence learning model that analyzes the image and generates an analysis result, and storing collected data corresponding to the first image based on the image analysis result.

[0009] After the step of storing the collected data, a step of training the artificial intelligence learning model using the collected data is included, and the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result, and a second artificial intelligence learning model that generates an image analysis result for the first image based on the judgment result and the information.

[0010] The step of training the artificial intelligence learning model may include a step of providing a first interface for receiving input of first data modification information for the judgment result from a user on the user terminal, a step of generating an image analysis result for the first image based on the first data modification information for the judgment result received through the first interface, a step of providing a second interface for receiving input of second data modification information for the image analysis result from a user on the user terminal, a step of performing retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the first data modification information for the judgment result received through the first interface and the second data modification information for the image analysis result received through the second interface, and a step of updating the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0011] After the step of providing the second interface, the step of displaying the reflection ratio for each item starting from the items that had the greatest influence on the image analysis result and displaying items that can be modified among the items, the step of displaying samples of the image analysis result with the reflection ratio for the first item changed when the first item among the items is selected by the user, and the step of receiving the first image analysis result selected by the user among the samples, and the step of performing re-learning may include the step of performing re-learning of the second artificial intelligence learning model based on the first image analysis result.

[0012] After the step of storing the above collected data, a step of providing the collected data to a user through an integrated dashboard may be included.

[0013] In addition, a device for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention may include an image receiving unit for receiving an image from a video camera, an image analysis unit for analyzing the image based on an artificial intelligence learning model to generate an image analysis result, and a data storage unit for storing collected data corresponding to the image.

[0014] The above image receiving unit receives a first image captured from the HMI (Human-Machine Interface) screen from the image camera, the image analysis unit analyzes the first image based on the artificial intelligence learning model that analyzes the image to generate an analysis result, and generates an image analysis result, and the data storage unit can store collected data corresponding to the first image based on the image analysis result.

[0015] The method may further include an artificial intelligence learning unit that trains the artificial intelligence learning model, wherein the artificial intelligence learning unit trains the artificial intelligence learning model using the collected data, and the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result, and a second artificial intelligence learning model that generates an image analysis result for the first image based on the judgment result and the information.

[0016] The artificial intelligence learning unit provides a first interface for receiving input of data correction information for the judgment result from a user on the user terminal, and generates an image analysis result for the first image based on the data correction information for the judgment result received through the first interface, and provides a second interface for receiving input of data correction information for the image analysis result from a user on the user terminal, and performs retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the data correction information for the judgment result received through the first interface and the data correction information for the image analysis result received through the second interface, and updates the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0017] The artificial intelligence learning unit may display a reflection ratio for each item starting from the items that had the greatest impact on the image analysis result through the second interface, display items that can be modified among the items, and, when a first item among the items is selected by the user, display samples of the image analysis result with a changed reflection ratio for the first item, receive a first image analysis result selected by the user among the samples, and perform retraining of the second artificial intelligence learning model based on the first image analysis result.

[0018] According to the present invention, data can be collected through HMI screen recognition and image analysis using a camera.

[0019] Additionally, according to the present invention, an artificial intelligence learning model can be trained using the collected data.

[0020] FIG. 1 is a block diagram showing the entities for data collection through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0021] FIG. 2 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0022] FIG. 3 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0023] FIG. 4 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0024] FIG. 5 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0025] FIG. 6 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0026] FIG. 7 is a diagram showing the configuration of a device that collects data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0027] FIG. 8 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0028] The present invention will be described in detail with reference to the attached drawings. Herein, repetitive descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations are omitted. The embodiments of the present invention are provided to more fully explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

[0029] Throughout the specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0030] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0031] FIG. 1 is a block diagram showing entities for providing responses to user questions by identifying the user's speech intention according to one embodiment of the present invention.

[0032] Referring to FIG. 1, the entities for providing a response to a user's question by identifying the user's speech intention according to one embodiment of the present invention include a data collection device (110) and a video camera (120).

[0033] The data collection device (110) may refer to a device that receives images from a video camera (120).

[0034] The data collection device (110) may be a device that analyzes an image using an artificial intelligence learning model to generate an image analysis result.

[0035] The data collection device (110) may be a device that stores collected data corresponding to an image.

[0036] The video camera (120) may be a device that provides video to the data collection device (110).

[0037] The data collection device (110) and the video camera (120) can be interconnected through a communication network.

[0038] A communications network refers to a connection path that enables data to be transmitted and received between the above entities. For example, a communications network may encompass wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communications. However, the scope of communications networks applicable to the present invention is not limited thereto.

[0039] FIG. 2 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0040] Referring to FIG. 2, a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention can first receive a first image of an HMI (Human-Machine Interface) screen captured from a video camera (S210).

[0041] Here, the first image may include an image captured in real time of the HMI screen.

[0042] Next, the first image can be analyzed to generate an image analysis result based on an artificial intelligence learning model that analyzes the image and generates an analysis result (S220).

[0043] Next, based on the image analysis results, the collected data corresponding to the first image can be stored (S230).

[0044] FIG. 3 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0045] Referring to FIG. 3, a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention can train the artificial intelligence learning model using the collected data after the step of storing the collected data (S310).

[0046] At this time, the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result, and a second artificial intelligence learning model that generates an image analysis result for the first image based on the judgment result and the information.

[0047] Here, the AI ​​learning model can be a model that learns through supervised learning using deep learning. Deep learning refers to a technology in which computers learn by combining and analyzing external data. This method utilizes artificial neural networks, which mimic the structure of neurons and synapses in the human brain, in machine learning, and increases prediction accuracy by overlapping various neural networks.

[0048] Additionally, supervised learning refers to a learning method that trains a model using data with known answers and then predicts the results for new data.

[0049] A typical deep learning process begins with initializing parameters and defining hyperparameters. Here, parameters are variables determined within the AI ​​learning model and can be derived from data, while hyperparameters are values ​​directly set by the algorithm user based on experience. Hyperparameters have no set optimal values ​​and can vary depending on the learning model or data.

[0050] Next, learning can be repeated a set number of times. When the learning process is in progress, the learning order can first be forward propagated through the artificial neural network. Next, the loss function can be calculated. Here, the loss function refers to a function that calculates the difference between the expected value and the actual value based on the input. Next, backward propagation can be performed through the artificial neural network. Next, the parameters can be updated. The above process can be repeated a set number of times to create a learned model.

[0051] FIG. 4 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0052] Referring to FIG. 4, a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention may first provide a first interface capable of receiving input of first data modification information for the judgment result from a user at the user terminal (S410).

[0053] Users can modify data through the first interface.

[0054] Next, an image analysis result for the first image can be generated based on the first data modification information for the judgment result received through the first interface (S420).

[0055] Next, a second interface capable of receiving input of second data modification information for the image analysis result from the user can be provided to the user terminal (S430).

[0056] Next, retraining of the first artificial intelligence learning model and the second artificial intelligence learning model can be performed based on the first data modification information for the judgment result received through the first interface and the second data modification information for the image analysis result received through the second interface (S440).

[0057] Next, the first artificial intelligence learning model and the second artificial intelligence learning model can be updated through the relearning (S450).

[0058] FIG. 5 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0059] Referring to FIG. 5, a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention may display the reflection ratio for each item starting from the items that had the greatest influence on the image analysis result after the step of providing the second interface, and display items that can be modified among the items (S510).

[0060] Next, when the first item among the above items is selected by the user, samples of the image analysis results with the reflection ratio for the first item changed can be displayed (S520).

[0061] Next, the first image analysis result selected by the user among the above samples can be received (S530).

[0062] At this time, the step S440 may perform re-learning of the second artificial intelligence learning model based on the first image analysis result.

[0063] This allows users to easily modify data by selecting one of the displayed samples without having to check and modify all the data themselves when training an artificial intelligence learning model.

[0064] As an optional embodiment, retraining of an artificial intelligence learning model can be performed using modified data directly modified by the user.

[0065] Specifically, it is possible to receive modified data directly modified from the user regarding the image analysis results through the second interface.

[0066] Next, retraining of the second artificial intelligence learning model can be performed based on the received modified data.

[0067] FIG. 6 is a flowchart illustrating a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0068] Referring to FIG. 6, a method for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention can provide the collected data to a user through an integrated dashboard after the step of storing the collected data (S610).

[0069] FIG. 7 is a diagram showing the configuration of a device that collects data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention.

[0070] Referring to FIG. 7, a device for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention may include an image receiving unit that receives an image from a video camera, an image analysis unit that analyzes the image based on an artificial intelligence learning model to generate an image analysis result, and a data storage unit that stores collected data corresponding to the image.

[0071] According to one embodiment, the image receiving unit receives a first image captured from the HMI (Human-Machine Interface) screen from the image camera, the image analysis unit analyzes the first image based on the artificial intelligence learning model that analyzes the image to generate an analysis result, and generates an image analysis result, and the data storage unit can store collected data corresponding to the first image based on the image analysis result.

[0072] According to one embodiment, the data collection device further includes an artificial intelligence learning unit that trains the artificial intelligence learning model, and the artificial intelligence learning unit can train the artificial intelligence learning model using the collected data.

[0073] At this time, the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result, and a second artificial intelligence learning model that generates an image analysis result for the first image based on the judgment result and the information.

[0074] According to one embodiment, the artificial intelligence learning unit provides a first interface for receiving input of data correction information for the judgment result from a user on the user terminal, generates an image analysis result for the first image based on the data correction information for the judgment result received through the first interface, provides a second interface for receiving input of data correction information for the image analysis result from a user on the user terminal, performs retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the data correction information for the judgment result received through the first interface and the data correction information for the image analysis result received through the second interface, and updates the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0075] According to one embodiment, the artificial intelligence learning unit may display a reflection ratio for each item starting from the items that had the greatest influence on the image analysis result through the second interface, display items that can be modified among the items, and, when a first item among the items is selected by the user, display samples of the image analysis result with a changed reflection ratio for the first item, receive a first image analysis result selected by the user among the samples, and perform retraining of the second artificial intelligence learning model based on the first image analysis result.

[0076] FIG. 8 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0077] A device for collecting data through HMI screen recognition and image analysis using a camera according to one embodiment of the present invention can be implemented in a computer system (1000) such as a computer-readable recording medium.

[0078] Referring to FIG. 8, a computer system (1000) may include one or more processors (1010), memory (1030), a user interface input device (1040), a user interface output device (1050), and storage (1060) that communicate with each other via a bus (1020). In addition, the computer system (1000) may further include a network interface (1070) connected to a network (1080). The processor (1010) may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory (1030) or the storage (1060). The memory (1030) and the storage (1060) may be various forms of volatile or non-volatile storage media. For example, the memory may include ROM (1031) or RAM (1032).

[0079] The specific implementations described in the present invention are merely exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely exemplary functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. Furthermore, unless specifically mentioned as "essential," "important," etc., a component may not be absolutely necessary for the application of the present invention.

[0080] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

[0081] According to the method for collecting data through HMI screen recognition and image analysis using a camera according to embodiments of the present invention, data can be collected through HMI screen recognition and image analysis using a camera, and an artificial intelligence learning model can be trained using the collected data, so there is industrial applicability.

Claims

1. A step of receiving a first image captured from a video camera of an HMI (Human-Machine Interface) screen; A step of generating an image analysis result by analyzing the first image based on an artificial intelligence learning model that analyzes the image and generates an analysis result; and A step of storing collected data corresponding to the first image based on the image analysis results. A method for collecting data through HMI screen recognition and image analysis using a camera, including:

2. In paragraph 1, After the step of saving the above collected data, A step of training the artificial intelligence learning model using the collected data. Including, The above artificial intelligence learning model is, A first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result; and A second artificial intelligence learning model that generates an image analysis result for the first image based on the above judgment result and the above information. A method for collecting data through HMI screen recognition and image analysis using a camera, including:

3. In paragraph 2, The step of training the above artificial intelligence learning model is: A step of providing a first interface capable of receiving input of first data modification information for the judgment result from a user on the user terminal; A step of generating an image analysis result for the first image based on first data modification information for the judgment result received through the first interface; A step of providing a second interface capable of receiving input of second data modification information for the image analysis result from a user on the user terminal; A step of performing retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on first data modification information for the judgment result received through the first interface and second data modification information for the image analysis result received through the second interface; and A step of updating the first artificial intelligence learning model and the second artificial intelligence learning model through the above relearning. A method for collecting data through HMI screen recognition and image analysis using a camera, including:

4. In paragraph 3, After the step of providing the second interface, A step of displaying the reflection ratio of each item starting from the items that had the greatest impact on the above image analysis results, and displaying items that can be modified among the above items; When the first item among the items is selected by the user, a step of displaying samples of the image analysis results with a changed reflection ratio for the first item; and A step of receiving a first image analysis result selected by the user among the above samples Including, The steps for performing the above relearning are: A step of performing retraining of the second artificial intelligence learning model based on the first image analysis result. A method for collecting data through HMI screen recognition and image analysis using a camera, including:

5. In paragraph 4, After the step of saving the above collected data, Steps to provide the above collected data to users through an integrated dashboard A method for collecting data through HMI screen recognition and image analysis using a camera, including:

6. A video receiving unit that receives video from a video camera; An image analysis unit that analyzes the image based on an artificial intelligence learning model and generates an image analysis result; and A data storage unit that stores collected data corresponding to the above image A device that collects data through HMI screen recognition and image analysis using a camera, including:

7. In paragraph 6, The above video receiving unit, Receive a first image captured from the above video camera, which is an HMI (Human-Machine Interface) screen, The above image analysis unit, Generating an image analysis result by analyzing the first image based on the artificial intelligence learning model that analyzes the image and generates an analysis result, The above data storage unit, A device for collecting data through HMI screen recognition and image analysis using a camera, based on the results of the image analysis, and storing collected data corresponding to the first image.

8. In paragraph 7, An artificial intelligence learning unit that trains the above artificial intelligence learning model Including more, The above artificial intelligence learning unit, Using the above collected data, the artificial intelligence learning model is trained, The above artificial intelligence learning model is, A first artificial intelligence learning model that determines what information is displayed on the HMI in the first image and generates a judgment result; and A second artificial intelligence learning model that generates an image analysis result for the first image based on the above judgment result and the above information. A device that collects data through HMI screen recognition and image analysis using a camera, including:

9. In paragraph 8, The above artificial intelligence learning unit, Provide a first interface that can receive input of data modification information for the judgment result from a user on the user terminal, Generate an image analysis result for the first image based on data modification information for the judgment result received through the first interface, Provide a second interface for receiving input of data modification information for the image analysis result from the user terminal, Retraining of the first artificial intelligence learning model and the second artificial intelligence learning model is performed based on data modification information for the judgment result received through the first interface and data modification information for the image analysis result received through the second interface. A device that collects data through HMI screen recognition and image analysis using a camera, and updates the first artificial intelligence learning model and the second artificial intelligence learning model through the above relearning.

10. In paragraph 9, The above artificial intelligence learning unit, Through the second interface, the items that had the greatest impact on the video analysis results are displayed, and the reflection ratio for each item is displayed, and among the items, the items that can be modified are displayed. When the first item among the above items is selected by the user, samples of the image analysis results with the reflection ratio for the first item are displayed, Receive the first image analysis result selected by the user among the above samples, A device that collects data through HMI screen recognition and image analysis using a camera, which performs retraining of the second artificial intelligence learning model based on the results of the first image analysis.

Images