Method and apparatus for training artificial intelligence learning model by using data collected through software for image capture

Abstract

Disclosed is a method for training an artificial intelligence learning model by using data collected through software for image capture. The method for training an artificial intelligence learning model by using data collected through software for image capture, according to an embodiment of the present invention, may comprise the steps of: receiving a first image, which is a screen captured by using the software for image capture, from a human-machine interface (HMI) device in which the software for image capture is installed; analyzing the first image on the basis of an artificial intelligence learning model, and generating collection target data corresponding to the first image by using a first analysis result according to the analysis; and training the artificial intelligence learning model by using the collection target data.

Description

Method and device for training an artificial intelligence learning model using data collected through video capture software

[0001] The present invention relates to a method and device for training an artificial intelligence learning model using data collected through software for image capture.

[0002] The content described below merely provides background information related to one embodiment of the present invention and does not constitute prior art.

[0003] In factories and power plants, the Supervisory Control and Data Acquisition (SCADA) system communicates with Programmable Logic Controllers (PLCs) and input / output sensors to obtain information about device operation. This information is then displayed on a Human-Machine Interface (HMI). The HMI presents this information in graphs, charts, or other easily readable and understandable visual formats.

[0004] However, since the S / W installation and system modification for data collection of these SCADA or PLC systems are very limited, it is very difficult to collect data for the integrated management system.

[0005] Therefore, a method is required to collect data by recognizing and analyzing charts and numbers from the HMI screen using image recognition, rather than the conventional method of directly modifying data from the system.

[0006] The purpose of the present invention is to train an artificial intelligence learning model using data collected through video capture software.

[0007] In addition, the present invention aims to store and provide data collected through video capture software to a user.

[0008] According to one embodiment of the present invention for achieving the above-described purpose, a method for training an artificial intelligence learning model using data collected through image capture software may include a step of receiving a first image, which is a screen captured using image capture software, from an HMI (Human-Machine Interface) device having the image capture software installed, a step of analyzing the first image based on an artificial intelligence learning model, and a step of generating collection target data corresponding to the first image using a first analysis result according to the analysis, and a step of training the artificial intelligence learning model using the collection target data.

[0009] The above artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result, and a second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information.

[0010] The step of training the artificial intelligence learning model may include a step of providing a first interface for receiving input of first data modification information for the judgment result from a user on the user terminal, a step of generating a second analysis result for the first image based on the first data modification information for the judgment result received through the first interface, a step of providing a second interface for receiving input of second data modification information for the second analysis result from a user on the user terminal, a step of performing retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the first data modification information and the second data modification information, and a step of updating the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0011] After the step of providing the second interface, the step of displaying the reflection ratio for each item starting from the items that had the greatest influence on the second analysis result, and displaying items that can be modified among the items, the step of displaying samples of the analysis result in which the reflection ratio for the first item has been changed when the first item among the items is selected by the user, and the step of receiving a third analysis result selected by the user among the samples, wherein the step of performing re-learning may include the step of performing re-learning of the second artificial intelligence learning model based on the third analysis result.

[0012] Before the step of receiving the first image, the method may include a step of determining whether the first information displayed on the HMI device is information that needs to be transmitted to the user, and a step of capturing the screen of the HMI device using the image capture software to generate the first image if the first information is determined to be information that needs to be transmitted to the user.

[0013] After the step of generating the above-mentioned collected data, the method may include a step of storing the collected target data corresponding to the first image based on the analysis result and a step of providing the collected target data to the user through an integrated dashboard.

[0014] In addition, a device for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may include an image receiving unit for receiving an image captured from an HMI (Human-Machine Interface) device on which image capture software is installed, a data generating unit for generating data to be collected by analyzing the image based on the artificial intelligence learning model, and an artificial intelligence learning unit for training the artificial intelligence learning model using the data to be collected.

[0015] The image receiving unit receives a first image, which is a screen captured using the image capture software, from the HMI device, the data generating unit analyzes the first image based on the artificial intelligence learning model, and generates collection target data corresponding to the first image using a first analysis result according to the analysis, the artificial intelligence learning unit trains the artificial intelligence learning model using the collection target data, and the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result, and a second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information.

[0016] The artificial intelligence learning unit provides a first interface for receiving input of first data modification information for the judgment result from a user on the user terminal, generates a second analysis result for the first image based on the first data modification information for the judgment result received through the first interface, provides a second interface for receiving input of second data modification information for the second analysis result from a user on the user terminal, performs retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the first data modification information and the second data modification information, and updates the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0017] The artificial intelligence learning unit may display the reflection ratio of each item starting from the items that had the greatest impact on the second analysis result through the second interface, display items that can be modified among the items, and, when the user selects a first item among the items, display samples of analysis results with a changed reflection ratio for the first item, receive a third analysis result selected by the user among the samples, and perform retraining of the second artificial intelligence learning model based on the third analysis result.

[0018] According to the present invention, an artificial intelligence learning model can be trained using data collected through software for image capture.

[0019] Additionally, according to the present invention, data collected through software for image capture can be stored and provided to a user.

[0020] FIG. 1 is a block diagram showing entities for learning an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0021] FIG. 2 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0022] FIG. 3 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0023] FIG. 4 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0024] FIG. 5 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0025] FIG. 6 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0026] FIG. 7 is a diagram showing the configuration of a device that trains an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0027] FIG. 8 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0028] The present invention will be described in detail with reference to the attached drawings. Herein, repetitive descriptions, well-known functions that may unnecessarily obscure the gist of the present invention, and detailed descriptions of configurations will be omitted. The embodiments of the present invention are provided to more fully explain the present invention to those of ordinary skill in the art. Accordingly, the shapes and sizes of elements in the drawings may be exaggerated for clarity.

[0029] Throughout the specification, whenever a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0030] Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0031] FIG. 1 is a block diagram showing entities for providing responses to user questions by identifying the user's speech intention according to one embodiment of the present invention.

[0032] Referring to FIG. 1, the entities for providing a response to a user's question by identifying the user's speech intention according to one embodiment of the present invention include an artificial intelligence model learning device (110) and an HMI device (120).

[0033] The artificial intelligence model learning device (110) may refer to a device that receives an image captured from an HMI device (120).

[0034] The artificial intelligence model learning device (110) may be a device that analyzes an image using an artificial intelligence learning model and generates data to be collected using the analysis results.

[0035] The artificial intelligence model learning device (110) may be a device that learns an artificial intelligence learning model using collected target data.

[0036] The HMI device (120) may be a device that provides captured images to the artificial intelligence model learning device (110).

[0037] The artificial intelligence model learning device (110) and the HMI device (120) can be interconnected through a communication network.

[0038] A communications network refers to a connection path that enables data to be transmitted and received between the above entities. For example, a communications network may encompass wired networks such as LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), or wireless networks such as wireless LANs, CDMA, Bluetooth, and satellite communications. However, the scope of communications networks applicable to the present invention is not limited thereto.

[0039] FIG. 2 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0040] Referring to FIG. 2, a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may first receive a first image, which is a screen captured using the image capture software, from an HMI (Human-Machine Interface) device on which the image capture software is installed (S210).

[0041] Next, the first image may be analyzed based on an artificial intelligence learning model, and the first analysis result according to the analysis may be used to generate the collection target data corresponding to the first image (S220).

[0042] Next, the artificial intelligence learning model can be trained using the collected target data (S230).

[0043] At this time, the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result, and a second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information.

[0044] Here, the AI ​​learning model can be a model that learns through supervised learning using deep learning. Deep learning refers to a technology in which computers learn by combining and analyzing external data. This method utilizes artificial neural networks, which mimic the structure of neurons and synapses in the human brain, in machine learning, and increases prediction accuracy by overlapping various neural networks.

[0045] Additionally, supervised learning refers to a learning method that trains a model using data with known answers and then predicts the results for new data.

[0046] A typical deep learning process begins with initializing parameters and defining hyperparameters. Here, parameters are variables determined within the AI ​​learning model and can be derived from data, while hyperparameters are values ​​directly set by the algorithm user based on experience. Hyperparameters have no set optimal values ​​and can vary depending on the learning model or data.

[0047] Next, learning can be repeated a set number of times. When the learning process is in progress, the learning order can first be forward propagated through the artificial neural network. Next, the loss function can be calculated. Here, the loss function refers to a function that calculates the difference between the expected value and the actual value based on the input. Next, backward propagation can be performed through the artificial neural network. Next, the parameters can be updated. The above process can be repeated a set number of times to create a learned model.

[0048] FIG. 3 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0049] Referring to FIG. 3, a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may first provide a first interface capable of receiving input of first data modification information for the judgment result from a user on the user terminal (S310).

[0050] Users can modify data through the first interface.

[0051] Next, a second analysis result for the first image can be generated based on the first data modification information for the judgment result received through the first interface (S320).

[0052] Next, a second interface can be provided to the user terminal to receive input of second data modification information for the second analysis result from the user (S330).

[0053] Next, retraining of the first artificial intelligence learning model and the second artificial intelligence learning model can be performed based on the first data modification information and the second data modification information (S340).

[0054] Next, the first artificial intelligence learning model and the second artificial intelligence learning model can be updated through the relearning (S350).

[0055] FIG. 4 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0056] Referring to FIG. 4, a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may display a reflection ratio for each item starting from the items that had the greatest impact on the second analysis result after providing the second interface, and display items that can be modified among the items (S410).

[0057] Next, when the first item among the above items is selected by the user, samples of the analysis results with the reflection ratio for the first item changed can be displayed (S420).

[0058] Next, the third analysis result selected by the user among the above samples can be received (S430).

[0059] At this time, the step S340 may perform re-learning of the second artificial intelligence learning model based on the third analysis result.

[0060] This allows users to easily modify data by selecting one of the displayed samples without having to check and modify all the data themselves when training an artificial intelligence learning model.

[0061] As an optional embodiment, retraining of an artificial intelligence learning model can be performed using modified data directly modified by the user.

[0062] Specifically, the second interface can receive modified data directly modified from the user regarding the second analysis results.

[0063] Next, retraining of the second artificial intelligence learning model can be performed based on the received modified data.

[0064] FIG. 5 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0065] Referring to FIG. 5, a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention can determine whether first information displayed on the HMI device is information that needs to be transmitted to the user before the step of receiving the first image (S510).

[0066] Next, if it is determined that the first information is information that needs to be transmitted to the user, the first image can be generated by capturing the screen of the HMI device through the image capture software (S520).

[0067] This prevents the collection of useless data by capturing information that does not need to be conveyed to the user.

[0068] FIG. 6 is a flowchart illustrating a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0069] Referring to FIG. 6, a method for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may store collected target data corresponding to the first image based on the analysis result after the step of generating the collected data (S610).

[0070] Next, the collected data can be provided to the user through an integrated dashboard (S620).

[0071] FIG. 7 is a diagram showing the configuration of a device that trains an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention.

[0072] Referring to FIG. 7, a device for training an artificial intelligence learning model using data collected through image capture software according to one embodiment of the present invention may include an image receiving unit for receiving an image captured from an HMI (Human-Machine Interface) device on which image capture software is installed, a data generating unit for generating data to be collected by analyzing the image based on an artificial intelligence learning model, and an artificial intelligence learning unit for training the artificial intelligence learning model using the data to be collected.

[0073] According to one embodiment, the image receiving unit receives a first image, which is a screen captured using the image capturing software, from the HMI device, the data generating unit analyzes the first image based on the artificial intelligence learning model, and generates collection target data corresponding to the first image using a first analysis result according to the analysis, and the artificial intelligence learning unit can train the artificial intelligence learning model using the collection target data.

[0074] At this time, the artificial intelligence learning model may include a first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result, and a second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information.

[0075] According to one embodiment, the artificial intelligence learning unit provides a first interface for receiving input of first data modification information for the judgment result from a user on the user terminal, generates a second analysis result for the first image based on the first data modification information for the judgment result received through the first interface, provides a second interface for receiving input of second data modification information for the second analysis result from a user on the user terminal, performs retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the first data modification information and the second data modification information, and updates the first artificial intelligence learning model and the second artificial intelligence learning model through the retraining.

[0076] According to one embodiment, the artificial intelligence learning unit may display a reflection ratio for each item starting from the items that had the greatest impact on the second analysis result through the second interface, display items that can be modified among the items, and, when a first item among the items is selected by the user, display samples of analysis results with a changed reflection ratio for the first item, receive a third analysis result selected by the user among the samples, and perform retraining of the second artificial intelligence learning model based on the third analysis result.

[0077] FIG. 8 is a diagram illustrating a computer system according to one embodiment of the present invention.

[0078] A device for training an artificial intelligence learning model using data collected through software for image capture according to one embodiment of the present invention can be implemented in a computer system (1000) such as a computer-readable recording medium.

[0079] Referring to FIG. 8, a computer system (1000) may include one or more processors (1010), memory (1030), a user interface input device (1040), a user interface output device (1050), and storage (1060) that communicate with each other via a bus (1020). In addition, the computer system (1000) may further include a network interface (1070) connected to a network (1080). The processor (1010) may be a central processing unit or a semiconductor device that executes processing instructions stored in the memory (1030) or the storage (1060). The memory (1030) and the storage (1060) may be various forms of volatile or non-volatile storage media. For example, the memory may include ROM (1031) or RAM (1032).

[0080] The specific implementations described in the present invention are merely exemplary embodiments and do not limit the scope of the present invention in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the lines connecting or connecting members between components depicted in the drawings are merely exemplary functional connections and / or physical or circuit connections, and may be replaced or represented as various additional functional connections, physical connections, or circuit connections in an actual device. Furthermore, unless specifically mentioned as "essential," "important," etc., a component may not be absolutely necessary for the application of the present invention.

[0081] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

[0082] According to a method for training an artificial intelligence learning model using data collected through image capture software according to embodiments of the present invention, data can be collected through HMI screen recognition and image analysis through software, and an artificial intelligence learning model can be trained using the collected data, so there is industrial applicability.

Claims

1. A step of receiving a first image, which is a screen captured using image capture software, from an HMI (Human-Machine Interface) device having image capture software installed thereon; A step of analyzing the first image based on an artificial intelligence learning model and generating data to be collected corresponding to the first image using the first analysis result according to the analysis; and A step of training the artificial intelligence learning model using the collected data. A method for training an artificial intelligence learning model using data collected through software for image capture, including:

2. In paragraph 1, The above artificial intelligence learning model is, A first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result; and A second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information. A method for training an artificial intelligence learning model using data collected through software for image capture, including:

3. In paragraph 2, The step of training the above artificial intelligence learning model is: A step of providing a first interface capable of receiving input of first data modification information for the judgment result from a user on the user terminal; A step of generating a second analysis result for the first image based on first data modification information for the judgment result received through the first interface; A step of providing a second interface capable of receiving input of second data modification information for the second analysis result from a user on the user terminal; A step of performing retraining of the first artificial intelligence learning model and the second artificial intelligence learning model based on the first data modification information and the second data modification information; and A step of updating the first artificial intelligence learning model and the second artificial intelligence learning model through the above relearning. A method for training an artificial intelligence learning model using data collected through software for image capture, including:

4. In paragraph 3, After the step of providing the second interface, A step of displaying the reflection ratio for each item starting from the items that had the greatest impact on the second analysis results and displaying items that can be modified among the items; When the first item among the items is selected by the user, a step of displaying samples of analysis results with a changed reflection ratio for the first item; and A step of receiving a third analysis result selected by the user among the above samples Including, The steps for performing the above relearning are: A step of performing retraining of the second artificial intelligence learning model based on the third analysis result. A method for training an artificial intelligence learning model using data collected through software for image capture, including:

5. In paragraph 1, Before the step of receiving the first image, A step of determining whether the first information displayed on the HMI device is information that needs to be transmitted to the user; and If it is determined that the first information is information that needs to be transmitted to the user, a step of capturing the HMI device screen through the image capture software to generate the first image A method for training an artificial intelligence learning model using data collected through software for image capture, including:

6. In paragraph 1, After the step of generating the above collected data, A step of storing the collected target data corresponding to the first image based on the analysis result; and Steps to provide the above collected data to users through an integrated dashboard A method for training an artificial intelligence learning model using data collected through software for image capture, including:

7. An image receiving unit that receives images captured from an HMI (Human-Machine Interface) device on which image capture software is installed; A data generation unit that analyzes the image based on an artificial intelligence learning model to generate data to be collected; and An artificial intelligence learning unit that trains the artificial intelligence learning model using the collected data. A device for training an artificial intelligence learning model using data collected through software for image capture, including:

8. In paragraph 7, The above image receiving unit, Receive a first image, which is a screen captured using the image capture software, from the HMI device, The above data generation unit, The first image is analyzed based on the artificial intelligence learning model, and the first analysis result according to the analysis is used to generate the collection target data corresponding to the first image, The above artificial intelligence learning unit, Using the above collected data, the artificial intelligence learning model is trained, The above artificial intelligence learning model is, A first artificial intelligence learning model that determines what information is displayed on the HMI device based on the first image and generates a judgment result; and A second artificial intelligence learning model that generates the first analysis result for the first image based on the judgment result and the information. A device for training an artificial intelligence learning model using data collected through software for image capture, including:

9. In paragraph 8, The above artificial intelligence learning unit, Provide a first interface for receiving input of first data modification information for the judgment result from a user on the user terminal, Generate a second analysis result for the first image based on the first data modification information for the judgment result received through the first interface, Provide a second interface for receiving input of second data modification information for the second analysis result from the user terminal, Retraining of the first artificial intelligence learning model and the second artificial intelligence learning model is performed based on the first data modification information and the second data modification information, A device for training an artificial intelligence learning model using data collected through image capture software, which updates the first artificial intelligence learning model and the second artificial intelligence learning model through the above retraining.

10. In paragraph 9, The above artificial intelligence learning unit, Through the second interface, the items that had the greatest impact on the second analysis results are displayed, and the reflection ratio for each item is displayed, and among the items, the items that can be modified are displayed. When the first item among the above items is selected by the user, samples of the analysis results with the reflection ratio for the first item are displayed, Receive the third analysis result selected by the user among the above samples, A device for training an artificial intelligence learning model using data collected through image capture software, which performs retraining of the second artificial intelligence learning model based on the third analysis results.

Images