CONTROL SYSTEM, SUPPORT DEVICE AND LABELING METHOD
The control system enhances labeling efficiency by correlating images with collected data, addressing the imbalance of labeled abnormal data in predictive maintenance systems, thereby improving the accuracy and efficiency of labeling processes.
Patent Information
- Application Number
- JP2020216403
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-25
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2040-12-25
AI Technical Summary
Existing methods for labeling data in predictive maintenance systems face challenges due to the imbalance of labeled abnormal data compared to normal data, making efficient labeling difficult at production sites.
A control system with an image collection unit, user interface, and generation unit that allows for efficient labeling by correlating images with collected data, enabling accurate and efficient labeling of abnormal conditions through image similarity and temporal correlation.
Improves the accuracy and efficiency of labeling processes by allowing users to set appropriate labels based on image analysis, facilitating the generation of learning data suitable for predictive maintenance systems.
Smart Images

Figure 0007581861000001 
Figure 0007581861000002 
Figure 0007581861000003
Abstract
Description
[Technical field]
[0001] The present technology relates to a control system, a support device connected to a control device included in the control system, and a labeling method for use in the control system. [Background technology]
[0002] In various production sites, there is a need to improve the equipment operation rate by predictive maintenance of machines and equipment. Predictive maintenance means a form of maintenance in which some abnormality occurring in machines and equipment is detected and maintenance work such as repair or replacement is performed before the equipment needs to be stopped. In order to realize such predictive maintenance, a mechanism using machine learning has been put into practical use that collects the status values of machines and equipment and determines whether or not some abnormality has occurred in the machines and equipment based on the collected status values.
[0003] To realize such machine learning, it is necessary to prepare a learning dataset consisting of labeled data. Generally, labeling is a labor-intensive and time-consuming process. For example, JP2018-537798A (Patent Document 1) discloses labeling or label verification for achieving efficient labeling of unlabeled components of a target dataset. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] Special Publication No. 2018-537798 Summary of the Invention [Problem to be solved by the invention]
[0005] When considering operations in actual production sites, there is less data labeled as abnormal compared to data labeled as normal, and it is not easy to apply the label verification method disclosed in JP2018-537798A (Patent Document 1).
[0006] There is a demand for methods of labeling and acquiring training data that are suitable for use in actual production sites. One of the aims of this technology is to meet this need. [Means for solving the problem]
[0007] A control system according to an example of the present technology includes a control calculation unit that executes control calculations for controlling a control object, an anomaly detection unit that detects anomalies in the control object by referring to a model based on information collected from the control object, an image collection unit that collects images of objects related to the target of anomaly detection, a user interface unit that outputs the information collected from the control object and the captured images in a time-correlated manner and accepts label designation, and a generation unit that generates learning data for constructing a model by adding the designated label to the information collected from the control object.
[0008] According to this configuration, an appropriate label can be set while referring to an image captured of an object related to the target of anomaly detection, thereby improving the accuracy of labeling and the efficiency of the labeling work.
[0009] The user interface unit may output an image corresponding to selected information from the information collected from the control target. With this configuration, even if a plurality of images are collected, the corresponding image can be appropriately output.
[0010] The user interface unit may reflect, for each of the pieces of information corresponding to a plurality of similar images, a label that is the same as a label designated for information corresponding to any one of the plurality of images. With this configuration, it is possible to improve the efficiency of labeling by utilizing the similarity of the images.
[0011] The user interface unit may reflect the same designated label to a plurality of pieces of information collected from the control target. With this configuration, for example, the same label can be collectively applied to pieces of information that are obviously abnormal, thereby improving the efficiency of labeling.
[0012] The image may include at least one of a still image and a video. With this configuration, either a still image or a video can be selected depending on the situation.
[0013] When the image includes a video, the user interface unit may display the position of the corresponding information among the information collected from the control target in conjunction with the playback of the video. With this configuration, even for videos recorded in chronological order, the image for labeling can be easily confirmed.
[0014] The control system may further include a model generating unit that generates a model based on the learning data. With this configuration, a model used for anomaly detection can be generated based on the generated image data.
[0015] The control system may further include a feature calculation unit that calculates feature values from state values of the control target as information collected from the control target. With this configuration, learning data suitable for use in, for example, detecting an anomaly using the feature values can be generated.
[0016] According to another example of the present technology, there is provided a support device connected to a control device including a control calculation unit that executes a control calculation for controlling a control target, and an anomaly detection unit that detects an anomaly in the control target by referring to a model based on information collected from the control target. The support device includes an image acquisition unit that acquires an image of an object related to the target of anomaly detection, a user interface unit that outputs the information collected from the control target and the captured image in a time-correlated manner and accepts the designation of a label, and a generation unit that generates learning data for constructing a model by adding the designated label to the information collected from the control target.
[0017] According to yet another example of the present technology, there is provided a labeling method for a control system that detects an abnormality in a control object by referring to a model based on information collected from the control object. The labeling method includes a step of acquiring information collected from the control object, a step of acquiring an image of an object related to the target of anomaly detection, a step of outputting the information collected from the control object and the captured image in a time-correlated manner, a step of accepting a label designation, and a step of generating learning data for constructing a model by adding the designated label to the information collected from the control object. Effect of the Invention
[0018] According to this technology, it is possible to realize a method of labeling and acquiring learning data suitable for operation in actual production sites. [Brief description of the drawings]
[0019] [Figure 1] 1 is a schematic diagram showing an example of the overall configuration of a control system according to an embodiment of the present invention; [Diagram 2] 1 is a diagram for explaining an example of a main configuration and an operation form of a control system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram for explaining an overview of a label assignment process in the control system according to the present embodiment. [Figure 4]2 is a block diagram showing an example of a hardware configuration of a control device constituting the control system according to the present embodiment. FIG. [Diagram 5] 2 is a block diagram showing an example of a hardware configuration of a support device constituting the control system according to the present embodiment. FIG. [Figure 6] 2 is a block diagram showing an example of a hardware configuration of an image collecting unit that constitutes the control system according to the present embodiment. FIG. [Figure 7] FIG. 2 is a schematic diagram showing an example of a functional configuration for implementing a label assignment process in the control system according to the present embodiment. [Figure 8] 4 is a diagram for explaining association between time-series data and images in the control system according to the present embodiment. FIG. [Figure 9] FIG. 4 is a schematic diagram showing an example of a user interface screen provided in the control system according to the present embodiment. [Figure 10] FIG. 11 is a schematic diagram showing an example of a procedure for adding a label on a user interface screen provided in the control system according to the present embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an example of a processing procedure for reflecting one label on another on a user interface screen provided in the control system according to the present embodiment. [Figure 12] FIG. 11 is a schematic diagram showing an example of a processing procedure for reflecting one label on another on a user interface screen provided in the control system according to the present embodiment. [Figure 13] FIG. 11 is a schematic diagram showing an example of a labeling process procedure when no image is present on a user interface screen provided in the control system according to the present embodiment. [Figure 14] FIG. 11 is a schematic diagram showing an example of a labeling process procedure when no image is present on a user interface screen provided in the control system according to the present embodiment. [Figure 15] 10 is a schematic diagram showing an example of a processing procedure for assigning a label to a unique feature on a user interface screen provided in the control system according to the present embodiment. FIG. [Figure 16] It is a schematic diagram showing an example of displaying a video on a user interface screen provided in the control system according to this embodiment. [Figure 17] It is a flowchart showing a processing procedure related to the labeling process in the control system according to this embodiment.
Embodiments for Carrying Out the Invention
[0020] Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the drawings are denoted by the same reference numerals and their descriptions will not be repeated.
[0021] <A. Application Example> First, an example of a scenario to which the present invention is applied will be described.
[0022] FIG. 1 is a schematic diagram showing an overall configuration example of a control system 1 according to this embodiment. Referring to FIG. 1, the control system 1 according to this embodiment mainly includes a control device 100 that controls a control target, a support device 200 connected to the control device 100, and an image collection unit 300 that collects images (still images and / or moving images) used for labeling.
[0023] The control device 100 may be embodied as a kind of computer such as a PLC (Programmable Logic Controller). The control device 100 is connected to the field device group 10 via a field bus 2. It is preferable that the field bus 2 adopts an industrial communication protocol. Such communication protocols include EtherCAT (registered trademark), EtherNet / IP (registered trademark), DeviceNet (registered trademark), CompoNet (registered trademark), etc.
[0024] The field devices 10 include devices that collect input data from controlled objects or manufacturing devices and production lines related to the control (hereinafter collectively referred to as "fields"). Assumed devices for collecting such input data include input relays and various sensors. The field devices 10 further include devices that perform some kind of action on the field based on commands (hereinafter also referred to as "output data") generated by the control device 100. Assumed devices for performing some kind of action on such fields include output relays, contactors, servo drivers and servo motors, and any other actuators. These field devices 10 exchange data including input data and output data with the control device 100 via the field bus 2.
[0025] In the exemplary configuration shown in FIG. 1, the field device group 10 includes a remote I / O (Input / Output) device 12, a relay group 14, a servo driver 18, and a servo motor 20.
[0026] The remote I / O device 12 includes a communication unit that communicates via the field bus 2, and an input / output unit (hereinafter also referred to as an "I / O unit") for collecting input data and outputting output data. Input data and output data are exchanged between the control device 100 and the field via such an I / O unit. Fig. 1 shows an example in which digital signals are exchanged as input data and output data via a relay group 14.
[0027] The I / O unit may be directly connected to the field bus. FIG 1 shows an example in which the I / O unit 16 is directly connected to the field bus 2.
[0028] The servo driver 18 drives the servo motor 20 in accordance with output data (eg, a position command) from the control device 100.
[0029] As described above, input data and output data are exchanged between the control device 100 and the field device group 10 via the field bus 2, and the exchanged data is updated at very short intervals of the order of several hundred μsec to several tens of msec. Such an update process of the exchanged data is sometimes called an "I / O refresh process."
[0030] The control device 100 has a PLC engine (PLC engine 130 shown in FIG. 2) that executes control calculations for controlling control targets such as equipment and machines. The PLC engine corresponds to a control calculation unit, and determines output data by executing control calculations based on input data. The control device 100 has a time series database (hereinafter also referred to as "TSDB (Time Series Data Base)") 140 that sequentially stores input data from the field device group 10, output data to the field device group 10, internal data managed within the control device 100, and the like. Hereinafter, the data stored in TSDB 140 will also be referred to as "time series data."
[0031] The control device 100 has an anomaly detection engine 150 that detects an anomaly in the control target by referring to a previously prepared model 160 based on information collected from the control target. Typically, the information collected from the control target is time-series data stored in the TSDB 140 (or features calculated from the time-series data). That is, the anomaly detection engine 150 judges the occurrence or possibility of an anomaly in the control target based on the time-series data stored in the TSDB 140.
[0032] The control device 100 may be connected to a server 400 via a higher-level network 6, and may be connected to one or more display devices 500 via a field bus 4. The server 400 and the display devices 500 are optional components and are not essential components of the control system 1.
[0033] The server 400 is responsible for processing such as providing any information to the control device 100 or collecting data from the control device 100 .
[0034] The display device 500 receives operations from a user, transmits commands to the control device 100 in response to the user operations, and graphically displays the results of calculations performed by the control device 100, etc.
[0035] The support device 200 is an information processing device (an example of a computer) that supports the preparations required for the control device 100 to control a control target. Specifically, the support device 200 provides a development environment (program creation and editing tools, parsers, compilers, etc.) for user programs executed by the control device 100, a setting environment for setting parameters (configurations) for the control device 100 and various devices connected to the control device 100, a function for transmitting the generated user programs to the control device 100, a function for modifying and changing the user programs executed on the control device 100 online, and the like.
[0036] Furthermore, the support device 200 has a function for supporting the generation and adjustment of the model 160 referenced by the anomaly detection engine 150 implemented in the control device 100.
[0037] The image collecting unit 300 collects images of objects related to the target of anomaly detection. More specifically, the image collecting unit 300 collects images to assist in generating learning data used by the support device 200 to generate and adjust the model 160. The image collecting unit 300 typically has a camera 330 disposed at an arbitrary position. The images may be either still images or videos, or may include both still images and videos. The images collected by the image collecting unit 300 are available for use by the support device 200.
[0038] Here, the term "things related to the target of anomaly detection" may refer to any information that can be used to generate learning data for learning the detection logic (i.e., model 160) of anomaly detection engine 150. Possible things related to the target of anomaly detection include, for example, a workpiece (finished product or semi-finished product) being produced and the production equipment itself.
[0039] Next, an example of the configuration and operation of the anomaly detection engine 150 of the control device 100 will be described. The support device 200 acquires time-series data stored in the TSDB 140, and determines parameters that define the model 160 referenced by the anomaly detection engine 150 through data mining. Then, the support device 200 transfers the determined parameters to the control device 100. This effectively configures the anomaly detection engine 150 of the control device 100.
[0040] After operation, the support device 200 acquires the time-series data stored in the TSDB 140 and acquires images from the image collector 300. A user operates the support device 200 to assign labels to the time-series data while viewing the images. The support device 200 adjusts parameters that define the model 160 based on the time-series data to which the labels have been assigned. That is, the model 160 is regenerated or adjusted. Then, the support device 200 transfers the adjusted parameters to the control device 100. This further optimizes the anomaly detection engine 150 of the control device 100.
[0041] Fig. 2 is a diagram for explaining an example of a main configuration and an operation form of the control system 1 according to the present embodiment. Referring to Fig. 2, the process of anomaly detection in the control system 1 is typically realized in the order of a data collection phase 30, a data analysis phase 32, and an operation phase 34.
[0042] In a data collection phase 30, time-series data related to a target for anomaly detection is collected. In a data analysis phase 32, feature quantities suitable for the target anomaly detection are considered by data mining or the like on the time-series data collected in the data collection phase 30, and a model 160 is generated according to the considered feature quantities. The operation phase 34 is started using the model 160 generated in the data analysis phase 32.
[0043] After the start of the operation phase 34, if an underdetection or erroneous detection occurs, such as an anomaly that should be detected being overlooked or an anomaly that should not be detected being detected, the model 160 may be updated by re-learning. Alternatively, if any change occurs in the target of anomaly detection (e.g., aging or changes in operating conditions), the model 160 may also be updated by re-learning.
[0044] In consideration of such a processing procedure, the support device 200 may include a data mining tool 240 and a model generation tool 250. The data mining tool 240 mainly provides the processing required in the data analysis phase 32. The model generation tool 250 mainly provides the processing required in the operation phase 34.
[0045] More specifically, the data mining tool 240 typically provides a feature generation process 242, a feature selection process 244, a model generation process 246, and a threshold setting process 248. The model generation tool 250 provides a model update process 252 and a threshold adjustment process 254. In addition, the model generation tool 250 provides a label assignment process 260.
[0046] The labeling process 260 is a process that supports the user in labeling the time-series data 142 (or the feature amount calculated from the time-series data 142) by using the image 342 provided from the image collection unit 300. The time-series data 142 (or the feature amount calculated from the time-series data 142) to which the label is assigned becomes training data. A collection of training data becomes a training data set, and re-training and updating of the model 160 are performed using this training data set. That is, the model updating process 252 and the threshold adjustment process 254 generate or update the model 160 based on a plurality of training data.
[0047] The image collecting section 300 has a database (hereinafter, also referred to as "DB (Time Series Data Base)") 340 that sequentially stores images 342 captured by a camera 330 placed at an arbitrary position.
[0048] As described above, the control device 100 includes the PLC engine 130, the TSDB 140, and the anomaly detection engine 150.
[0049] The PLC engine 130 periodically executes control operations defined by a user program 132 that is arbitrarily created according to a control target. An AI library 134 is available in the user program 132. Alternatively, the AI library 134 may be incorporated into the user program 132. The contents, characteristics, behavior, etc. of the AI library 134 may be determined in the data analysis phase 32 using a data mining tool 240.
[0050] When the PLC engine 130 executes a portion of the AI library 134 included in the user program 132, one or more features 152 are calculated from pre-specified data, and the calculated features 152 are provided to the anomaly detection engine 150.
[0051] The PLC engine 130 has a data management unit 136, and holds input data, output data, and internal data in a form that can be referenced by the user program 132. The values of the data managed by the data management unit 136 are updated at predetermined intervals (I / O refresh intervals) by I / O refresh processing. Furthermore, pre-specified data among the data managed by the data management unit 136 is stored at predetermined intervals in the TSDB 140. As a result, the TSDB 140 outputs the change in value of the specified data at each predetermined interval, i.e., time-series data 142.
[0052] The anomaly detection engine 150 determines the occurrence or possibility of an anomaly by referring to the model 160 based on the feature quantity 152 from the PLC engine 130. The model 160 may be determined using the data mining tool 240 in the data analysis phase 32, or may be determined using the model generation tool 250 in the operation phase 34.
[0053] Typically, model 160 is a function that outputs a value (probability or likelihood) indicating the occurrence of an anomaly when one or more feature quantities 152 are input. Parameters that define model 160 may include coefficients that define a function that is the entity of model 160, and a threshold value for evaluating a value output from model 160.
[0054] The control system 1 according to this embodiment provides a function of assisting in the generation of a set of learning data (learning data set) by the labeling process 260 and the image collection unit 300 of the support device 200, and this function makes it easier to perform re-learning and the like.
[0055] Fig. 3 is a diagram for explaining an outline of the labeling process 260 in the control system 1 according to the present embodiment. With reference to Fig. 3, the support device 200 acquires a set of time-series data 142 and an image 342 captured at the timing when the time-series data 142 was collected (observed).
[0056] The support device 200 presents the image 342 to the user, accepts the label 144 specified by the user, and associates it with the corresponding time-series data 142 (or the feature amount calculated from the time-series data 142). The label 144 includes, for example, a value indicating whether the corresponding time-series data 142 is "normal" or "abnormal".
[0057] A set of the time-series data 142 and the corresponding label 144 is used as one piece of learning data 146. A collection of the learning data 146 becomes a learning data set. By re-learning using a plurality of the learning data 146 generated by the labeling process 260, the model 160 is regenerated or adjusted.
[0058] In the labeling process 260 according to the present embodiment, since the image 342 captured at the timing when the time-series data 142 is collected is presented to the user, the user can easily grasp the label to be assigned to each of the time-series data 142. Therefore, in the operation phase 34 and the like, the learning data 146 can be created more easily.
[0059] <B. Hardware Configuration Example> Next, a hardware configuration example of the main devices constituting the control system 1 according to the present embodiment will be described.
[0060] (b1: Hardware Configuration Example of Control Device 100) Fig. 4 is a block diagram showing an example of a hardware configuration of a control device 100 constituting a control system 1 according to the present embodiment. Referring to Fig. 4, the control device 100 includes a processor 102 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a chipset 104, a main memory device 106, a secondary memory device 108, a host network controller 110, a USB (Universal Serial Bus) controller 112, a memory card interface 114, an internal bus controller 122, field bus controllers 118 and 120, and I / O units 124-1, 124-2, ....
[0061] The processor 102 reads out various programs stored in the secondary storage device 108, deploys them in the main storage device 106, and executes them to implement the PLC engine 130 and the anomaly detection engine 150. The chipset 104 controls data transmission between the processor 102 and each component, etc.
[0062] The secondary storage device 108 stores a system program 131 for implementing the PLC engine 130 and the anomaly detection engine 150, as well as a user program 132 executed using the PLC engine 130. A portion of the area of the secondary storage device 108 may be used as a TSDB 140.
[0063] The upper network controller 110 controls data exchange with other devices via the upper network 6. The USB controller 112 controls data exchange with the support device 200 via a USB connection.
[0064] The memory card interface 114 is configured so that a memory card 116 can be attached and detached, and it is possible to write data to the memory card 116 and read various data (user programs, trace data, etc.) from the memory card 116.
[0065] The internal bus controller 122 is an interface for transmitting and receiving data between the I / O units 124-1, 124-2, . . . mounted on the control device 100.
[0066] A fieldbus controller 118 controls data exchange with other devices via the fieldbus 2. Similarly, a fieldbus controller 120 controls data exchange with other devices via the fieldbus 4.
[0067] FIG. 4 shows an example of a configuration in which the processor 102 executes a program to provide the necessary functions, but some or all of these provided functions may be implemented using a dedicated hardware circuit (e.g., an Application Specific Integrated Circuit (ASIC) or a Field-Programmable Gate Array (FPGA)). Alternatively, the main part of the control device 100 may be realized using hardware that conforms to a general-purpose architecture (e.g., an industrial PC based on a general-purpose PC). In this case, virtualization technology may be used to run multiple operating systems (OSs) with different uses in parallel, and necessary applications may be executed on each OS.
[0068] (b2: Hardware configuration example of the support device 200) The support device 200 according to the present embodiment is realized, for example, by executing a program using hardware (for example, a general-purpose personal computer) that complies with a general-purpose architecture.
[0069] Fig. 5 is a block diagram showing an example of a hardware configuration of a support device 200 constituting the control system 1 according to the present embodiment. With reference to Fig. 5, the support device 200 includes a processor 202 such as a CPU or an MPU, an optical drive 204, a main storage device 206, a secondary storage device 208, a USB controller 212, a network controller 214, an input unit 216, and a display unit 218. These components are connected via a bus 220.
[0070] The processor 202 reads out various programs stored in the secondary storage device 208, deploys them in the main storage device 206, and executes them to realize various processes including a labeling process, which will be described later.
[0071] The secondary storage device 208 is composed of, for example, a hard disk drive (HDD) or a flash solid state drive (SSD). The secondary storage device 208 typically stores a PLC interface program 224 for exchanging data related to an anomaly detection function between the OS 222 and the control device 100, a development program 226 for creating a user program executed in the support device 200, debugging the created user program, defining a system configuration, setting various parameters, and the like, a data mining program 228 for implementing a data mining tool 240, and a model generation program 230 for implementing a model generation tool 250. The secondary storage device 208 may store necessary programs other than the programs shown in FIG. 5.
[0072] The support device 200 has an optical drive 204, and a computer-readable program stored in a recording medium 205 (e.g., an optical recording medium such as a DVD (Digital Versatile Disc)) is read from the recording medium 205 in which the program is non-transiently stored, and the program is installed in a secondary storage device 208, etc.
[0073] Various programs executed by the support device 200 may be installed via a computer-readable recording medium 205, or may be installed by downloading from a server device on a network. In addition, the functions provided by the support device 200 according to this embodiment may be realized by using some of the modules provided by the OS 222.
[0074] A USB controller 212 controls data transfer to and from the control device 100 via a USB connection. A network controller 214 controls data transfer to and from other devices via any network.
[0075] The input unit 216 is made up of a keyboard, a mouse, etc., and receives user operations. The display unit 218 is made up of a display, various indicators, a printer, etc., and outputs the processing results from the processor 202, etc.
[0076] FIG. 5 shows an example of a configuration in which the necessary functions are provided by the processor 202 executing a program, but some or all of these provided functions may be implemented using a dedicated hardware circuit (e.g., ASIC or FPGA).
[0077] (b3: Example of hardware configuration of image acquisition unit 300) The image collecting section 300 according to the present embodiment is realized, for example, by executing a program using hardware (for example, a general-purpose personal computer) that complies with a general-purpose architecture.
[0078] Fig. 6 is a block diagram showing an example of a hardware configuration of an image collecting unit 300 constituting the control system 1 according to the present embodiment. With reference to Fig. 6, the image collecting unit 300 includes a processor 302 such as a CPU or an MPU, a main memory device 306, a secondary memory device 308, a camera controller 312, a network controller 314, an input unit 316, and a display unit 318. These components are connected via a bus 320.
[0079] The processor 302 reads various programs stored in the secondary storage device 308, expands and executes them in the main storage device 306, thereby realizing various processes including the model generation process as described later.
[0080] The secondary storage device 308 is composed of, for example, an HDD or an SSD. Typically, an OS 322 and a system program 324 for realizing processes related to imaging and the like are stored in the secondary storage device 308. Necessary programs other than the program shown in FIG. 6 may be stored in the secondary storage device 308.
[0081] The various programs executed in the image collection unit 300 may be installed in the form of being downloaded from a server device or the like on the network. Also, the functions provided by the image collection unit 300 according to the present embodiment may be realized by using a part of the modules provided by the OS 322.
[0082] The camera controller 312 gives an imaging command to the camera 330 and acquires the image captured by the camera 330.
[0083] The network controller 314 controls the exchange of data with other devices (for example, the support device 200) via an arbitrary network.
[0084] The input unit 316 is composed of a keyboard, a mouse, etc., and receives user operations. The display unit 318 is composed of a display, various indicators, a printer, etc., and outputs the processing results from the processor 302 and the like.
[0085] FIG. 6 shows a configuration example in which functions necessary for the processor 302 to execute a program are provided. However, some or all of these provided functions may be implemented using a dedicated hardware circuit (for example, an ASIC or an FPGA).
[0086] <C. Functional Configuration Example> Next, a functional configuration example for implementing the labeling process 260 in the control system 1 according to the present embodiment will be described.
[0087] Fig. 7 is a schematic diagram showing an example of a functional configuration for realizing the label assignment process 260 in the control system 1 according to the present embodiment. Referring to Fig. 7, the control device 100 includes a time management unit 138 that manages time. The time management unit 138 manages and holds the current time, and may be realized using, for example, a radio clock, a GPS (Global Positioning System), a real-time clock, a counter, or the like. The time information managed by the time management unit 138 is added as a timestamp to the time series data 142 stored in the TSDB 140.
[0088] The image collecting unit 300 also has a time management unit 338. The time management unit 338 also manages and holds the current time. Since the times managed by the time management unit 138 and the time management unit 338 need to be synchronized, for example, the time may be synchronized between the time management unit 138 and the time management unit 338 using NTP (Network Time Protocol) or the like. In this case, the image collecting unit 300 may be connected to the control device 100 via a network, or an independent network may be constructed between the time management unit 138 and the time management unit 338.
[0089] Furthermore, when time management unit 138 and time management unit 338 manage the time using a radio clock or a GPS, they can each obtain the correct time and do not need to be synchronized with each other.
[0090] The time information managed by the time management unit 338 is added as a time stamp to the image 342 stored in the DB 340 .
[0091] Therefore, a timestamp is added to both the time series data 142 provided from the control device 100 to the support device 200 and the image 342 provided from the image collection unit 300, and correspondence between the time series data 142 and the image 342 can be achieved based on the timestamp.
[0092] The model generation tool 250 of the support device 200 includes a time-series data acquisition module 261, an image acquisition module 262, a time association module 263, a user interface module 264, and a learning data generation module 266 as functions for realizing the label assignment process 260.
[0093] The time series data acquisition module 261 acquires the time series data 142 from the TSDB 140 of the control device 100. The time series data acquisition module 261 includes a feature amount calculation module 267.
[0094] The feature calculation module 267 calculates a feature having a predetermined content from the acquired time-series data 142. That is, the feature calculation module 267 calculates a feature from state values of the control target (each value included in the time-series data 142) as information collected from the control target.
[0095] The image acquisition module 262 acquires the image 342 from the image collection unit 300 . The time association module 263 associates the time series data 142 acquired by the time series data acquisition module 261 with the image 342 acquired by the image acquisition module 262 with respect to time.
[0096] FIG. 8 is a diagram for explaining the association between the time-series data 142 and the image 342 in the control system 1 according to the present embodiment. Referring to FIG. 8, the time-series data 142 has a timestamp indicating the time. The image 342 also has a timestamp. By associating the time-series data 142 and the image 342 whose timestamp values match or are approximate, the image 342 at the time when the time-series data 142 was collected can be determined.
[0097] When a moving image is used as the image 342, a file of the corresponding moving image may be specified, and a position in the specified moving image may be determined.
[0098] Referring to FIG. 7 again, the user interface module 264 temporally associates and outputs the information collected from the control target (the time-series data 142 or the feature amount calculated from the time-series data 142) and the captured image 342, and receives a label specification. Typically, the user interface module 264 presents a user interface screen including the time-series data 142 (or the feature amount calculated from the time-series data 142) and the image 342 to the user, and receives a label specification by the user.
[0099] The user interface module 264 includes a label setting support module 265. The label setting support module 265 is in charge of processing for supporting the label specification by the user. An example of the processing for supporting the label specification will be described later.
[0100] The learning data generation module 266 generates learning data 146 by attaching the label specified by the user to the information collected from the control target (the time-series data 142 or the feature amount calculated from the time-series data 142).
[0101] <D. Example of User Interface Screen> Next, an example of a user interface screen related to the labeling process 260 in the control system 1 according to the present embodiment will be described.
[0102] Fig. 9 is a schematic diagram showing an example of a user interface screen provided in the control system 1 according to the present embodiment. Referring to Fig. 9, a user interface screen 270 includes a feature amount display area 271, a histogram display area 272, and an image display area 276.
[0103] Feature amount display area 271 displays feature amounts calculated from time-series data 142 for each predetermined time interval called a frame. The horizontal axis of feature amount display area 271 is the frame number, and the vertical axis is the feature amount. Threshold value displays 273 and 274 showing two threshold values, respectively, are displayed in feature amount display area 271.
[0104] The histogram display area 272 displays a histogram, which is a distribution of the magnitudes of all or part of the features displayed in the feature display area 271 .
[0105] The image display area 276 displays images 342-1 to 342-5 corresponding to the collection timing of the feature included in the selection area 275 that is arbitrarily set in the feature display area 271. The user interface module 264 of the support device 200 outputs an image 342 corresponding to information selected from the information (time-series data 142, or feature calculated from the time-series data 142) collected from the control target. In the example shown in Fig. 9, the displayed images 342-1 to 342-5 each correspond to one frame.
[0106] In cases where there are too many images 342 included in the selection area 275 to all be displayed in the image display area 276, the images 342 displayed in the image display area 276 may be changed sequentially in response to pressing of the back button 278 and the forward button 279.
[0107] As an example, an image 342 of a workpiece produced by the equipment to be controlled is displayed in Fig. 9 (the same applies to the following figures). A scratch 344 has occurred on the workpiece included in image 342-4 shown in Fig. 9.
[0108] The user performs a labeling operation on the user interface screen 270. Fig. 10 is a schematic diagram showing an example of a labeling process on a user interface screen provided in the control system 1 according to the present embodiment. Referring to Fig. 10, the user selects one of the images 342 displayed in the image display area 276 and assigns a label to it. More specifically, when the user selects one of the images 342, a labeling window 280 is displayed on the image 342.
[0109] The labeling window 280 displays three labels: "Normal", "Abnormal", and "Ignore" (for example, an outlier that is excluded from the learning data 146). The user selects one of the labels while viewing the image 342 (selection display 282). Note that the types of labels are not limited to three, and may be more or less than three.
[0110] 10, an image 342-4 in which a scratch 344 has occurred on the workpiece is labeled as "abnormal." On the other hand, an image 342 in which a scratch 344 has not occurred on the workpiece is labeled as "normal." At this time, a label assigned to one of the images 342 may be reflected in the other images 342 based on the similarity of the images 342.
[0111] 11 and 12 are schematic diagrams showing an example of a processing procedure for reflecting one label on another on a user interface screen provided in the control system 1 according to this embodiment.
[0112] 11, for example, the user selects image 342-1 displayed in image display area 276 and assigns a label of "normal." When the user then instructs reflection to similar image 342, support device 200 searches for images 342 similar to image 342-1 among images 342 to which no label has been assigned, and reflects the same label (in this example, "normal") to the searched image 342.
[0113] 12, image 342-1 is determined to be similar to images 342-2, 342-3, and 342-5, and the label of "normal" assigned to image 342-1 is also reflected in images 342-2, 342-3, and 342-5. In this manner, the user interface module 264 (label setting assistance module 265) of the support device 200 may assign, to information (time-series data 142 or feature amounts calculated from the time-series data 142) corresponding to each of a plurality of similar images, a label that is the same as a label designated for information corresponding to any one of the plurality of images.
[0114] By adopting such a process of reflecting labels based on image similarity, the burden on the user in assigning labels can be reduced.
[0115] 13 and 14 are schematic diagrams showing an example of a labeling process procedure when no image is present on the user interface screen provided in the control system 1 according to the present embodiment.
[0116] 13, depending on the situation, there may be cases where an image 342 corresponding to the timing of collecting the feature does not exist or cannot be acquired. In the example shown in Fig. 13, only images 342-1 and 342-2 are acquired, and the remaining images 342 cannot be acquired. Even in such cases, it is possible to assign a label.
[0117] 14, the user can specify a label based on the relationship between each feature and a threshold value by referring to the histogram displayed in the histogram display area 272. In the example shown in Fig. 14, the label "normal" is assigned to all feature amounts that do not have a corresponding image 342.
[0118] In the control system 1 according to the present embodiment, the label is assigned to the time-series data 142 (or to features calculated from the time-series data 142), so that the label assignment process can be performed even if the image 342 does not exist.
[0119] FIG. 15 is a schematic diagram showing an example of a processing procedure for attaching a label to a unique feature amount on a user interface screen provided in the control system 1 according to this embodiment.
[0120] Referring to FIG. 15, since the selection region 275 can be set arbitrarily, for example, the selection region 275 may be set so as to include one or more feature quantities that exhibit values significantly different from normal values, and the feature quantities included in the set selection region 275 may be labeled as “abnormal.”
[0121] In this case, after assigning a label of "abnormal" to one image 342, the same label may be reflected to other images 342, or the label of "abnormal" may be assigned in a state where multiple images 342 are selected. In this manner, the user interface module 264 (label setting assistance module 265) of the support device 200 may reflect the same designated label to multiple pieces of information collected from the control target.
[0122] Furthermore, as described with reference to FIGS. 11 and 12, a label given to one of the images 342 may be reflected in the other images 342 based on the similarity between the images 342.
[0123] FIG. 16 is a schematic diagram showing an example of displaying a video on a user interface screen provided in the control system 1 according to the present embodiment. Referring to FIG. 16, a video is displayed as an image 342 in the image display area 276 of the user interface screen 270.
[0124] The image 342, which is the video displayed in the image display area 276, is played and stopped in response to the pressing of the play button 287 and the stop button 288. The position indicator 286 in the feature amount display area 271 moves corresponding to the position during the playback of the video. Note that the user may be able to operate the position indicator 286. In this case, the position of the video corresponding to the position of the position indicator 286 is played or displayed.
[0125] Thus, when the user interface module 264 of the support device 200 employs a video as the image 342, it may display the position of the corresponding information among the information collected from the control target in accordance with the playback of the video.
[0126] The labeling window 280 is displayed on the image 342 by a predetermined user operation. The user selects one of the labels while viewing the image 342 corresponding to the position indicator 286.
[0127] Thus, even when a video is used as the image 342, the user can easily assign labels in the same manner as when a still image is used.
[0128] <E. Processing Procedure> Next, the processing procedure of the labeling process 260 in the control system 1 according to the present embodiment will be described.
[0129] FIG. 17 is a flowchart showing the processing procedure of the labeling process 260 in the control system 1 according to the present embodiment. The main steps shown in FIG. 17 may typically be realized by the processor 202 of the support device 200 executing the model generation program 230.
[0130] Referring to FIG. 17, as a preliminary preparation, the user positions the camera 330 of the image collecting unit 300 so that an object related to the target of anomaly detection is included in the field of view (step S2).
[0131] The control device 100 sequentially stores the time series data 142 in the TSDB 140 (step S4), and the image collection unit 300 sequentially stores the images captured by the camera 330 in the DB 340 (step S6).
[0132] In parallel with the operation of the control device 100 and the image collecting section 300, or after the operation of the control device 100 and the image collecting section 300, the support device 200 acquires the time series data 142 from the control device 100 (step S10) and acquires the image 342 from the image collecting section 300 (step S12). In this way, the support device 200 executes a process of acquiring information collected from the control target and a process of acquiring an image of an object related to the target of anomaly detection.
[0133] The support device 200 calculates a feature amount for each frame from the acquired time series data 142 (step S14). Furthermore, the support device 200 associates the feature amount for each frame calculated from the time series data 142 with the image 342 based on the timestamp added to the acquired time series data 142 and the timestamp added to the acquired image 342 (step S16). Then, the support device 200 displays the feature amount for each frame calculated from the time series data 142 (step S18). In this way, the support device 200 outputs the information collected from the control target and the captured image 342 in a time-correlated manner.
[0134] When the selection area 275 is set for the displayed feature amount (YES in step S20), the support device 200 displays the image 342 corresponding to the feature amount included in the selection area 275 (step S22).
[0135] When any of the images 342 is selected (YES in step S24), the support device 200 displays the labeling window 280 in association with the selected image 342 (step S26). When the user selects a label on the labeling window 280 (YES in step S28), the support device 200 associates the selected label with the time-series data 142 of the corresponding frame (or with a feature amount calculated from the time-series data 142) (step S30). This association generates the learning data 146. In this way, the support device 200 executes a process of receiving a label designation and a process of generating the learning data 146 for constructing the model 160 by adding the designated label to information collected from the control target.
[0136] The support device 200 judges whether or not reflection based on the similarity of the image 342 has been instructed (step S32). When reflection based on the similarity of the image 342 has been instructed (YES in step S32), the support device 200 searches for an image 342 similar to the previously selected image 342 (step S34), and assigns to the searched image 342 the same label as that assigned to the previously selected image 342 (step S36).
[0137] If reflection based on the similarity of image 342 is not instructed (NO in step S32), the processes of steps S34 and S36 are skipped.
[0138] The support device 200 determines whether or not another image 342 has been selected (step S38). If another image 342 has been selected (YES in step S38), the processes in and after step S26 are repeated.
[0139] The support device 200 determines whether or not another selection area 275 has been set (step S40). If another selection area 275 has been set (YES in step S40), the processing from step S22 onwards is repeated.
[0140] The support device 200 determines whether an instruction to end the labeling process has been given (step S42). When an instruction to end the labeling process is given (YES in step S42), the labeling process ends.
[0141] After the labeling process, the support device 200 performs the generation process of the model 160 using a plurality of pieces of learning data 146 generated in step S30.
[0142] <F. Modification Example> In the above description, a configuration in which the labeling process 260 is included in the model generation tool 250 has been illustrated. However, the labeling process 260 may be implemented alone or may be included in the data mining tool 240. Further, the labeling process 260 may be implemented in a device different from the support device 200 (for example, computing resources on the cloud).
[0143] That is, the hardware implementing the labeling process 260 according to the present embodiment may be any type.
[0144] In the above description, as a typical example, a method in which a user designates a label for the feature amount calculated for each frame has been illustrated. However, a label may be designated for the time-series data 142 for each frame. Also, regarding the generated learning data 146, it may be the time-series data 142 for each frame to which a label is assigned, or it may be the feature amount for each frame to which a label is assigned.
[0145] In the above description, as a form of temporally associating and outputting the information collected from the control target and the captured image 342, the user interface screen 270 on the display has been illustrated as an example. However, the present invention is not limited to this, and any output form can be adopted.
[0146] <G. Supplementary Note> The present embodiment as described above includes the following technical ideas. [Configuration 1] A control calculation unit (130) that executes control calculations for controlling a control target; an anomaly detection unit (150) that detects an anomaly in the control target by referring to a model (160) based on information (152) collected from the control target; an image acquisition unit (300) that acquires an image (342) of an object related to an anomaly detection target; a user interface unit (264) that outputs the information collected from the control target and the captured image in a time-correlated manner and receives a label designation; and a generation unit (266) that generates learning data (146) for constructing the model by adding a specified label to information collected from the control target. [Configuration 2] 2. The control system according to claim 1, wherein the user interface unit outputs an image corresponding to information selected from the information collected from the control target. [Configuration 3] The control system according to configuration 1 or 2, wherein the user interface unit (264, 265) reflects, for information corresponding to each of a plurality of similar images, a label that is the same as a label specified for information corresponding to any one of the plurality of images. [Configuration 4] 4. The control system according to any one of configurations 1 to 3, wherein the user interface unit (264, 265) reflects the same designated label on a plurality of pieces of information collected from the control target. [Configuration 5] 5. The control system according to any one of configurations 1 to 4, wherein the image includes at least one of a still image and a moving image. [Configuration 6] The control system according to configuration 5, wherein, when the image includes a video, the user interface unit displays a position of corresponding information among the information collected from the control target in conjunction with playback of the video. [Configuration 7] The control system according to any one of configurations 1 to 6, further comprising a model generation unit (252, 254) that generates the model based on the learning data. [Configuration 8] The control system according to any one of configurations 1 to 7, further comprising a feature amount calculation unit (267) that calculates a feature amount from a state value of the control target as information collected from the control target. [Configuration 9] A support device (200) connected to a control device (100) including a control calculation unit (130) that executes control calculations for controlling a control target, and an abnormality detection unit (150) that detects an abnormality in the control target by referring to a model (160) based on information (152) collected from the control target, An image acquisition unit (262) that acquires an image (342) of an object related to a target for anomaly detection; a user interface unit (264) that outputs the information collected from the control target and the captured image in a time-correlated manner and receives a label designation; and a generation unit (266) that generates learning data (146) for constructing the model by adding a specified label to information collected from the control target. [Configuration 10] A labeling method for a control system (1) that detects an anomaly in a control object by referring to a model (160) based on information (152) collected from the control object, comprising: A step (S10) of acquiring information collected from a control target; A step (S12) of acquiring an image of an object related to the target of anomaly detection; Steps (S16, S18) of outputting the information collected from the control target and the captured image in a time-correlated manner; A step (S24) of accepting a label designation; and generating learning data for constructing the model by adding a specified label to information collected from the control target (S30).
[0147] <Advantages> In the control system according to the present embodiment, the user can attach a label while viewing the captured image, so that the labeling for generating learning data can be efficiently performed. In addition, since it is possible to support the labeling work using the similarity of images, the efficiency can be further improved. Further, the camera for capturing an image only needs to be able to acquire an image necessary for generating learning data, so it can be arbitrarily arranged ad hoc and can flexibly respond to restrictions specific to the control target.
[0148] As a result, it is possible to perform labeling and acquire learning data suitable for operation in an actual production site.
[0149] The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is shown not by the above description but by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included.
Explanation of Signs
[0150] 1 control system, 2, 4 field bus, 6 upper network, 10 field devices, 12 remote I / O devices, 14 relays, 16, 124 I / O unit, 18 servo driver, 20 servo motor, 30 data collection phase, 32 data analysis phase, 34 operation phase, 100 control device, 102, 202, 302 processor, 104 chip set, 106, 206, 306 main memory device, 108, 208, 308 secondary memory device, 110 upper network controller, 112, 212 USB controller, 114 memory card interface, 116 memory card, 118, 120 field bus controller, 122 internal bus controller, 130 PLC engine, 131, 324 system program, 132 user program, 134 library, 136 data management unit, 138, 338 time management unit, 142 Time series data, 144 Label, 146 Learning data, 150 Anomaly detection engine, 152 Features, 160 Model, 200 Support device, 204 Optical drive, 205 Recording medium, 214, 314 Network controller, 216, 316 Input unit, 218, 318 Display unit, 220, 320 Bus, 222,322 OS, 224 interface program, 226 development program, 228 data mining program, 230 model generation program, 240 data mining tool, 242 feature generation process, 244 feature selection process, 246 model generation process, 248 setting process, 250 model generation tool, 252 model update process, 254 adjustment process, 260 label assignment process, 261 acquisition module, 262 image acquisition module, 263 module, 264 user interface module, 265 label setting support module, 266 learning data generation module, 267 feature calculation module, 270 user interface screen, 271 feature display area, 272 histogram display area, 273 display, 275 selection area, 276 image display area, 278 back button, 279 forward button, 280 label assignment window, 282 selection display, 286 Position indicator, 287 play button, 288 stop button, 300 image acquisition unit, 312 camera controller, 330 camera, 342 image, 344 scratch, 400 server, 500 display device.
Claims
1. a control calculation unit that executes a control calculation for controlling a control target; an anomaly detection unit that detects an anomaly in the control target by referring to a model based on information collected from the control target; an image collecting unit that collects images of objects related to the target of anomaly detection; a user interface unit that displays and outputs the information collected from the control target and the captured image in a time-correlated manner and receives a label designation; A generation unit that generates learning data for constructing the model by adding a specified label to information collected from the control object.
2. The control system according to claim 1 , wherein the user interface unit outputs an image corresponding to information selected from the information collected from the controlled object.
3. The control system according to claim 1 or 2, wherein the user interface unit reflects, for information corresponding to each of a plurality of similar images, a label that is the same as a label designated for information corresponding to any one of the plurality of images.
4. 4. The control system according to claim 1, wherein the user interface unit reflects the same designated label for a plurality of pieces of information collected from the controlled object.
5. The control system according to any one of claims 1 to 4, wherein the image includes at least one of a still image and a video image.
6. The control system according to claim 5 , wherein, when the image includes a moving image, the user interface unit displays a position of corresponding information among the information collected from the controlled object in conjunction with playback of the moving image.
7. The control system according to claim 1 , further comprising a model generation unit that generates the model based on the learning data.
8. The control system according to claim 1 , further comprising a feature amount calculation unit that calculates a feature amount from a state value of the control target as information collected from the control target.
9. A support device connected to a control device including a control calculation unit that executes a control calculation for controlling a control object, and an anomaly detection unit that detects an anomaly in the control object by referring to a model based on information collected from the control object, an image acquisition unit that acquires an image of an object related to a target for anomaly detection; a user interface unit that displays and outputs the information collected from the control target and the captured image in a time-correlated manner and receives a label designation; and a generation unit that generates learning data for constructing the model by adding a specified label to information collected from the control target.
10. A labeling method for a control system that detects an anomaly in a control object by referring to a model based on information collected from the control object, comprising: Obtaining information collected from a control target; acquiring an image of an object related to a target for anomaly detection; a step of displaying and outputting the information collected from the control target and the captured image in a time-correlated manner; accepting a label specification; and generating learning data for constructing the model by adding a specified label to information collected from the control target.
Citation Information
Patent Citations
Method of producing transparent conductive sheet
JP1986064020A
Semi-automatic labeling of datasets
JP2018537798A
Control system and control device
JP2020086481A
Control device and program
JP2020101904A
