Control System

Abstract

To manage a time lag between control data of an object of control and an image taken by an imaging unit.SOLUTION: A control system includes a control unit that controls an object of control, and an imaging unit that covers an imaging range within which the object of control can be imaged as a subject. When detecting an event, the control unit transmits an imaging initiation command to the imaging unit. A data acquisition unit acquires control data in association with an acquisition time. The imaging unit initiates imaging in response to the imaging initiation command sent from the control unit. An image acquisition unit acquires the taken image in association with an imaging time. The control system manages a time lag between the control data acquired by the data acquisition unit and the image acquired by the image acquisition unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present technology relates to a control system that controls a control target, and more particularly to a control system that collects control data related to control. [Background technology]

[0002] In various production sites where FA (Factory Automation) is applied, there is a need to improve the equipment operation rate by predictive maintenance of machines and equipment as production facilities. Predictive maintenance means a form of maintenance in which any abnormality occurring in machines and equipment is detected and maintenance work such as repair or replacement is performed before the equipment reaches a state where it should be stopped. In order to realize such predictive maintenance, a system has been put into practical use that uses data collected from machines and equipment during operation to support users in determining whether any abnormality has occurred in the machines and equipment.

[0003] For example, Japanese Patent Laid-Open Publication No. 2012-99972 (Patent Document 1) discloses an image processing device that displays waveform data based on an output signal of an object and a moving image captured by a camera on the same screen. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-99972 A Summary of the Invention [Problem to be solved by the invention]

[0005] There is a delay of at least the time it takes to send the command from when a command to start capturing images is sent to the camera until the camera actually starts capturing images. Therefore, when a video is played back, the video is displayed with a delay from the waveform display time by a time based on this delay, so it is not appropriate to use such display information as support information for the above-mentioned predictive maintenance. JP2012-99972A (Patent Document 1) does not disclose a configuration for managing the time delay between the waveform data and the video captured by the camera.

[0006] One object of the present disclosure is to provide an environment capable of managing a time lag between control data relating to control of a control target and an image captured by an imaging unit. [Means for solving the problem]

[0007] The control system according to the present disclosure includes a control unit that controls a control object, and an imaging unit having an imaging range capable of imaging the control object as a subject, the control unit includes a communication unit that sends an imaging start command to the imaging unit when an event is detected, and a data collection unit that collects control data related to control in association with collection time, the imaging unit includes an image collection unit that starts imaging in accordance with the imaging start command transferred from the control unit and collects the captured images in association with the imaging time, and the control system further includes a time management unit that manages time information indicating a time lag between the control data collected by the data collection unit and the images collected by the image collection unit.

[0008] According to this disclosure, it is possible to manage the time lag between control data collected in association with a collection time and a captured image of a control target that is associated with an image capturing time.

[0009] In the above disclosure, the time lag includes the time from when the control unit transmits an image capture start command to the imaging unit until when the control unit receives a response to the image capture start from the imaging unit.

[0010] According to this disclosure, the time lag can be obtained as the time from the transmission of an imaging start command to the reception of a response thereto.

[0011] In the above disclosure, the system further includes an information management unit that manages, in association with each other, the control data associated with the collection time collected by the data collection unit, the images associated with the imaging time collected by the image collection unit, and time information indicating a time lag.

[0012] According to this disclosure, the information management section can manage the control data associated with the acquisition time, the image associated with the imaging time, and the time information indicating the time lag between them in association with each other.

[0013] In the above disclosure, the control unit has a first connector that is connected to a network to which the information management unit and the imaging unit belong, and communicates imaging start commands and responses with the imaging unit via the network.

[0014] According to this disclosure, the time lag can be obtained as the time from when the control unit transmits an image capture start command via a network to when the control unit receives a response thereto via the network.

[0015] In the above disclosure, the control unit communicates an imaging start command and a response with the imaging unit via a signal line.

[0016] According to this disclosure, the time lag can be obtained as the time from when the control unit transmits an image capture start command via a signal destination to when the control unit receives a response thereto via a signal line.

[0017] In the above disclosure, each time an event is detected, a time lag is detected. According to this disclosure, the time lag can be acquired each time an event is detected. Therefore, even if the time required for communication of the imaging start command and the response may vary due to a change in the load on the network or the processing load on the imaging unit, the time lag can be acquired taking the load change into consideration.

[0018] In the above disclosure, the collection time associated with the control data collected by the data collection unit includes a collection start time, the imaging time associated with the captured image collected by the image collection unit includes an imaging start time, and the time lag includes a time difference between the collection start time and the imaging start time.

[0019] According to this disclosure, the time lag can be obtained from the time difference between the start time of collecting control data and the start time of capturing an image.

[0020] In the above disclosure, the event includes a predetermined event related to the control of a control target.

[0021] According to this disclosure, each time a control-related event is detected, it is possible to obtain control data associated with the acquisition time, an image associated with the imaging time, and time information indicating the time lag between the two.

[0022] In the above disclosure, the control unit periodically controls the controlled object, and the predetermined event includes detection of the passage of a predetermined period related to the control.

[0023] According to this disclosure, each time the passage of a predetermined period related to control is detected, control data associated with the collection time, an image associated with the imaging time, and time information indicating the time lag between the two can be obtained.

[0024] In the above disclosure, the device further includes a UI (user interface) unit, and the control unit periodically controls the control object. The UI unit has a waveform display unit that converts a plurality of control data collected by the data collection unit into a waveform based on the collection time associated with each of the plurality of control data and displays the converted waveform, and an image playback unit that plays back images collected by the image collection unit in accordance with the imaging time associated with the images.

[0025] According to this disclosure, the UI unit can present the control data to the user as a waveform converted based on the associated collection time, and can also play back the collected images according to the imaging time associated with the images and present them to the user.

[0026] In the above disclosure, the image includes a video composed of a plurality of frame images, each of which is associated with an imaging time, and the UI unit has an image adjustment unit that identifies and plays back a frame image from the plurality of frame images, the frame image having an imaging time associated with a user-specified point on the displayed waveform, based on the collection time associated with control data corresponding to the user-specified point and the time lag.

[0027] According to this disclosure, when a user specifies a point on a waveform, the UI unit obtains the imaging time corresponding to the specified point from the collection time and time shift of the specified point, plays back the image associated with this imaging time, and presents it to the user.

[0028] In the above disclosure, the imaging unit has an imaging range capable of imaging the control object as a subject, and includes a plurality of imaging units having different imaging directions for the control object, and the time management unit manages, for each of the plurality of imaging units, time information indicating a time lag between the control data collected by the data collection unit and the image collected by the image collection unit of the imaging unit.

[0029] According to this disclosure, the control system can include a plurality of imaging units that capture images of a control target from different imaging directions. In this case, for each imaging unit, it is possible to manage a time lag between control data collected in association with a collection time and an image captured by the imaging unit of the control target and associated with the capture time.

[0030] In the above disclosure, for each imaging unit, the image adjustment unit identifies and plays back a frame image, the imaging time of which corresponds to the user-specified point, from among the multiple frame images constituting the image captured by the imaging unit, based on the collection time associated with the control data corresponding to the user-specified point and the time shift indicated by the time information of the imaging unit.

[0031] According to this disclosure, when a user specifies a point on the waveform, the image adjustment unit identifies, for each imaging unit, an image associated with the imaging time corresponding to the specified point, i.e., an imaging time taking into account the time lag, from among the multiple frame images captured by that imaging unit, and plays it back.

[0032] In the above disclosure, when the UI unit receives a user operation during playback by the image playback unit, it identifies a playback position in multiple frame images corresponding to the time of reception, and stores information about a specified section based on the amount of user operation for an image starting from the identified playback position in association with the image.

[0033] According to this disclosure, when the UI unit receives a user operation while playing an image, it can store information about frames corresponding to a specified section starting from a playback position identified based on the received user operation, in association with the image.

[0034] In the above disclosure, when playing back an image, the image adjustment section adjusts the playback speed of the image based on information of a predetermined section associated with the image.

[0035] According to this disclosure, when playing back an image, the image adjustment section adjusts the playback speed of frames corresponding to a specific section of the image based on information about the specific section designated by the user, and plays back the image. Effect of the Invention

[0036] According to the present disclosure, control data associated with a collection time is collected, and captured images of a control object that are associated with the capture time are collected, making it possible to manage the time lag between the collected control data and the captured images. [Brief description of the drawings]

[0037] [Figure 1] FIG. 1 is a diagram illustrating a schematic diagram of an overall configuration of a control system 1a according to the present embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a schematic diagram of an overall configuration of a control system 1b according to the present embodiment. [Diagram 3] 1 is a block diagram showing an example of a hardware configuration of a PLC 100 according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of a server 200 according to the present embodiment. [Diagram 5] FIG. 5 is a diagram illustrating a DB manager provided by executing the DB management program 211 in FIG. 4. [Figure 6] FIG. 2 is a block diagram showing an example of a hardware configuration of a terminal 300 according to the present embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of the configuration of an imaging device 40 according to the present embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a module configuration of a UI tool 340 provided for monitor processing according to the present embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a sequence of a collection process according to the present embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a time-series DB 250 according to the present embodiment. [Figure 11] 10A and 10B are diagrams illustrating the relationship between deviation and display according to the present embodiment. [Figure 12] 10A and 10B are diagrams illustrating the relationship between deviation and display according to the present embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of a display according to the present embodiment. [Figure 14]It is a diagram schematically showing an example of display according to this embodiment. [Figure 15] It is a diagram schematically showing an example of display according to this embodiment. [Figure 16] It is a diagram schematically showing another example of display according to this embodiment. [Figure 17] It is a diagram showing an example of a flowchart of monitor processing according to this embodiment.

Embodiments for Carrying Out the Invention

[0038] Embodiments of the present invention will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and the description thereof will not be repeated.

[0039] <A. Application Example> First, an example of a scene to which the present invention is applied will be described.

[0040] FIG. 1 is a diagram schematically showing the overall configuration of a control system 1a according to this embodiment. Referring to FIG. 1, the control system 1a is applied to FA and includes a PLC system 2, a server 200 which is an example of an “information management unit”, and a terminal 300 responsible for the development and maintenance of the control system 1a.

[0041] The PLC system 2, also referred to as a line, may control the entire FA factory or may control specific manufacturing equipment or manufacturing devices installed in the factory. The PLC system 2 mainly includes a PLC (Programmable Logic Controller) 100 which is a typical example of a “control unit” for controlling a control target, and an imaging device 40 which is an example of an “imaging unit”. The imaging device 40 can have its angle or optical system adjusted so as to have an imaging range capable of imaging the control target as a subject. A lighting device may be provided in relation to the imaging device 40.

[0042] The PLC 100 is connected to a field device 10. The field device 10 includes any device necessary to control a controlled object. More specifically, the field device 10 includes a device for exchanging information with a controlled object (e.g., a manufacturing facility, a manufacturing device, a sensor and an actuator included in the manufacturing facility or the manufacturing device, etc.). In the example shown in FIG. 1, the field device 10 includes a relay group 14 and a servo motor 18. The field device 10 may further include any device, such as a servo driver 16 and a remote I / O (Input / Output) 12 shown in FIG. 1.

[0043] The PLC 100 acquires data from the field device 10 and executes control calculations according to a user program created in advance to generate data to be provided to the field device 10. Hereinafter, the data acquired by the PLC 100 from the field device 10 will also be referred to as "input data", and the information provided to the field device 10 will also be referred to as "output data". The input data and output data will also be collectively referred to as "input / output data" or "I / O data". The input data includes a value indicating the state of the field device 10 (e.g., an output value of a sensor), and the output data may include a command value for controlling the field device 10 based on the result of the control calculation.

[0044] In the example shown in FIG. 1, the PLC 100 and the field device 10 are connected via a control network 4, but the present invention is not limited to this and may be connected by hard wires.

[0045] It is preferable to adopt an industrial communication protocol for the control network 4. Known examples of such communication protocols include EtherCAT (registered trademark), EtherNet / IP (registered trademark), DeviceNet (registered trademark), and CompoNet (registered trademark).

[0046] The PLC system 2 is capable of accessing a cloud-based server 200 present on the Internet 8 via an information network 6 and a relay device 20 such as a gateway or a router. The information network 6 may be adapted to, for example, the EtherNet / IP (registered trademark) protocol.

[0047] The server 200 communicates with the PLC 100 and the imaging device 40 via the information network 6 and the Internet 8. The server 200 mainly manages information related to the PLC system 2 and provides various services.

[0048] The server 200 may be placed on a cloud via the Internet 8, or may be placed in a local network where the PLC system 2 exists without going through the Internet 8. The implementation form of the server 200 can be determined arbitrarily depending on the required performance, functions, and the like.

[0049] The terminal 300 is configured, for example, from a general-purpose computer. The terminal 300 can access the server 200 and perform user operations to realize various processes as described below. In this embodiment, this user operation may be realized by an application 311 and a UI program 313 of the terminal 300, which will be described later, working together to provide a GUI (Graphical User Interface) for operating the desktop environment of the server 200 connected from the terminal 300 via the Internet 8. The terminal 300 may also exchange information with the PLC 100 through USB (Universal Serial Bus) communication, without going through the information network 6 and the Internet 8. If the PLC 100 also has the configuration of a general-purpose computer, the PLC 100 may provide services such as a UI (User Interface), GUI, etc., provided by the terminal 300.

[0050] The PLC 100 executes a control calculation to calculate a control command, and outputs the calculated control command to the control object, thereby controlling the control object. The PLC 100 periodically controls the control object according to a predetermined control period. The PLC 100 has a data collection unit 118a that collects data related to the control in the process of the periodic control in association with a collection time. The data collected by the data collection unit 118a includes the control command and a state value indicating the state of the control object. The PLC 100 executes a control calculation using the collected state value, and calculates a control command.

[0051] When a predetermined event is detected, the PLC 100 transmits an image capture start command 1 to the image capture device 40 (step S1). The predetermined event may include detection of an abnormality or a sign of an abnormality, or detection of the passage of a predetermined period. The predetermined period includes, for example, a period based on a control period.

[0052] When the imaging device 40 receives the imaging start command 1 from the PLC 100, it starts imaging in accordance with the received imaging start command 1 (step S5a) and transmits a response 21 to the imaging start command 1 to the PLC 100 (step S2a). The imaging device 40 can transmit the response 21 before or after starting imaging. The imaging device 40 has an image collection unit 46 that starts imaging and collects images captured after the imaging start in association with the imaging time.

[0053] The PLC 100 has a time management unit 119a that measures the time required from transmitting the imaging start command 1 to receiving the imaging start response 21 from the imaging device 40, and manages time information 110 based on the measurement. This time information 110 indicates a delay in imaging timing of the imaging device 40. This deviation corresponds to the time required from transmitting the imaging start command 1 by the PLC 100 to the imaging device 40 starting imaging in accordance with the imaging start command 1. More specifically, the deviation indicates a delay in time from the collection time of control data, which is data related to control and collected by the data collection unit 118a when the imaging start command 1 is output, to the imaging time of the image collected by the image collection unit 46 after imaging is started in accordance with the imaging start command 1. This deviation dynamically changes depending on the load related to communication between the PLC 100 and the imaging device 40 (including the communication load such as traffic of the information system network 6), the load related to processing of the imaging device 40, and the like.

[0054] Each time the PLC 100 transmits the imaging start command 1, it detects a time delay as a deviation by measuring the time until it receives the response 21 with a timer (step S3), and the time management unit 119a manages the time information 110 of the detected deviation (step S4). In this way, by detecting a time deviation each time an event is detected, in other words, each time the imaging start command 1 is transmitted, even if there is a dynamic change in the load as described above, it is possible to detect the latest deviation by following the change.

[0055] This allows the PLC 100 to manage the time lag between the control data collected in relation to the control of the control target and the captured image of the imaging device 40 by the time management unit 119a.

[0056] The imaging device 40 transmits the captured image associated with the imaging time to the server 200 (step S5b). The PLC 100 transmits the control data associated with the collection time to the server 200 (step S6) and transmits the time difference information 110 to the server 200 (step S6a).

[0057] The server 200 manages the control data transferred from the PLC 100 and associated with the collection time, and the captured image transferred from the imaging device 40 and associated with the imaging time, by storing them in the storage as temporally associated association information (step S14). As a result, the control data and the captured image are stored in a time-series searchable manner, and are managed in a temporally associated manner based on the collection time and the imaging time. The server 200 also links, i.e., associates, the time information 110 transferred from the PLC 100 indicating the difference with the above-mentioned association information stored in the storage (step S15).

[0058] This allows the control system 1a to manage, in the storage of the server 200, control data relating to control collected in relation to the control of a control target and captured images of the imaging device by associating the time lag between the two.

[0059] More specific application examples of the present invention will now be described. FIG. 2 is a diagram showing a schematic diagram of the overall configuration of a control system 1b according to the present embodiment. The control system 1b in FIG. 2 has a plurality of PLC systems 2 and imaging devices 40 shown in FIG. 1. The control system 1b in FIG. 2 has PLC systems 2-1, 2-2, and 2-3 (hereinafter also collectively referred to as "PLC systems 2") provided in different manufacturing processes, and imaging devices 40-1 and 40-2 provided in these manufacturing processes. The control system 1b realizes distributed control of a series of processes, for example, by providing the PLC system 2-1 in a parts process, the PLC system 2-2 in an assembly process, and the PLC system 2-3 in an inspection process. The other configuration of the control system 1b is similar to that of the control system 1a in FIG. 1, and therefore the description will not be repeated. The imaging devices 40-1 and 40-2 capture images of the same control target (subject) with different camera orientations (imaging directions) by, for example, changing the camera angles.

[0060] In the following description, PLC systems 2-1, 2-2, and 2-3 have the same configuration as the PLC system 2 shown in FIG. 1. Also, imaging devices 40-1 and 40-2 have the same configuration as the imaging device 40 in FIG. 1. Here, three PLC systems 2 and two imaging devices 40 are shown, but the number of PLC systems 2 and imaging devices 40 is not limited to these. In the following, in the description of functions or configurations common to PLC systems 2-1, 2-2, and 2-3, they are collectively referred to as "PLC system 2", and in the description of functions or configurations common to imaging devices 40-1 and 40-2, they are collectively referred to as "imaging device 40".

[0061] <B. Configuration of Each Device> Next, an example of the hardware configuration of the devices constituting the control system according to the present embodiment will be described.

[0062] (b1:PLC100) FIG. 3 is a block diagram showing an example of the hardware configuration of PLC100 according to the present embodiment. Referring to FIG. 4, PLC100 includes a processor 102 such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit), a chipset 104, a main memory 106, a storage 109, an information network controller 120 connected to a connector 121 composed of physical circuits, a control network controller 122 connected to a connector 123 composed of physical circuits, a USB (Universal Serial Bus) controller 124, a timer 125 including a counter circuit, and a memory card interface 126. The connector 121 is connected to the information network 6 to which the server 200, PLC100, and imaging device 40 belong, and the connector 123 is connected to a control network 4 which is a network different from the information network 6 and to which the field device 10 to be controlled belongs.

[0063] The processor 102 reads out various programs stored in the storage 109, expands them in the main memory 106, and executes them to realize control calculations for controlling the controlled object and communication processing for communicating with external devices (the server 200, the terminal 300, the imaging device 40, and the field device 10). The chipset 104 controls data transmission between the processor 102 and each component.

[0064] The storage 109 stores an OS (Operating System) 111, a system program 113, and a user program 114. The system program 113 includes communication firmware for communicating with external devices including the server 200 via the information network 6, and communication firmware for communicating with the field device 10 via the control network 4.

[0065] The user program 114 includes a control program 115, a communication program 116, an IO (Input / Output) refresh 117, a collection program 118, and a camera communication program 119. The IO refresh 117 collects the state values ​​of the field device 10, which is the control target, via the control network 4 and stores them in the IO refresh area 112. The IO refresh 117 also transmits the control command stored in the IO refresh area 112 to the control target via the control network 4. The control program 115 performs a control operation of the control target using the state values ​​stored in the IO refresh area 112 to calculate the above control command. The collection program 118, when executed, realizes the module of the data collection unit 118a described above. The camera communication program 119, when executed, realizes the module of the time management unit 119a described above. The storage 109 stores the time information 110 acquired by the time management unit 119a and the collection control data 400 collected by the data collection unit 118a. The user programs 114 may include various other processing programs including a time synchronization program.

[0066] The time management unit 119a generates and stores time information 110 every time an event is detected and an imaging start command 1 is transmitted, and manages the time information 110. Referring to FIG. 3, the time information 110 includes an event timing ST indicating the time when the event is detected, a camera ID indicating an identifier of the imaging device 40, and a delay DT. The camera ID indicates an identifier of the imaging device 40 that has transferred a response 21 to the imaging start command 1, and the delay DT indicates a time difference between the time when the imaging start command 1 is transmitted to the imaging device 40 and the time when the response 21 is received from the imaging device 40. This time difference corresponds to a required time from the transmission of the imaging start command 1 to the reception of the response 21. The event timing ST and the delay DT are measured using a timer 125.

[0067] The PLC 100 of each PLC system 2 performs time synchronization processing to match the measured time of the timer 125 with that of the other PLCs 100 connected to the information network 6 according to the time synchronization program. Also, the PLC 100 of each PLC system 2 performs time synchronization processing to match the measured time of the timer of the field device 10 connected to the control network 4 according to the time synchronization program.

[0068] The information network controller 120 controls data exchange with external devices via the information network 6 connected to the connector 121 .

[0069] The control network controller 122 controls data exchange with the field device 10 via the control network 4 connected to the connector 123 .

[0070] The USB controller 124 controls data exchange with external devices (eg, a support device, the terminal 300, etc.) via a USB connection.

[0071] The memory card interface 126 is configured so that a memory card 128 can be freely attached and detached, and it is possible to write data to the memory card 128 and read various data (such as the user program 114 and data) from the memory card 128.

[0072] The PLC 100 may have an optical drive. The optical drive reads a computer-readable program from a recording medium (e.g., an optical recording medium such as a DVD (Digital Versatile Disc)) that non-transiently stores the program, and stores the program in the storage 109 or the like.

[0073] The various programs executed by the PLC 100 may be installed via a computer-readable recording medium or a memory card 128, or may be installed by downloading from a device such as an arbitrary computer on a network.

[0074] FIG. 3 shows an example of a configuration in which the processor 102 executes a program to provide the necessary processing, but some or all of the provided processing may be implemented using a dedicated hardware circuit (e.g., an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). Alternatively, the main part of the PLC 100 may be realized using hardware that conforms to a general-purpose architecture (e.g., an industrial PC based on a general-purpose PC). In this case, virtualization technology may be used to run multiple OSs with different uses in parallel, and necessary applications may be executed on each OS.

[0075] (b2: Server 200) Fig. 4 is a block diagram showing an example of a hardware configuration of a server 200 according to this embodiment. Referring to Fig. 4, the server 200 includes a processor 202, a main memory 204, an input unit 206, an output unit 208, a storage 210, an optical drive 215, a USB controller 220 for communicating with an external device, and a network controller 413 for connecting a network including the Internet 8. These components are connected via a processor bus 218.

[0076] The processor 202 is composed of a CPU, a GPU, etc., and realizes various processes to be described later by reading out a program stored in the storage 210, expanding it in the main memory 204, and executing it.

[0077] The main memory 204 is configured with a volatile storage device such as a DRAM or an SRAM, etc. The storage 210 is configured with a non-volatile storage device such as an HDD or an SSD, etc.

[0078] The storage 210 stores various programs for providing the functions of the server 200, in addition to an OS 212 for implementing basic functions. The various programs include a DB (Data Base) management program 211. The storage 210 also has an area for storing a time-series DB 250.

[0079] The input unit 206 is composed of a keyboard, a mouse, etc., and receives user operations for the server 200. The output unit 208 is composed of a display, various indicators, a printer, etc., and outputs processing results from the processor 202, etc.

[0080] The server 200 has an optical drive 215, and a program stored therein is read from a recording medium 214 (e.g., an optical recording medium such as a DVD (Digital Versatile Disc)) that non-transiently stores a computer-readable program, and is installed in the storage 210, etc.

[0081] FIG. 4 shows an example of a configuration in which the functions required for server 200 are provided by processor 202 executing a program, but some or all of these provided functions may be implemented using a dedicated hardware circuit (e.g., ASIC or FPGA).

[0082] Fig. 5 is a diagram illustrating a DB manager provided by executing the DB management program 211 in Fig. 4. Referring to Fig. 5, a DB manager 251 provided by executing the DB management program 211 by the processor 202 manages a time-series DB 250 (e.g., generating, aggregating, editing, analyzing, outputting, etc. a DB) in accordance with a database operation command conforming to, for example, SQL (Structured Query Language). Details of the time-series DB 250 will be described later.

[0083] (b3: Terminal 300) Fig. 6 is a block diagram showing an example of a hardware configuration of a terminal 300 according to this embodiment. Referring to Fig. 6, the terminal 300 includes a processor 302 such as a CPU or an MPU, an optical drive 304, a main memory 306, a storage 310, a network controller 320, a USB controller 324, an input unit 326, and a display unit 328. These components are connected via a bus 308.

[0084] The processor 302 reads out various programs stored in the storage 310, loads them into the main memory 306, and executes them to realize the processes required by the terminal 300.

[0085] The storage 310 is composed of, for example, a HDD or SSD. The storage 310 stores an OS 312, an application 311, and a UI (User Interface) program 313. The UI program 313 is executed in cooperation with the application 311 to provide a UI tool 340, which will be described later. The UI program 313 displays the collection control data 400 collected by the PLC 100 and the image collected by the imaging device 40 on the display of the display unit 328, either individually or in association with both, and accepts user operations on the displayed control data or image. The UI program 313 includes a waveform conversion program 314 that converts the control data into waveform data according to the collection time and outputs the data, an image playback program 315 that plays the image (still image, video) collected by the imaging device 40, a data adjustment program 316 that adjusts the displayed control data according to a user operation, an image adjustment program 317 that adjusts the displayed image according to a user operation, and an adjustment reflection program 318 that provides a tool for reflecting an adjustment amount based on the user operation in the image.

[0086] Note that the storage 310 may store necessary programs other than the programs shown in FIG.

[0087] The network controller 320 controls data exchange with devices such as the PLCs 100 and the server 200 via the information network 6 .

[0088] The USB controller 324 controls data transfer to and from external devices via a USB connection.

[0089] The input unit 326 is composed of a mouse, a keyboard, a touch panel, etc., and receives instructions from a user. The display unit 328 is composed of a display, various indicators, etc., and outputs processing results from the processor 302. The input unit 326 and the display unit 328 may form a touch screen integrated with a display.

[0090] The terminal 300 may have an optical drive 304. The optical drive 304 reads a computer-readable program from a recording medium 305 (for example, an optical recording medium such as a DVD (Digital Versatile Disc)) that non-transiently stores the program, and stores the program in the storage 310 or the like.

[0091] The various programs executed on the terminal 300 may be installed via a computer-readable recording medium 305, or may be installed by downloading from an arbitrary server on the network.

[0092] FIG. 6 shows an example of a configuration in which the necessary processing is provided by the processor 302 executing a program, but some or all of the provided processing may be implemented using a dedicated hardware circuit (e.g., an ASIC or an FPGA).

[0093] (b4: Imaging device) Fig. 7 is a diagram showing a schematic example of a configuration of an imaging device 40 according to the present embodiment. Referring to Fig. 7, imaging device 40 includes, as basic components, an optical system 41 including a lens, an imaging element 42, a DSP (Digital Signal Processor) 43 constituting a processor circuit having an image processing engine 44 and an image memory 47, and a communication I / F (Interface) 48 including a communication circuit connecting to an information network 6. Imaging device 40 integrally includes a processing system of DSP 43 and communication I / F 48, and an imaging system of optical system 41 and imaging element 42, but the processing system may be configured separately from the imaging system.

[0094] Reflected light from a subject in the imaging range is incident on a photoelectric conversion element, such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, of the imaging element 42 via the lens of the optical system 41, and is converted into an image signal by the image sensor. The image processing engine 44 inputs the image signal from the imaging element 42, processes the image signal based on the time output by the timer 45, and converts it into an image of digital data in units of frames. For example, the image is converted into N (N>1) frame images per second. The image processing engine 44 stores the frame images in the image memory 47 as still images or moving images in association with the imaging time clocked by the timer 45. The image memory 47 cooperates with the image processing engine 44 to form an image collection unit 46 that collects images in association with the imaging time clocked by the timer 45.

[0095] The communication I / F 48 receives the imaging start command 1 transferred from the PLC 100 via the information network 6, and outputs the received imaging start command 1 to the DSP 43. In response to the imaging start command 1, the DSP 43 controls the communication I / F 48 to start image processing of the image signal output from the imaging system, and to send a response 21 to the information network 6.

[0096] For example, in this embodiment, the imaging device 40 starts imaging in response to an imaging start command 1, and continues imaging for a predetermined period after the imaging start. The length of this period can be set. Images captured in units of frames after the imaging start are stored in the image memory 47 as, for example, a still image file or a moving image file.

[0097] (b5: UI tool configuration) FIG. 8 is a diagram showing an example of a module configuration of a UI tool 340 provided for monitor processing according to the present embodiment. The UI tool 340 is realized by the processor 302 of the terminal 300 executing a UI program 313. The UI tool 340 includes a waveform display module 341, an image reproduction module 342, a data adjustment module 343, an image adjustment module 344, and an adjustment reflection module 345, which are realized by the processor 302 executing a waveform conversion program 314, an image reproduction program 315, a data adjustment program 316, an image adjustment program 317, and an adjustment reflection program 318, respectively.

[0098] <C. Outline of the collection processing sequence> FIG. 9 is a diagram schematically showing an example of a sequence of collection processing according to the present embodiment. The sequence in FIG. 9 includes communication between devices including a PLC 100 related to collection of control data and captured images, an imaging device 40, and a server 200, and processing performed by each device.

[0099] The PLC 100 determines whether to detect a predetermined event (step S0). When it is determined that the event is not detected (NO in step S0), the PLC 100 repeats step S0, but when it is determined that the event is detected (YES in step S0), the PLC 100 transmits an imaging start command 1 to the imaging device 40 (step S1).

[0100] When the imaging device 40 receives the imaging start command 1 from the PLC 100 (step S1a), the imaging device 40 starts imaging according to the received imaging start command 1, performs image processing (step S5a), and transmits a response 21 to the imaging start command 1 to the PLC 100 (step S2a).

[0101] The PLC 100 receives the response 21 (step S2b). The PLC 100, as the time management unit 119a, detects a "delay" that is a time difference between when the imaging start command 1 is transmitted and when the imaging start response 21 is received, using the timer 125 (step S3), and manages the detected time difference by storing it as the time information 110 (step S4).

[0102] The imaging device 40 starts imaging in response to the imaging start command 1, and transmits the captured images, which are associated with the imaging time and collected in the image memory 47 after the imaging start, to the server 200 (step S5b). For example, the captured images may be transferred to the server 200 on a file-by-file basis. The PLC 100 also transmits to the server 200 collection control data 400, which is associated with the collection time, to the server 200 (step S6), and transmits the time difference information 110 to the server 200 (step S6a).

[0103] The server 200 receives the control data associated with the collection time transferred from the PLC 100 and the image associated with the imaging time transferred from the imaging device 40 (steps S11 and S12). The server 200 also receives the delay time information 110 transferred from the PLC 100 (step S13).

[0104] The server 200, as the DB manager 251, executes a process of storing the received control data and captured images in the time-series DB 250 in association with each other based on the collection time and image capture time (step S14). Furthermore, the server 200, as the DB manager 251, links (associates) the time information 110 received from the PLC 100 with the association information (step S15). More specifically, the server 200 links (associates) the time information 110 with the association information acquired in step S14, in which the collection time or image capture time indicating approximately the same period as the event timing ST of the time information 110 is associated (step S15).

[0105] As a result, in the time-series DB 250 of the server 200, it is possible to manage the data related to control collected in relation to the control of the control target and the captured image of the imaging device 40 while associating the time shift between the two.

[0106] <D. Example of Time-Series DB> FIG. 10 is a diagram schematically showing an example of the time-series DB 250 according to the present embodiment. Referring to FIG. 10, the processor 202 of the server 200 stores, as a DB manager 251, the data related to control transferred from the PLC 100 and the captured image transferred from the imaging device 40 in the time-series DB 250. More specifically, for each event, the processor 202 stores the collected control data 400 collected for the event and the captured image in file units collected in the image memory 47 as the collected control data 400 and the collected image data 500, respectively.

[0107] The collected control data 400 includes an event timing 410 indicating the time when an event was detected, a collection start time 411 indicating the start of collection of control data indicating one or more control commands or state values ​​to be collected, and time series data 420. The time series data 420 includes time series data 421, 422, etc. Each of the time series data 421, 422, etc. includes a collection time associated with the control data. The PLC 100 collects the control data at a period synchronized with the control period, so that the collection time indicates a time based on the period synchronized with the control period. The collected image data 500 includes an imaging timing 510 indicating the time when an imaging start command 1 was received, and time series image data 550 captured over a predetermined period in response to the imaging start command 1. The image data 550 includes data 515 for each of the imaging devices 40 provided in the control system 1b. The data 515 includes an identifier 520 of the corresponding imaging device 40, an acquisition start time 525, a time series frame 530, a shift 535, and an adjustment amount 536. The acquisition start time 525 indicates the time when the imaging device 40 started capturing captured images in response to an imaging start command 1. The time series frame 530 indicates a time series set of frames captured from the acquisition start time 525 onward, i.e., after the start of imaging. Each frame is associated with an acquisition time.

[0108] The deviation 535 is set to the deviation DT of the time information 110 acquired for the corresponding imaging start command 1. More specifically, the processor 202, as the DB manager 251, searches the collected image data 500 for data 515 corresponding to the imaging timing 510 and identifier 520 indicating the time and camera ID, based on the time and camera ID indicated by the event timing ST of the time information 110 transferred from the PLC 100. The processor 202 sets the deviation 535 of the searched data 515 to the deviation DT of the time information 110.

[0109] As a result, the server 200 can manage by associating with each other the control data associated with the collection time collected by the data collection unit 118a, the image associated with the imaging time collected by the image collection unit 46, and the time information 110 indicating the deviation DT (deviation 535).

[0110] <E. Relationship between Control Data and Image Deviation and Display> In the present embodiment, the time-series frames 530 captured by the imaging device 40 are stored in a storage device such as the time-series DB 250 so as to be reproducible. In the present embodiment, "reproduction" of an image means reading out image data (still image data or moving image data) stored in the storage device and driving a display based on the read image data. Thereby, the display displays an image based on the image data.

[0111] FIGS. 11 and 12 are diagrams schematically showing the relationship between deviation and display according to the present embodiment. FIG. 11 shows an example of data created in cooperation with the server 200 based on the time-series DB 250 for the processor 302 of the terminal 300 to display a screen on the display unit 328 via the UI tool 340. More specifically, the data includes, for each event, the data acquired by the PLC 100 (event timing 410 and collection start time 411) and the data of the imaging device 40 (for each of the imaging device 40-1 and the imaging device 40-2, the identifier 520 of the imaging device, the collection start time 525, and the deviation 535). The data in FIG. 11 has, for each of the two event timings 410, the collection start time 411 of the time-series data 420 and the collection start time 525 and the deviation 535 of the time-series frames 530 by the imaging device 40-1 and the imaging device 40-2.

[0112] Using the UI tool 340, the terminal 300 reproduces on the display unit 328 "Data 1" of the time series data 420 of the PLC 100, "Video 1-1" according to the time series frames 530 of the imaging device 40-1, and "Video 1-2" according to the time series frames 530 of the imaging device 40-2, according to the data in FIG. 10. The PLC 100 controls a device that repeats the same operation, for example, an industrial robot, in synchronization with a period according to a control period. The repeated operation may include, but is not limited to, an operation in which an arm of the industrial robot picks up and places a workpiece. "Data 1" indicates a control command or a state value of a servo motor of the industrial robot collected in a certain repeated pick-and-place operation. "Video 1-1" and "Video 1-1" indicate videos of an industrial robot performing a corresponding pick-and-place operation, taken from different angles.

[0113] The data in FIG. 11 further includes "Data 2" indicating a control command or state value of the servo motor of the industrial robot collected in the next pick-and-place operation, and captured "Video 2-1" and "Video 2-2."

[0114] Fig. 12 shows a shift 535 between "Data 1" and "Video 1-1" and "Video 1-2" shown in Fig. 11. Referring to Fig. 12, when the UI tool 340 outputs "Data 1", "Video 1-1", and "Video 1-2" to the display unit 328 on the same time axis, the UI tool 340 starts outputting "Data 1" as a waveform. "Video 1-1" starts playing 0.506 seconds after the start of the output of the waveform, as indicated by the shift 535, and "Video 1-2" starts playing 0.672 seconds after the start of the output of the waveform, as indicated by the shift 535.

[0115] For example, a case will be described in which a certain time 132 after the start of collection of "data 1" is designated as a certain point (position) on the waveform by a user operation or the like. Based on the time 132 of the designated point, the processor 302, as the UI tool 340, specifies the collection time of the control data corresponding to the designated point as 1.606 seconds from the start of collection. Based on the shift 535, the processor 302 specifies the image of the designated point from the time series frame 530 as a frame image (e.g., a still image) 1.1 seconds (i.e., 1.606 seconds - 0.506 seconds = 1.1 seconds) after the start of imaging for "video 1-1", and plays it. Based on the shift 535, the processor 302 also specifies the image of the designated point from the time series frame 530 as a frame image 0.934 seconds (i.e., 1.606 seconds - 0.672 seconds = 0.934 seconds) after the start of imaging for "video 1-2", and plays it.

[0116] In this way, when "video 1-1" and "video 1-2" are played back in parallel with the output of the waveform of "data 1", the processor 302, as the UI tool 340, can play "video 1-1" and "video 1-2" in synchronization with the waveform output of "data 1" after correcting the time lag of each of them based on the associated lag 535. This is also called "synchronous playback" in the present embodiment.

[0117] (e1: An example of display and adjustment) 13, 14, and 15 are diagrams showing an example of a display according to the present embodiment. FIG. 13 is an example of a screen showing, for example, a time-series state value of the collection control data 400 in the display unit 328 as a waveform. The screen in FIG. 13 includes waveforms for "Data 1" and "Data 2". The vertical axis (Y axis) of the screen in FIG. 13 shows the values ​​of "Data 1" and "Data 2", and the horizontal axis (X axis) shows time. In FIG. 13, the waveforms of "Data 1" and "Data 2" are shifted in the direction in which the time axis (X axis) extends. The user can adjust one waveform by operating the UI tool 340 to slide it in the time axis direction so as to match it with the other waveform. An example of the display of the waveform after the waveform adjustment is shown in FIG. 15.

[0118] In FIG. 13, the UI tool displays buttons 12a and 12b for each waveform, which enable the user to operate the associated "video 1-x" or "video 2-x." By operating button 12a for "video 1-x," the user can play "action 1-1" and "video 1-2" associated with the button 12a via the UI tool 340. By operating button 12b for "video 2-x," the user can play "action 2-1" and "video 2-2" associated with the button 12b via the UI tool 340.

[0119] An example of the display of the screen of "Video 1-x" and the screen of "Video 2-x" being played back is shown in FIG. 14. In FIG. 14, a total of four videos, "Video 1-x" and "Video 2-x" indicated by the title 131, are simultaneously displayed on one screen, but the display mode is not limited to this, and each video may be displayed by switching between them. Also, the video of "Video 1-x" may be displayed as a thumbnail on the button 12a of FIG. 13, and the video of "Video 2-x" may be displayed as a thumbnail on the button 12b of FIG. 13. Also, the screen of FIG. 13 and the screen of FIG. 14 may be displayed in a multi-window on the display unit 328 by the UI tool 340.

[0120] In the screens of Figures 13 and 14, as shown in Figures 10 and 12, when the waveform of "Data 1" starts to be output, "Video 1-1" starts to be played 0.506 seconds after the start of said output, and "Video 1-2" starts to be played 0.672 seconds after the start of said output.

[0121] The UI tool 340 automatically causes the buttons 12a and 12b in FIG. 13 to slide in the direction along the X-axis (time axis) in accordance with the playback speeds of "video 1-x" and "video 2-x," respectively.

[0122] The user can also manually slide the image by operating button 12a or button 12b in Fig. 13. In this case, the image is played back in reverse or fast forward by UI tool 340 according to the moving direction based on the user's operation. Also, the playback speed can be changed by UI tool 340 according to the moving speed of buttons 12a and 12b based on the user's operation.

[0123] 15 shows point 14a where the waveforms of "DATA 1" and "DATA 2" are misaligned. The user slides buttons 12a and 12b so that line 12c extending in the Y-axis direction from buttons 12a and 12b reaches the position of point 14c on the waveform. When line 12c reaches the position of point 14a, UI tool 340 searches time-series frames 530 for frames of "video 1-x" and "video 2-x" that correspond to point 14c, and plays them.

[0124] (e2: Other examples of display and adjustment) 16 is a diagram showing a schematic diagram of another example of the display according to the present embodiment. The user can adjust the difference in playback time between images by operating the UI tool 340.

[0125] 16, the processor 302 displays four images, "Video 1-1", "Video 1-2", "Video 2-1", and "Video 2-2", on the same screen on the display unit 328 as the UI tool 340. A title 131 that identifies the image may also be displayed in the window that displays each image on the screen.

[0126] The images in each window in Figure 16 show, for example at time 132, frame images (still images) of "Video 1-1" and "Video 1-2" that have been "playback synchronized" with "Data 1", and frame images (still images) of "Video 2-1" and "Video 2-2" that have been "playback synchronized" with "Data 2".

[0127] The user can visually determine whether the capture timings of the four playback images captured at time 132 are the same or different. Based on the determination result, the user can instruct the UI tool 340 to play the images frame by frame in a forward or backward manner.

[0128] For example, a case will be described in which the user determines that the frame image of "video 2-2" is captured at a different timing than the other three frame images. When the user performs a designation operation around area 151a of the window of "video 2-2," UI tool 340 accepts the designation operation and displays user-operable buttons 151 in area 151a in accordance with the designation operation. Buttons 151 are operated by the user to play the image in the window frame by frame in either a frame forward or frame backward manner.

[0129] When the user operates button 151, UI tool 340 plays the image in the "Video 2-2" window frame by frame, either forward or backward, in accordance with the amount of operation received. When the user compares the image being forwarded or reversed frame with the image in the other window and determines that the capture timing of these images matches, he or she stops the frame forward or reverse operation of button 151 and performs a confirm operation. When UI tool 340 receives the confirm operation of button 151, it stops the frame forward or reverse of the image in the "Video 2-2" window.

[0130] The processor 302 may store, as the UI tool 340, the amount of frame operations for frame advancement or frame rewinding. More specifically, the processor 302 acquires the amount of frame operations for the user's button 151, associates the acquired amount of frame operations with the data 515 of "Video 2-2" associated with "Data 2" in the time-series DB 250, and stores it as the adjustment amount 536. At the next playback of "Video 1-x" and "Video 2-x" associated with "Data 2", the processor 302 may display, as the UI tool 340, the associated adjustment amount 536 as information supporting frame advancement or frame rewinding operations for "Video 2-2". The processor 302, as the UI tool 340, specifies the playback position of the image data at the time when a user operation input is received during playback, for example, at time 132, and can use, as the adjustment amount 536, the section information indicating a specific section in the time-series frame 515 of the image data with the specified playback position as the starting point. For example, the section information indicated by the adjustment amount 536 includes the number of frames based on the amount of user operations for frame advancement or frame rewinding from the starting point. In FIG. 10, the time-series frame 515 is respectively associated with section information S(1) and section information S(2).

[0131] Factors causing such a deviation in imaging timing between images may include, for example, variations in the load related to image processing of the imaging device 40, or variations in the load related to frame-based image processing.

[0132] <F. Monitor Processing> FIG. 17 is a diagram schematically showing an example of a flowchart of monitor processing according to the present embodiment. The flowchart of FIG. 17 is mainly realized by a program stored and read and executed by the processor 302 of the terminal 300. The monitor processing according to the present embodiment includes the processing related to the display and adjustment described above.

[0133] 17, the processor 302, as the UI tool 340, performs the process of Fig. 17. First, the processor 302, as the waveform display module 341, causes the display unit 328 to display a waveform screen based on control data, for example, the screen of Fig. 13, based on a user operation on the terminal 300 accepted from the input unit 326 (steps R1 and R2).

[0134] The processor 302, as a data adjustment module 343, adjusts the waveform displayed on the screen of FIG. 13 based on user operation, for example by sliding it in the time axis direction, and displays the adjusted waveform of FIG. 15, for example, on the display unit 328 (step R5).

[0135] The processor 302 accepts a user operation for the "video 1-x" or the "video 2-x" based on the user operation (step R7). The "video 1-x" or the "video 2-x" specified by the user operation corresponds to the collected image data 500 associated with the collection control data 400 that was the target of the operation in step R3.

[0136] The processor 302, as the image playback module 342, plays back the "video 1-x" or the "video 2-x", and displays, for example, the screen of Fig. 14 on the display unit 328 (step R9). In the playback scene, as shown in Fig. 15, for example, the processor 302, as the image adjustment module 344, plays back the time-series frames 530 of each video by applying "synchronized playback" based on the shift 535 associated with the video (step R11).

[0137] The processor 302 judges whether an operation to advance or rewind an image frame by frame has been received based on a user operation (step R13). If the processor 302 judges that the operation has not been received (NO in step R13), it judges whether or not to end the monitor process (step R19). For example, when it is judged that the monitor process is to be ended based on a user operation (YES in step R19), the processor 302 ends the series of processes, but when it is judged that the monitor process is not to be ended (NO in step R19), the processor 302 returns to step R9 and continues playing the image.

[0138] When the processor 302 determines that it has received an operation for advancing or rewinding an image frame based on a user operation (YES in step R13), the processor 302, as the image adjustment module 344, advances or rewinds the image in frame units for reproduction based on the amount of the user operation for advancing or rewinding with respect to the button 151 as shown in FIG. 16 (step R15).

[0139] The processor 302, as the adjustment reflection module 345, stores the amount of the operation for advancing or rewinding obtained in step R15 as the adjustment amount 536 shown in FIG. 10 in association with the data 515 of the image operated in step R15 (step R17). Then, the process proceeds to step R19.

[0140] In the embodiment, the adjustment amount 536 is obtained in units of one frame (picture), but the adjustment amount 536 is not limited to one frame unit. It can also be configured to enable adjustment in units of a plurality of frames with one operation amount. Further, it is not limited to frame units, and the adjustment amount 536 can also be obtained from the user operation amount in units of time (for example, 0.1 second, etc.).

[0141] Also, in a scene where the collected image data 500 of the imaging device 40 is reproduced, the processor 302, as the image adjustment module 344, adjusts the reproduction speed (for example, the number of reproduced frames per unit time) of the frames corresponding to the section based on the section information (section information S(1) or section information S(2) in FIG. 10) indicated by the adjustment amount 536 associated with the collected image data 500. This adjustment includes, but is not limited to, fast forward, rewind, skip playback, etc. Thereby, the processor 302 can correct the deviation in imaging timing between the imaging devices 40 in the above section during reproduction, and thereby reproduce the collected image data 500 with the reproduction timings of the imaging devices 40 being matched.

[0142] <G. Modification Example> In the above embodiment, the collection control data 400 and the collected image data 500 associated therewith are managed by the server 200, but the managing device is not limited to the server 200 and may be the PLC 100. The PLC 100 stores the collection control data 400 and the collected image data 500 associated therewith in the storage 109.

[0143] In addition, the time difference 535 between the collection control data 400 and the collected image data 500 associated therewith is obtained as the time from when the time management unit 119a of the PLC 100 transmits an imaging start command 1 to each of the imaging devices 40-1 and 40-2 to when the imaging start response 21 is received from each imaging device 40, but the method of obtaining the difference 535 is not limited to the method of obtaining it from such communication time.

[0144] For example, the time management unit 119a of the PLC 100 may obtain the deviation 535 by calculating the time difference between the collection start time 411 of the collection control data 400 stored in the storage 109 and the collection start time 525 of each data 515 of the collected image data 500. The function of the time management unit 119a may be configured in the server 200. More specifically, the server 200 may obtain the deviation 535 by calculating the time difference between the collection start time 411 of the collection control data 400 stored in the time series DB 250 and the collection start time 525 of each data 515 of the collected image data 500. The function of the time management unit 119a may be configured in the terminal 300. More specifically, as the UI tool 340, the terminal 300 may obtain the deviation 535 by calculating the time difference between the collection start time 411 and the collection start time 525 of each data 515 for the data to be monitored, i.e., the collection control data 400 provided by the server 200 and the collected image data 500 associated therewith.

[0145] Furthermore, regarding the deviation 535, both the communication time and the time difference between the collection start time 411 and the collection start time 525 may be detected, and a representative value (for example, an average) of both may be set as the deviation 535.

[0146] In the above-described embodiment, the imaging start command 1 and the response 21 are transferred over the information network 6, but the transfer path is not limited to the information network 6. For example, the PLC 100 is directly connected to each of the imaging devices 40-1 and 40-2 by a signal line of a wire or a cable. That is, the PLC 100 connects each imaging device 40 by a signal line without passing through other devices, and communicates the imaging start command 1 and the response 21 via the signal line. In this case, the traffic on the signal line between the PLC 100 and the imaging device 40 does not affect the above-described time difference. Therefore, the time shift 535 depends on the communication processing load in the imaging device 40.

[0147] In the above-described embodiment, the PLC 100 uses the time measured by the timer 125 as the collection time of the control data. The PLC 100 performs time synchronization so that the timer 125 measures the same time as a timer (not shown) of the field device 10 belonging to the control network 4. The field device 10 can attach the detection time to the detected state value and transfer it to the PLC 100. In this case, since the PLC 100 can use the detection time attached to the state value collected from the field device 10 as the collection time, the PLC 100 can omit the process of associating the collection time indicated by the timer 125 with the control data (state value) collected from the field device 10.

[0148] Further, the PLC 100 may be configured to perform time synchronization with the imaging device 40 so that the timer 125 and the timer 45 measure the same time. When the PLC 100 and the imaging device 40 are time-synchronized in this way, the above-described time shift 535 may include a time synchronization shift.

[0149] Further, the PLC 100 may be configured to perform time synchronization with other PLCs 100 of the control system 1b so that the timers 125 of the respective PLCs 100 measure the same time.

[0150] <H. Supplementary Note> The above-described embodiment as described above includes the following technical ideas. [Configuration 1] A control system (1a, 1b), A control unit (100) for controlling a control target (10); an imaging unit (40) having an imaging range capable of imaging the control target as a subject, The control unit is a communication unit (119) that transmits an image capture start command (1) to the image capture unit when an event is detected; a data collection unit (118a) that collects control data related to control in association with a collection time; The imaging unit includes: an image acquisition unit (46) that starts imaging in accordance with the imaging start command transferred from the control unit, and acquires the captured images in association with imaging times; The control system further comprises: A control system comprising a time management unit (119a) that manages time information (10) indicating a time delay (DT) between the control data collected by the data collection unit and the images collected by the image collection unit. [Configuration 2] The time lag is The control system according to configuration 1, further comprising a time period from when the control unit transmits the imaging start command to the imaging unit until when the control unit receives a response to the imaging start from the imaging unit. [Configuration 3] The control system according to configuration 2 further comprises an information management unit (200, 250) that manages the control data associated with the collection time collected by the data collection unit, the images associated with the imaging time collected by the image collection unit, and time information indicating the time lag in association with each other. [Configuration 4] The control unit is a first connector (121) for connecting to a network (6) to which the information management unit and the imaging unit belong; The control system according to configuration 3, wherein the imaging start command and the response are communicated with the imaging unit via the network. [Configuration 5] The control system according to configuration 3, wherein the control unit communicates the imaging start command and the response with the imaging unit via a signal line. [Configuration 6] 6. The control system according to any one of configurations 1 to 5, wherein the time lag is detected every time the event is detected. [Configuration 7] The collection time associated with the control data collected by the data collection unit includes a collection start time (411); The image capture time associated with the captured image acquired by the image acquisition unit includes an image capture start time (525); The control system according to any one of configurations 1 to 6, wherein the time lag includes a time difference between the collection start time and the imaging start time. [Configuration 8] The control system according to any one of configurations 1 to 7, wherein the event includes a predetermined event related to control of the controlled object. [Configuration 9] The control unit periodically controls the control target, 8. The control system of claim 7, wherein the predetermined event includes detecting the passage of a predetermined period related to the control. [Configuration 10] The control system further includes a UI (user interface) unit (340), The control unit periodically controls the control target, The UI unit is a waveform display unit (341) that converts the plurality of control data collected by the data collection unit into a waveform based on the collection time associated with each of the plurality of control data, and displays the converted waveform; and an image playback unit (342) that plays back the images collected by the image collection unit according to the imaging times associated with the images. [Configuration 11] the image includes a moving image composed of a plurality of frame images each associated with the image capture time, The control system of configuration 10, wherein the UI unit has an image adjustment unit (344) that identifies and plays back from the plurality of frame images the frame image whose imaging time corresponds to the user-specified point on the displayed waveform, based on the collection time associated with the control data corresponding to the user-specified point and the time shift. [Configuration 12] The imaging unit includes: a plurality of imaging units (40-1, 40-2) having an imaging range capable of imaging the control target as a subject and imaging directions of the control target different from each other; The time management unit The control system of configuration 11, further comprising: managing, for each of the plurality of imaging units, time information indicating the time lag between the control data collected by the data collection unit and the image collected by the image collection unit of the imaging unit. [Configuration 13] The image adjustment unit includes: The control system of configuration 12, wherein for each of the imaging units, the frame image associated with the imaging time corresponding to the user-specified point is identified from the multiple frame images constituting the image captured by the imaging unit based on the collection time associated with the control data corresponding to the user-specified point and the time shift indicated by the time information of the imaging unit, and played back. [Configuration 14] The UI unit is A control system described in any one of configurations 11 to 13, wherein, when a user operation is accepted during playback by the image playback unit, a playback position in the multiple frame images corresponding to the time of the acceptance is identified, and information of a specified section based on the amount of user operation on the image starting from the identified playback position is stored in association with the image. [Configuration 15] 14. The control system according to any one of configurations 11 to 13, wherein, when playing back the image, the image adjustment unit adjusts the playback speed of the image based on information about the specified section associated with the image.

[0151] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, not the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0152] 1 imaging start command, 1a, 1b control system, 2 PLC system, 4 control network, 6 information network, 8 Internet, 10 field device, 12a, 12b, 151 button, 12c line, 14 relay group, 14a, 14c point, 16 servo driver, 18 servo motor, 20 relay device, 21 response, 40 imaging device, 41 optical system, 42 imaging element, 43 DSP, 44 image processing engine, 45, 125 timer, 46 image acquisition unit, 47 image memory, 102, 202, 302 processor, 104 chip set, 106, 306 main memory, 109, 210, 310 storage, 110 time information, 112 IO refresh area, 113 system program, 114 user program, 115 control program, 116 communication program, 117 IO refresh, 118 collection program, 118a data collection section, 119 camera communication program, 119a time management section, 120 information system network controller, 121, 123 connector, 122 control system network controller, 124, 220, 324 USB controller, 126 memory card interface, 128 memory card, 131 title, 132 time, 151a area, 200 server, 204 main memory, 206, 326 input section, 208 output section, 211 management program, 214, 305 recording medium, 215, 304 optical drive, 218 processor bus, 340 tool, 251 DB manager, 300 terminal, 308 bus, 311 application, 313 UI program, 314 waveform conversion program, 315 image playback program, 316 data adjustment program, 317 image adjustment program, 318 Adjustment reflection program, 320, 413 network controller, 328 display unit, 341 waveform display module, 342 image playback module, 343 data adjustment module, 344 image adjustment module, 345 adjustment reflection module, 400 acquisition control data, 410, ST event timing, 411, 525 acquisition start time, 420 time series data, 500 acquired image data, 510 imaging timing, 520 identifier, 530 time series frame, 535,DT deviation, 536 Adjustment amount, 550 Image data, 250 Time series DB, 48 Communication I / F, R1, R3, R5, R7, R9, R11, R13, R15, R17, R19, S0, S1, S1a, S2a, S2b, S3, S4, S5a, S5b, S6, S6a, S11, S12, S13, S14, S15 Step. ,

Claims

1. 1. A control system comprising: A control unit that controls a control target; an imaging unit having an imaging range capable of imaging the control target as a subject, The control unit is a communication unit that transmits an image capture start command to the image capture unit when an event is detected; a data collection unit that collects control data related to the control in association with a collection time; The imaging unit includes: an image acquisition unit that starts imaging in accordance with the imaging start command transferred from the control unit, and acquires the captured images in association with imaging times; The control system further comprises: a time management unit that manages time information indicating a time lag between the control data collected by the data collection unit and the images collected by the image collection unit, The time lag is The control system includes a time from when the control unit transmits the imaging start command to the imaging unit to when the control unit receives a response to start imaging from the imaging unit.

2. The control system according to claim 1 , further comprising an information management unit that manages the control data associated with the collection time collected by the data collection unit, the images associated with the imaging time collected by the image collection unit, and time information indicating the time lag in association with each other.

3. The control unit is a first connector for connecting to a network to which the information management unit and the imaging unit belong; The control system according to claim 2 , wherein the imaging start command and the response are communicated with the imaging unit via the network.

4. The control system according to claim 2 , wherein the control unit communicates the imaging start command and the response with the imaging unit via a signal line.

5. 5. The control system according to claim 1, further comprising: detecting the time lag each time the event is detected.

6. the collection time associated with the control data collected by the data collection unit includes a collection start time; The imaging time associated with the captured image acquired by the image acquisition unit includes an imaging start time, The control system according to claim 1 , wherein the time lag includes a time difference between the acquisition start time and the imaging start time.

7. The control system according to claim 1 , wherein the event includes a predetermined event related to control of the controlled object.

8. The control unit periodically controls the control target, The control system according to claim 7 , wherein the predetermined event includes detection of the passage of a predetermined period related to the control.

9. The control system further includes a UI (user interface) unit, The control unit periodically controls the control target, The UI unit is a waveform display unit that converts the plurality of control data collected by the data collection unit into a waveform based on the collection time associated with each of the plurality of control data, and displays the converted waveform; The control system according to claim 1 , further comprising: an image playback unit that plays back the images collected by the image collection unit in accordance with the image capture times associated with the images.

10. the image includes a moving image composed of a plurality of frame images each associated with the image capture time, The control system of claim 9, wherein the UI unit has an image adjustment unit that identifies and plays back from the plurality of frame images the frame image whose imaging time corresponds to a user-specified point on the displayed waveform, based on the collection time associated with the control data corresponding to the user-specified point and the time lag.

11. The imaging unit includes: a plurality of imaging units each having an imaging range capable of imaging the control target as a subject, the imaging units having different imaging directions for the control target; The time management unit The control system according to claim 10, further comprising: a control unit configured to control a first image capturing unit and a second image capturing unit, the control unit being configured to control a first image capturing unit and a second image capturing unit, the control unit being configured to control a second image capturing unit and a second image capturing unit, the ...

12. The image adjustment unit includes: The control system according to claim 11, wherein for each of the imaging units, the frame image associated with the imaging time corresponding to the user-specified point is identified from the multiple frame images constituting the image captured by the imaging unit and played back based on the collection time associated with the control data corresponding to the user-specified point and the time shift indicated by the time information of the imaging unit.

13. The UI unit is A control system as described in any one of claims 10 to 12, wherein, when a user operation is accepted during playback by the image playback unit, a playback position in the plurality of frame images corresponding to the time of acceptance is identified, and information of a specified section based on the amount of user operation on the image starting from the identified playback position is stored in association with the image.

14. The control system according to claim 13 , wherein, when playing back the image, the image adjustment section adjusts a playback speed of the image based on information of the predetermined section associated with the image.

Images