Programmable Logic Controller

Abstract

The present invention relates to a programmable logic controller. The object of this subject is to appropriately acquire and analyze camera images for surveillance and constantly monitor a surveillance object with high precision. The programmable logic controller (PLC) includes: an image processing unit configured to sequentially acquire image data of camera images from a camera sensor and generate feature quantity data indicating a feature quantity of the image data in a preset surveillance area; a time series data acquisition unit configured to sequentially collect the feature quantity data from the image processing unit and acquire time series data of the feature quantity; and a surveillance unit configured to monitor the time series data of the current feature quantity acquired by the time series data acquisition unit according to a surveillance timing defined by the device in a device memory.

Description

Technical Field

[0001] The present invention relates to a programmable logic controller. Background Art

[0002] A programmable logic controller (PLC) is a controller (control device) that controls industrial machines such as manufacturing equipment, transportation equipment, and inspection equipment in factory automation (JP 2018-097663 A).

[0003] In addition, an image sensor is used to detect abnormalities in equipment, control devices, and manufactured products in a factory.

[0004] However, there are static and dynamic types of abnormalities in the facilities to be controlled, and the image sensor only determines whether the pattern matching is successful and is not suitable for determining static and dynamic abnormalities. The image sensor can clearly distinguish between normal and abnormal, but has difficulty in detecting a situation that is "different from normal". Summary of the Invention

[0005] In view of at least one of the above problems, an object of the present invention is to appropriately acquire and analyze camera images for monitoring and constantly monitor a monitoring object with high precision.

[0006] The present subject matter is, for example, a programmable logic controller including: an execution engine that repeatedly executes a user program; a device memory that stores device values of devices to be accessed by the execution engine according to the user program; a camera input interface that connects to a camera sensor that sequentially captures images according to shooting conditions and sequentially generates image data of the captured camera images; an image processing unit that sequentially acquires image data of camera images from the camera sensor via the camera input interface and generates feature quantity data indicating a feature quantity of image data in a preset monitoring area in the acquired image data of the camera images; a time series data acquisition unit that sequentially collects the feature quantity data from the image processing unit and acquires time series data of the feature quantity; and a monitoring unit that monitors the time series data of the current feature quantity acquired by the time series data acquisition unit according to a monitoring timing defined by the devices in the device memory.

[0007] According to the present subject matter, camera images for monitoring are appropriately acquired and analyzed, and a monitoring object can be constantly monitored with high precision. Brief Description of the Drawings

[0008] Figure 1 is a diagram showing a PLC system;

[0009] Figure 2 is a diagram for explaining a personal computer (PC);

[0010] Figure 3 It is a figure for explaining the PC;

[0011] Figure 4 It is a figure for explaining the PLC;

[0012] Figure 5 It is a figure for explaining the basic unit;

[0013] Figure 6 It is a figure for explaining the analysis unit;

[0014] Figure 7 It is a figure for explaining the expansion unit;

[0015] Figure 8 It is a figure for explaining the format of data recording;

[0016] Figure 9 It is a figure for explaining the transfer timing;

[0017] Figure 10 It is a figure showing the configuration related to camera image analysis in the PLC system;

[0018] Figure 11 It is a flowchart showing the processing procedure of camera image analysis;

[0019] Figure 12 It is a figure for explaining how to extract the feature quantity of the camera image;

[0020] Figure 13 It is a flowchart showing the processing procedure of camera image acquisition and feature quantity generation;

[0021] Figure 14 It is a flowchart showing the processing procedure of the monitoring process;

[0022] Figure 15 It is a figure for explaining data cut-out;

[0023] Figure 16 It is a flowchart showing the processing procedure of preprocessing;

[0024] Figure 17 It is a figure showing the monitoring item setting screen;

[0025] Figure 18 It is a figure showing the monitoring item confirmation screen;

[0026] Figure 19 It is a figure for explaining the feature quantity;

[0027] Figure 20 It is a figure showing the cut-out setting screen;

[0028] Figure 21It is a diagram showing a modified example of the analysis unit;

[0029] Figure 22 It is a diagram showing a modified example of the monitoring item setting screen;

[0030] Figure 23 It is a diagram showing the classification of the collected data;

[0031] Figure 24A It is a flowchart showing the processing procedure of the classification; and

[0032] Figure 24B It is a flowchart showing the processing procedure of the classification. Detailed Description of the Embodiment

[0033] Hereinafter, embodiments will be described in detail with reference to the drawings. Note that the following embodiments are not intended to limit the subject matter according to the claims, and not all combinations of the features described in the embodiments are essential for the subject matter. Two or more of the multiple features described in the embodiments can be optionally combined. In addition, the same or similar components are given the same reference numerals, and repeated descriptions are omitted. Lowercase letter characters may be added to the end of the reference numerals indicating the same or similar elements. When describing matters common to multiple elements, the lowercase letters are omitted.

[0034] <First Embodiment>

[0035] <System Configuration>

[0036] Hereinafter, the first embodiment of the present subject matter will be described. First, in order for those skilled in the art to better understand a programmable logic controller (PLC, which may be abbreviated as a programmable controller), the configuration and its operation of a general PLC will be described.

[0037] Figure 1 It is a conceptual diagram showing a configuration example of a programmable logic controller system according to an embodiment of the present subject matter. As Figure 1 shown, the system includes a PC 2 configured to edit a user program (such as a ladder diagram program, etc.) and a programmable logic controller (PLC) 1 configured to overall control various control devices installed in a factory, etc. PC is an abbreviation for personal computer. A user program can be created using a graphical programming language (such as ladder diagram language or sequential function chart (SFC), etc.) such as a flowchart type motion program, or a user program can be created using a high-level programming language such as C language. Hereinafter, for the sake of convenience of description, the user program is a ladder diagram program. The PLC 1 includes a basic unit 3 combined with a CPU and one or more expansion units 4. One or more expansion units 4 can be attached to or detached from the basic unit 3.

[0038] The basic unit (also referred to as the CPU unit) 3 includes a display unit 5 and an operation unit 6. The display unit 5 can display the operation status, etc. of each expansion unit 4 attached to the basic unit 3. The display unit 5 switches the display content according to the operation content of the operation unit 6. The display unit 5 generally displays the current value (device value) of the devices in the PLC 1 and the error information generated in the PLC 1. Here, the devices include various devices (such as relays, timers, counters, etc.) included in the basic unit 3 and the expansion units 4, refer to the areas on the memory where the device values are set, and can be called device memories. The basic unit 3 collects and stores the device values of each device. Note that the device memory is a non-volatile memory, and can be composed of a rewritable non-volatile ROM, and the non-volatility can be achieved by battery backup of a non-volatile RAM, etc. ROM is the abbreviation of read-only memory. RAM is the abbreviation of random access memory. The device value is information indicating the input status from the input device, the output status to the output device, and the status of internal relays (auxiliary relays), timers, counters, and data memories, etc. set in the user program. The types of device values are bit type and word type, etc. The bit device stores a 1-bit device value, such as 0 / 1, ON / OFF, and H / L, etc. The word device stores a device value of any size. As the collection object of the control program, etc. described in detail below, variables can be specified as devices. The variable is also a holding unit for holding information, and is accessed by the execution engine according to the user program. Therefore, in the following description, the device also refers to the variable. Note that the memory for holding the device can be called the device memory. The memory for holding the collected data can be called the data memory.

[0039] The expansion unit 4 is prepared to expand the function of the PLC 1. Field devices corresponding to the functions of the expansion units 4 can be connected to each of the expansion units 4, whereby each field device is connected to the basic unit 3 via the expansion unit 4. The field device can be an input device such as a sensor or a camera, or can be an output device such as an actuator. A plurality of field devices can be connected to one expansion unit 4. In Figure 1 the example, an example is shown in which a camera sensor 10 as a field device is connected to a camera input unit 4a as an expansion unit. In addition, an example is shown in which other field devices 19 (such as sensors and switches, etc.) are connected to the expansion unit 4c.

[0040] The camera input unit 4a, which serves as an expansion unit, sequentially acquires camera images captured by the connected camera sensor 10 according to shooting conditions, extracts feature amounts related to the received light amount or luminance from the acquired camera images, and sequentially generates feature amount data. The generated feature amount data is collected by the analysis unit 4b during a constant monitoring period. In this system, in addition to constant monitoring, operation record analysis is also performed, in which device values and image data of camera images are held as operation records of the system, and analysis is performed at a predetermined analysis timing using the held device values and camera images. In this case, feature amount data can be generated based on the image data held in the camera input unit 4a at the time of analysis, or feature amount data can be similarly acquired and held in the storage device 32 of the basic unit 3 at the timing of acquiring the device values of the respective devices from the basic unit 3, and when the monitoring conditions are satisfied, the feature amount data is collected as data to be monitored and transmitted to the analysis unit 4b. That is, in the system according to the present embodiment, even during the constant monitoring period, information related to the operation record is held by the basic unit 3 and the expansion unit 4 and used by the analysis unit 4b as data for analysis at a predetermined analysis timing. According to this subject matter, during the constant monitoring period, for example, camera images from the camera sensor 10 are monitored according to the monitoring timing defined by the devices of the device memory. The feature amount data to be monitored during the constant monitoring period can be acquired at the scan time level synchronized with the scan period (which is the acquisition period of the device values described later), or can be acquired asynchronously. When the feature amount data is acquired synchronously with the scan period, the image of the camera image can be easily evaluated with respect to the device values of other devices, so that an abnormality can be detected more accurately and efficiently. The camera sensor 10 acquires moving image data and generates image data such as BMP and JPEG. Feature amount data related to the received light amount and luminance respectively is generated based on the image data, and the feature amount data is sequentially collected in the memory and saved as time series data. The type of the camera sensor is not particularly limited, but it can be a camera dedicated to the camera input unit, or it can be a web camera or a Gigabit Ethernet (GigE) camera, but in the case of being combined with control data, the camera sensor is preferably a camera dedicated to the camera input unit 4a.

[0041] In the present embodiment, the analysis unit 4b, which serves as an expansion unit, is a unit for analyzing the feature amount data from the camera input unit 4a and the device values of the respective devices, and is shown in a form in which no field device is connected.

[0042] The analysis unit 4b is an expansion unit that acquires the device values and feature quantity data generated by the camera input unit 4a during constant monitoring and analyzes whether the state is a "state different from normal" based on the periodic changes in the acquired data. The analysis unit 4b can also acquire the time series data of the device obtained by the basic unit 3 and collected by the first buffer 37a of the basic unit 3 as an operation record at an analysis timing such as when the storage condition of the device is satisfied, and analyze the cause of the "state different from normal". In addition, the analysis unit 4b creates display object data such as a monitoring screen by executing a program such as the process described later to perform data processing on the collected data, and creates display data for displaying a dashboard on the display unit 7b of the PC 2b or a display (not shown) provided in the analysis unit 4b. In the present embodiment, an example of operating the analysis unit 4b as an analysis device will be described, but this is not intended to limit the subject matter, and the basic unit 3 can be used as an analysis device. Although not shown, other PCs can be connected to the expansion unit via a network to display the generated monitoring screen or the like on the PC. The flow program of the process described below is only an example of a control program. The basic unit 3 is sometimes referred to as a CPU unit. Note that the system including the PLC 1 and the PC 2 can be referred to as a programmable logic controller system.

[0043] The PC 2a transfers project data to the PLC 1. On the other hand, the PC 2b is a computer mainly operated by the on-site person in charge. The project data includes setting information such as thresholds and monitoring areas in the control programs of the basic unit 3, the camera input unit 4a, and the analysis unit 4b, as well as in various functions. The control program is a program described in ladder diagram language or flow executed by the control engine. The setting information of various functions includes, for example, flags indicating the attributes of data and the write destinations of data. By determining the appropriate write destinations of each piece of information according to the flags and writing them, the basic unit can be set for each unit. The setting information may also include the setting information of the camera itself, such as exposure time and contrast. The PC 2a can be, for example, a portable notebook-type or tablet-type personal computer or a smart phone, and is an external computer including a display unit 7 and an operation unit 8. The external computer refers to a computer located outside the PLC 1. A ladder diagram program is created using the PC 2a, which is an example of a user program for controlling the PLC 1. The created ladder diagram program is converted into mnemonic codes in the PC 2a. The PC 2a is connected to the basic unit 3 of the PLC 1 via a communication cable 9a such as a universal serial bus (USB) cable. For example, the PC 2a sends the ladder diagram program converted into mnemonic codes to the basic unit 3. The basic unit 3 converts the ladder diagram program into machine code and stores the machine code in the memory set in the basic unit 3. Here, the mnemonic codes are sent to the basic unit 3, but the subject matter is not limited to this. For example, the PC 2a can convert the mnemonic codes into intermediate codes and send the intermediate codes to the basic unit 3.

[0044] Although not shown in Figure 1 the operation unit 8 of the PC 2 may include an indicating device such as a mouse connected to the PC 2. The PC 2 may be configured to be detachably connected to the basic unit 3 of the PLC 1 via other communication cables 9 in addition to the USB cable. In addition, the PC 2 can be connected to the basic unit 3 of the PLC 1 by wireless communication without the need for the communication cable 9.

[0045] <Program creation support device>

[0046] Figure 2 is a block diagram for explaining the electrical configuration of the PC 2. As Figure 2 shown, the PC 2a includes a CPU 11a, a display unit 7a, an operation unit 8a, a storage device 12a, and a communication unit 13a. The display unit 7a, the operation unit 8a, the storage device 12a, and the communication unit 13a are each electrically connected to the CPU 11a. The storage device 12a includes a RAM, a ROM, an HDD, and an SSD, and may also include a removable memory card. CPU is the abbreviation of the central processing unit. HDD is the abbreviation of the hard disk drive. SSD is the abbreviation of the solid state drive.

[0047] The user of the PC 2a causes the CPU 11a to execute the item editing program 14a stored in the storage device 12a, and edits the item data 15 through the operation unit 8a. When the CPU 11a executes the item editing program 14a, the item creation unit 16 and the item transfer unit 17 are implemented. The item creation unit 16 creates the item data 15 according to the user's input. The item transfer unit 17 transfers the item data 15 to the PLC 1. The item data 15 includes one or more user programs (such as ladder diagram programs, control programs, motion programs, and data utilization programs, etc.), the configuration information of each of the basic unit 3 and the expansion unit 4, and the setting information of the identification functions set in each of the basic unit 3 and the expansion unit 4. The configuration information includes the connection positions of multiple expansion units 4 to the basic unit 3 and the allocation information of the devices. The configuration information may include information indicating the functions set in the basic unit 3 (such as data collection function, communication function, and positioning function, etc.) and may include information indicating the functions of the expansion unit 4 (such as communication function, positioning function, and shooting function, etc.). The setting information of the identification functions includes the setting information related to the functions set in the basic unit 3 (such as data collection function, communication function, and positioning function, etc.), and for example, in the case of the data collection function, the setting information includes the data collection conditions and the setting information of the data to be collected, and the setting information includes the setting information related to the functions of the expansion unit 4 (such as communication function, positioning function, data utilization function, and shooting function, etc.). The editing of the item data 15 includes the creation and change of the item data 15. The user can read the item data 15 stored in the storage device 12a as needed and change the item data 15 by using the item editing program 14a. The communication unit 13a communicates with the basic unit 3 via the communication cable 9a. The item transfer unit 17 transfers the item data to the basic unit 3 via the communication unit 13a.

[0048] <PC for displaying the dashboard>

[0049] Figure 3 is a block diagram for explaining the electrical configuration of the PC 2b. As Figure 3 shown, the PC 2b includes a CPU 11b, a display unit 7b, an operation unit 8b, a storage device 12b, and a communication unit 13b. The display unit 7b, the operation unit 8b, the storage device 12b, and the communication unit 13b are electrically connected to the CPU 11b respectively. The storage device 12b includes a RAM, a ROM, an HDD, and an SSD, and may also include a removable memory card.

[0050] The CPU 11b executes the web browser program 14d to implement the web browser 18. The web browser 18 accesses, via the communication unit 13b, the setting page of the data utilization application or the page of the dashboard provided by the analysis unit 4b. In addition, the CPU 11b displays, on the display unit 7b, the recognition result (judgment result) at the time of occurrence of an abnormality, based on the screen information indicating the recognition result sent from the analysis unit 4b.

[0051] <plc>

[0052] Figure 4 is a block diagram for explaining the electrical configuration of the PLC 1. As Figure 4 shown, the basic unit 3 includes a CPU 31, a display unit 5, an operation unit 6, a storage device 32, a communication unit 33, and a memory card 44. The display unit 5, the operation unit 6, the storage device 32, and the communication unit 33 are electrically connected to the CPU 31 respectively. The storage device 32 may include a RAM or a ROM. The storage device 32 has a plurality of storage areas such as a device unit 34, an item storage unit 35, and a first buffer 37a. The device unit 34a has a bit device, a word device, etc., and each device stores a current device value according to a predetermined acquisition setting, and the device unit 34a corresponds to a device memory. The item storage unit 35 stores item data input from the PC 2a. The first buffer 37a generally collects device values to be collected from the device unit 34a according to a predetermined collection setting for each scan cycle as historical data of the operation record of the system. In addition, the first buffer 37a may store feature quantity data of a camera image captured by the camera sensor 10 as a device value, which is generated in the camera input unit 4a.

[0053] Here, for example, constant monitoring means monitoring the camera image according to the monitoring timing defined by the device value of the device memory, and shows a monitoring form of collecting one or more predetermined device values and the camera image in units of a cycle. That is, in the present embodiment, the term "constant monitoring" does not mean always continuously monitoring, but should be understood as monitoring the device value and the camera image at the timing adjusted to a specific predetermined cycle. On the other hand, "operation record analysis" is a monitoring form as follows: the device value, the camera image, or its feature quantity data is held as historical data according to a predetermined collection timing and a predetermined collection setting as described above, and then, at a predetermined analysis timing, for example, when a state different from the normal state is detected in the constant monitoring or when a storage condition described later is satisfied, the historical data is used for cause analysis. Therefore, constant monitoring and operation record analysis are not performed exclusively, but can be performed in parallel.

[0054] Note that, as described above, the data collected by the analysis unit 4b and optionally collected by the basic unit 3 from the camera input unit 4a is feature quantity data of the camera image captured by the camera sensor 10, rather than the image data itself. Therefore, the communication load between each unit can be reduced, and the processing load in each unit can be reduced. Of course, according to the processing performance and communication performance of the system, the image data of the camera image itself can be collected, and this subject is not limited.

[0055] will be referred to later Figure 5 Describe the details of the first buffer 37a. When the save condition is satisfied, the memory card 44 stores the time-series data from each device stored in the first buffer 37a. By temporarily holding the time-series data in the memory card 44, for example, the data to be verified can be verified later in an external device or the like, and important latest data can be held as log information. The data stored in the memory card 44 is historical data for analyzing the factors of "a state different from normal" in the operation record analysis when "a state different from normal" is detected in the constant monitoring. Therefore, when a change equal to or greater than a predetermined value or a change different from normal is detected, it is preferable to save the device values of all devices to the memory card 44. Here, the camera sensor 10 can be included in all devices, and in this case, the feature amount data can be saved to the memory card 44 as the device value. As a result, the feature amount detected as being different from normal in the constant monitoring can be used for the operation record analysis. Note that, as described later, the feature amount is not always the same when the monitoring conditions such as the monitoring area are reset, but even in this case, the feature amount data obtained from the image data held in the moving image memory 37e of the camera input unit 4a described later can be used.

[0056] Here, all devices at least include the devices used in the control program, and can be selected and set by the user in units of programs or units. In this case, the data to be saved is preferably the data for a predetermined period before and after the detection of a state different from normal. Therefore, a control configuration is required in which, for example, the first buffer 37a as a circular buffer is used to always hold the data of all devices for a predetermined period, and when an abnormality occurs, the data in the circular buffer is saved to the memory card 44. Here, when an abnormality (a state different from normal) is detected in the analysis of the captured image using the camera sensor 10, the save condition is satisfied. In addition, when the user gives an analysis command, the save condition is satisfied. The save condition can be preset by the user and saved as part of the item data in the storage device 32 of the basic unit 3. When the CPU 31 detects that the save condition has been satisfied, the CPU 31 issues a command to save the device value stored in the first buffer 37a as a circular buffer to a memory card 44 such as a Secure Digital (SD) card, and notifies the analysis unit 4b that the data to be analyzed has been saved. For example, the execution engine changes a value in the device memory, and the analysis unit 4b monitors whether there is a change in the predetermined value of the device memory, thereby identifying that the data has been saved, and the analysis unit 4b acquires the data from the memory card 44 of the basic unit 3 and performs analysis to identify the abnormal device and the content of the abnormality. This monitoring form is an example of the monitoring form of the above operation record analysis.

[0057] The control program for the basic unit 3 is stored as part of the project data in the project storage unit 35. The control program for the basic unit 3 may be stored separately from the project storage unit 35 in the storage device 32 or may be stored in the storage device 32 in addition to the project storage unit 35. As Figure 4 shown, the basic unit 3 and the expansion unit 4 are connected to each other via the expansion bus 90, which is a type of communication bus and internal bus. In Figure 4 , the communication circuit related to the expansion bus 90 is installed on the CPU 31, but it may also be installed as part of the communication unit 33. The communication unit 33 may have a network communication circuit. The CPU 31 receives the project data from the PC 2a via the communication unit 33.

[0058] Here, a supplementary description of the expansion bus 90 will be given. The expansion bus 90 is a communication bus for input / output refreshing. Input / output refreshing is a process of acquiring input values before each execution of the ladder diagram program when executing the ladder diagram program, and repeatedly executing the ladder diagram program based on the acquired input values, and input / output refreshing is a process of maintaining the previous value even when the input value changes during one execution. The values acquired through input / output refreshing are stored in the device memory as the device values of the devices corresponding to the respective inputs. Through input / output refreshing, device values are transmitted / received between the basic unit 3 and the expansion unit 4, and the device values of the corresponding devices in the device memory are updated. Input / output refreshing is performed in each scan cycle (i.e., each scan) of executing the ladder diagram program once. One scan cycle includes an input / output refreshing execution time period, a ladder diagram program execution time period, and an end processing execution time period. In addition, as described above, the feature quantity data from the camera input unit 4a can be acquired in the basic unit 3 synchronously with the scan cycle. Note that this is not intended to limit the subject matter, and the feature quantity data can be acquired asynchronously with the scan cycle. For example, in the camera input unit 4a, when an abnormal value (a value different from normal) is acquired in the analysis of the captured image of the camera sensor 10, the basic unit 3 and the analysis unit 4b can be notified, and the basic unit 3 and the analysis unit 4b can acquire the feature quantity data in a predetermined period before and after the occurrence of the abnormality.

[0059] The expansion unit 4 includes a CPU 41 and a memory 42. The camera input unit 4a further includes a camera input unit 45 that acquires a camera image captured by the camera sensor 10. The analysis unit 4b further includes a communication unit 43 that sends data to and receives data from the PC 2b, and the analysis unit 4b may include a memory card such as a Secure Digital (SD) card that stores time-series data of control data from each device, and the memory card is detachable from the analysis unit 4b. The memory 42 includes a RAM and a ROM. In particular, in the RAM, a storage area to be used as a buffer memory is ensured. The memory 42 may have a buffer that temporarily holds data such as device values, still image data, and video data acquired by the field devices.

[0060] The CPU 41b of the analysis unit 4b communicates with the PC 2b via the communication unit 43 and the cable 9b. The CPU 41b includes functions for acquiring and analyzing control data such as device values and feature quantity data stored in the memory card 44 provided in the basic unit 3 or the memory card provided in the analysis unit 4b, a dashboard for displaying the results of the analysis process, and a function for sending the results of the process to the outside. These processing functions are stored in the memory 42b as a flow program and are read and executed in the CPU 41b.

[0061] Note that the function of collecting control data can be implemented by a user program. The functions for performing processing related to the collected data include: an identification function for identifying an abnormal device when an abnormality of the programmable logic controller is detected; an analysis function for analyzing the deviation from the time-series data in the normal state and verifying device values different from the normal ones; and a monitoring function for periodically performing the above analysis. The flow program may have arithmetic blocks for collecting data, arithmetic blocks for executing data processing, and arithmetic blocks for creating display data, etc. The dashboard has a graphic display section, a digital display section, etc. These display sections can be implemented by HyperText Markup Language (HTML) data, Cascading Style Sheets (CSS) data, or JavaScript (registered trademark) code, etc. Note that the collection of HTML data, CSS data, and JavaScript (registered trademark) code can be referred to as a web application. In this embodiment, the flow program is implemented by a flow template. The flow template is prepared in advance for each application and has one or more arithmetic blocks, and flow template parameters specified by the user are set in the arithmetic blocks. The dashboard is also enabled by a template. The dashboard template has one or more display sections, and dashboard template parameters specified by the user are set in these display sections. The dashboard template parameters are various information such as a dashboard name, a device name, a numerical value, and a unit variable name, for example. The unit variable is a variable for the expansion unit 4a to hold the execution result of the flow.

[0062] <Functions Implemented by the CPU of the Basic Unit>

[0063] Figure 5 Shows the functions for data utilization implemented by the CPU 31. The execution engine 51a repeatedly executes the user program for each scan cycle that is an execution cycle. The execution engine 51a can be implemented by an ASIC or an FPGA etc. provided outside the CPU 31. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field Programmable Gate Array. These dedicated circuits can generally execute specific data processing at a higher speed than the combination of a CPU and a program. The setting unit 56a makes various settings according to the information input via the PC 2a. Note that some setting functions of the setting unit 56a overlap with the setting unit 71 of the analysis unit 4b to be described later. That is, the setting can be made from any unit, and the setting can be performed via the PC 2b connected to the analysis unit 4b.

[0064] The collection unit 52a collects the device values to be collected from the device unit 34 for each scan cycle that is the execution cycle, and optionally collects the feature quantity data from the camera input unit 4a synchronously or asynchronously with the scan cycle, creates a data record, and stores the data in the first buffer 37a. The collected data is saved as the operation record of the PLC 1 as described above, and becomes the historical data for detailed analysis when the save condition is satisfied. In the case where the execution engine executes the ladder diagram program as the user program, the scan cycle of the ladder diagram program corresponds to the execution cycle, and in the case where the execution engine executes the motion program as the user program, the control cycle of the motion program corresponds to the execution cycle. In the case of collecting the above-described device values and feature quantity data for each scan cycle of the ladder diagram program, the device values to be collected can be collected from the device unit 34 during the end processing time period of the scan cycle, and a data record can be created and stored in the first buffer 37a. Note that it is not necessary to collect data during the end processing time period, and the user program executed by the execution engine 51 can include the description of program codes (such as trigger instructions, etc.) for collecting data. However, in the case of collecting data through end processing, there is an advantage that it is not necessary to create a user program. The collection cycle is set by the user, transmitted as part of the project data, and stored in the storage device 32 of the basic unit 3 as the collection setting 36a. The collection setting 36a includes not only the collection cycle but also the devices to be the collection objects. Note that the collection cycle can be a cycle different from the scan cycle that is the execution cycle or the control cycle of the motion program. In this case, for each collection cycle specified by the collection setting 36a, the device values to be collected and the feature quantity data can be collected from the device unit to create a data record, and a data record can be created and stored in the first buffer 37a. Note that the feature quantity data can be collected synchronously with the collection cycle, can be collected asynchronously, or can be collected in both ways. Here, the basic unit 3 and the analysis unit 4b can analyze the features such as the cycle attribute and continuity of each device value during data collection, and classify the devices according to the analysis results. On the other hand, this analysis can be not performed during data collection, but can be performed by the basic unit 3 or by the expansion unit 4b at the timing when the time series data of each scan cycle in a predetermined number of scan cycles has been collected, or at the timing when it is necessary to analyze data when an abnormality occurs.

[0065] By providing the first buffer 37a, the execution engine 51a is less susceptible to an increase in the scan time due to the collection or transfer process. The device values to be collected are specified by the collection setting 36a. The collection setting 36a can be stored in the basic unit 3 by the PC 2a or the analysis unit 4b. The transfer unit 53a stores one or more data records stored in the first buffer 37a into the memory card 44 set in the basic unit 3, and transfers the data records stored in the memory card 44 to the analysis unit 4b. As described above, when the save condition is satisfied, the data records are saved to the memory card 44. Therefore, there is no need to transfer unnecessary data to the analysis unit 4b, thereby enabling a reduction in the weight of the communication traffic and the amount of data occupied by the memory card. Note that when the communication traffic in the expansion bus 90 is idle, the transfer unit 53a performs the transfer process, but an expansion bus dedicated to sending data records to the analysis unit 4b can be provided.

[0066] Optionally, the transfer unit 53a can transfer one or more data records stored in the first buffer 37a to the analysis unit 4b via the expansion bus 90, and the transferred data can then be stored in the memory card set in the analysis unit 4b. In this case, it is preferable to provide an expansion bus dedicated to sending data records to the analysis unit 4b.

[0067] The transfer process can be performed while avoiding the input / output refresh period or during the period when data is read from the buffer memory of the expansion unit 4 according to the read command described in the user program. Note that the communication traffic of the expansion bus 90 is monitored by the monitoring unit 54a. To reduce the data record transfer time, the compression engine 55a can compress multiple data records. Note that the compression engine 55a does not have to be implemented by the CPU 31, but can be implemented by an ASIC or an FPGA. Therefore, by adopting the first buffer 37a, the transfer process and the user program can be executed asynchronously.

[0068] <Functions of the analysis unit>

[0069] Figure 6 It is a diagram for explaining the functions implemented by the CPU 41b of the analysis unit 4b.

[0070] The collection unit 52c acquires data from the first buffer 37a of the basic unit 3 or the memory card 44 at the analysis timing of the operation record analysis according to the collection setting 36c, and collects device values from each device to be monitored including the camera input unit 4a during the constant monitoring period. Note that the collection unit 52c may collect the device values of the device memory collected in the basic unit 3 during the constant monitoring period. The collection setting 36c is a setting for collecting or acquiring data to be used for the analysis in the analysis unit 4b. The collection setting 36c is set by the user via the PC 2a or the PC 2b, transmitted as part of the item data, and stored in the memory 42b of the analysis unit 4b. The collection unit 52c may be implemented by the CPU 41b executing a control program such as a user program. The collection unit 52c may set the basic unit 3 so that the basic unit 3 collects the device values specified by the collection setting 36c as historical data of the operation record, and transmits the device values to the second buffer 37b of the analysis unit 4b, and the collection unit 52c may set each unit so that the analysis unit 4b collects the device values of the predetermined devices to be monitored and the feature amount data from the camera input unit 4a during the constant monitoring period. Note that the collection unit 52c may write the collection setting 36a of the basic unit 3 included in the collection setting 36c into the storage device 32 of the basic unit 3.

[0071] The collection unit 52c collects device values of one or more pre-specified devices and feature amount data indicating features of the camera image from the camera input unit 4a during constant monitoring according to the collection setting 36c. The analysis unit 4b collects the data to be monitored in this way so that real-time performance can be improved. On the other hand, at the timing of operation record analysis, the collection unit 52c obtains the data collected in the first buffer 37a of the basic unit 3 and stored in the memory card 44. Therefore, factor analysis can be performed by using the historical data of the operation record before and after a state different from normal is detected during constant monitoring. Specifically, when performing operation record analysis, the device values collected by the basic unit 3 according to the collection setting 36a are transferred from the basic unit 3 to the analysis unit 4b at the timing of analysis by the analysis unit 4b. When the feature amount data is not collected in the first buffer 37a of the basic unit 3 as historical data of the operation record together with the device values, the analysis unit 4b generates the feature amount data in its own unit based on the image data held in the camera input unit 4a. The feature amount data can be generated in any of the camera input unit 4a, the analysis unit 4b, or the basic unit 3. For example, the analysis unit 4b and the basic unit 3 generate the feature amount data by using the image data of the camera image held in the camera input unit 4a at the timing of operation record analysis. Therefore, when the analysis unit 4b and the basic unit 3 are to generate a feature amount during operation record analysis, a new feature amount can be generated and analyzed under monitoring conditions different from the monitoring conditions of the feature amount monitored during constant monitoring, and the factor analysis when a change different from normal is detected during constant monitoring can be analyzed in detail. On the other hand, when the feature amount is to be generated in the camera input unit 4a and saved as an operation record, the configuration of the analysis unit 4b can be shared between constant analysis and operation record analysis to reduce the design cost. In addition, the feature amount in which a change different from normal is detected during constant analysis can also be used as it is for operation record analysis, and the processing load for newly generating the feature amount can be reduced.

[0072] By providing the second buffer 37b and the third buffer 37c, data can be collected without loss even when the processing load of the data processing unit 73 fluctuates. The collection unit 52c can write the collection settings included in the collection settings 36c for collecting data from the camera input unit 4a into the memory 42b of the analysis unit 4b. Note that these setting functions can be implemented by the setting unit 71. The setting unit 71 receives the collection settings 36c, the processing settings 61, and the display settings 62 from the PC 2a or the PC 2b, and writes the received settings into the memory 42b. The processing settings 61 include information and a flow program for defining the data processing to be performed on the collected data by the data processing unit 73. The display settings 62 include a dashboard template (HTML data, CSS, JavaScript (registered trademark) code, etc.), which provides the data processing result to the web browser 18 through the web server 70.

[0073] In addition, the CPU 41b of the analysis unit 4b implements the display control unit 74, the monitoring unit 76, and the transmission unit 79 as functional configurations. The monitoring unit 76 monitors the time-series data of each device collected by the collection unit 52c and the feature amount data generated by the camera input unit 4a, and monitors changes different from normal. As described above, the monitoring unit 76 basically performs the monitoring process at a predetermined cycle according to the monitoring timing defined by the device value in the device memory as constant monitoring, but "constant" can be applied in various modes. For example, constant monitoring can be implemented in the real-time mode, the continuous mode, and the scan time level (scan cycle) mode. These modes can be changed according to the required safety performance level, the processing capacity of the PLC 1, and the settings of the administrator. Although the monitoring unit 76 monitors data different from normal during constant monitoring, the monitoring unit 76 can determine that the data is different from normal when detecting a predetermined amount or more of change compared with, for example, previously acquired data or the average value within a predetermined range, or can determine an abnormality determination parameter based on the time-series data in the normal state, learn according to the determined parameter, and input the collected data into the learned model to determine that the data is different from the normal data. Learning can be performed using teacher data including the time-series data in the normal state and the time-series data with abnormal data to generate the learned model. Preferably, the once-generated learned model is updated by additional learning of the learning data and the teacher data to cope with changes over time and improve the accuracy of abnormality detection. Instead of learning the intentionally created teacher data, positions that seem to be normal, positions that seem to be abnormal, and positions not used for learning can be extracted from the continuously collected data and learned automatically. In this case, when it comes to periodic changes, preferably, for each cycle, the part where the difference between values is within a certain range is determined to be normal for each cycle, extracted, and learned. When learning the intentionally created teacher data, there are advantages that the same teacher data can be used in multiple different systems, and the same teacher data can be used each time to learn a somewhat predictable operation. On the other hand, when additional learning is performed automatically, the advantage is that changes over time in the system can be handled. Considering these advantages, it is preferably switched appropriately by the administrator or the like. Therefore, the administrator or the like can perform the setting for switching the learning method via the PC 2a or the PC 2b.

[0074] The display control unit 74 substitutes the data processing result into the template of the dashboard according to the display setting 62 that defines the display unit of the dashboard, so as to generate the display data of the dashboard. The display control unit 74 also creates the display data for displaying the monitoring result based on the monitoring result of the monitoring unit 76. The display data can be, for example, HTML data, image data, Cascading Style Sheets (CSS), or JavaScript (registered trademark) code, etc. The display unit includes, for example, a camera image and a pie chart unit, a bar chart unit, a line chart unit, or a numerical display unit that indicates the feature quantity data as the feature quantity of the camera image. When accessing the web page of the dashboard through the web browser 18, the web server 70 sends the display data of the dashboard to the web browser 18. The web browser 18 receives the display data and displays the display data in the dashboard. The sending unit 79 sends the display data generated by the display control unit 74 to an external device. There are no particular restrictions on the communication line and communication method used during sending. A wireless connection or a wired connection can be used. The display control unit 74 can display the generated result screen on the display set in the analysis unit 4b.

[0075] Note that multiple analysis applications (such as an analysis application for each device value and an analysis application for feature amount data from the camera input unit 4a, etc.) can be provided. In this case, for each analysis application, the required data and read timing can be different, and for each analysis application, a sub-buffer can be ensured in the memory 42b. The collection unit 52c reads the data record stored in the second buffer 37b and stores the data for the first analysis application into the first sub-buffer 38a. The collection unit 52c reads the data record stored in the second buffer 37b and stores the data for the second analysis application into the second sub-buffer 38b. Note that the collection unit 52c can read the data record stored in the third buffer 37c and store the data for the first analysis application into the first sub-buffer 38a. The collection unit 52c can read the data record stored in the third buffer 37c and store the data for the second analysis application into the second sub-buffer 38b. The data processing unit 73 reads data from the first sub-buffer 38a according to the first analysis application, performs data processing, and generates a processing result. The data processing unit 73 reads data from the second sub-buffer 38b according to the second analysis application, performs data processing, and generates a processing result. The decompression engine 75 has a function paired with the compression engine 55a of the basic unit 3 and the compression engine 55b of the analysis unit 4b. The decompression engine 75 decompresses the data compressed and transmitted by the basic unit 3 and stores the data in the second buffer 37b. The decompression engine 75 decompresses the data compressed and transmitted by other expansion units and stores the data in the third buffer 37c. This will reduce the congestion of the communication traffic on the expansion bus 90. The decompression engine 75 can be implemented by an ASIC or an FPGA. As described above, the data transmission between the basic unit 3, the camera input unit 4a, the analysis unit 4b, and the expansion unit 4c is performed via the expansion bus 90.

[0076] There may be a case where multiple data are required for each analysis application. In this case, multiple data required for the sub-buffer can be stored while maintaining all the data collected in each scan in the circular buffer. In addition, when data are also allocated to the sub-buffer, a timestamp or the like can be given to each record. As Figure 8 shown, the second buffer 37b or the third buffer 37c holds the collected data block. Although an example of holding data related to the device of each device is shown here, it is also reasonably assumed to hold the feature amount data from the camera input unit 4a. One record includes a scan number, a timer value as a timestamp, and the collected data. The collected data includes, for example Figure 8 The device values Dev1, Dev2, and Dev10 shown in and data such as relays RL1 to RL3. The first analysis application requires relays RL1 to RL3 of the collected data. Therefore, the scan number, timer value, and relays RL1 to RL3 are read from the second buffer 37b and stored in the first sub-buffer 38a. The first analysis application reads the scan number, timer value, and relays RL1 to RL3 from the first sub-buffer 38a to create display screen data. The second analysis application requires relays RL3 of the collected data and devices Dev1 and Dev2. Therefore, the scan number, timer value, relay RL3, and devices Dev1 and Dev2 are read from the second buffer 37b and stored in the second sub-buffer 38b. The second analysis application reads the scan number, timer value, relay RL3, and devices Dev1 and Dev2 from the second sub-buffer 38b to create display screen data. By using the sub-buffers in this way, the original data can be maintained in the buffer without being changed. The original data held in the buffer can be used for other purposes.

[0077] <Function of the expansion unit 4a related to camera input>

[0078] Figure 7 It is a diagram for explaining the functions implemented by the CPU 41a of the camera input unit 4a. The camera input unit 4a is connected to a camera sensor 10 as a field device. Note that the configuration of the expansion unit 4c to which field devices 19 such as sensors and switches are connected is substantially the same as that of the camera input unit 4a, so its description is omitted. The expansion unit 4c is different in that the expansion unit 4c does not include the feature quantity generation unit 54b and the compression engine 55b of the camera input unit 4a to be described later, and the expansion unit 4c is different in the functional configuration in the memory.

[0079] The execution engine 51b acquires and saves the captured image of the camera sensor 10, which is the basic function of the camera input unit 4a. The execution engine 51b sequentially acquires the image data of the camera images from the camera sensor 10 that sequentially captures images according to the imaging conditions, saves the image data to the moving image memory 37e, and saves the feature quantity data indicating the predetermined feature quantity of the captured image sequentially generated by the feature quantity generation unit 54b based on the image data to the fourth buffer 37d. The details of the feature quantity data will be described later. The data stored in the fourth buffer 37d can be transmitted to the analysis unit 4b as data for constant monitoring during the constant monitoring period, and can also be transmitted to the basic unit 3 as historical data for operation recording.

[0080] The image data stored in the moving image memory 37e is transferred to the basic unit 3 or an expansion unit having a Web server (e.g., the analysis unit 4b) as needed. The image data is used to generate feature quantity data at the transfer destination or to display a monitoring image on a confirmation screen. Thus, the operator of the PLC system can confirm the moving image to be captured with the monitoring image. For example, the feature quantity data can be transferred to the analysis unit 4b according to a scan cycle, and the image data can be transferred to the basic unit 3 or the analysis unit 4b at a timing different from the above timing. Further, when the image data itself is only used for the display of the monitoring image and not for the generation of the feature quantity data, since the data is not the data to be monitored, it is preferable to reduce the data amount by deleting data at intervals. This enables a reduction in the communication volume between the respective units. For example, the execution engine 51b can detect a timing when the communication traffic of the expansion bus 90 is small, and the transfer unit 53b can transfer the image data of the camera image.

[0081] The feature quantity generation unit 54b generates feature quantity data obtained by extracting a predetermined feature quantity from the image data acquired by the camera sensor 10, and the generated feature quantity data is stored in the fourth buffer 37d. At the timing for generating the feature quantity data, the feature quantity data is basically generated according to the monitoring timing defined by the device memory during a constant monitoring period, but the feature quantity data can be generated according to the scanning period of the device value depending on the image acquisition period of the camera sensor or the like. In addition, since the scanning period is generally shorter than the image acquisition period of the camera sensor 10, there is no camera image corresponding to the scanning period. Therefore, by analyzing the feature quantity data generated by the latest shooting, analysis synchronized with the scanning period can be achieved. In this case, the same feature quantity data can be used in multiple scanning periods. For example, it is assumed that the camera sensor makes one acquisition during the period when four scanning periods arrive, and in this case, the feature quantity data from the same camera image corresponds in four scanning periods. As described above, by generating the feature quantity data according to the scanning period and analyzing the relationship with other device values, the detection accuracy of devices different from the normal ones can be improved. Note that this is not intended to limit the subject matter, and the feature quantity data can be generated at other timings. In addition, the feature quantity generation unit 54b generates feature quantity data obtained by extracting the feature quantity of at least a partial range (predetermined area) of the camera image captured by the camera sensor 10. This range will be described in detail later, but it corresponds to the area set by the operator in the camera image displayed on the setting screen. That is, the operator can change the range from the PC 2a connected to the basic unit 3, the PC 4b connected to the analysis unit, the PC connected to another expansion unit, and the like. When the range is changed, the coordinate values corresponding to the changed range are notified to the camera input unit 4a via the expansion bus 90 from the basic unit 3 or the analysis unit 4b, and the feature quantity generation unit 54b changes the subsequent extraction area of the feature quantity according to the notified content.

[0082] The compression engine 55b compresses the data record according to the collection setting 36. That is, the transfer unit 53b can transfer the data record whose information has been compressed by the compression engine 55b to the basic unit 3 and the analysis unit 4b. The compression engine 55b can be implemented by the CPU 41a, but from the perspective of high-speed processing and reducing the processing load of the CPU 41b, it can be implemented by an ASIC or an FPGA..

[0083] <Example of data record>

[0084] Figure 8 Shows the data record 91 written by the collection unit 52a to the first buffer 37a. Here, as an example, the case where the basic unit 3 collects the operation records of the PLC 1 as data will be described, but it is equally applicable to the case where the analysis unit 4b collects device values, etc. from each unit during constant monitoring. The multiple data records 91 are examples of time-series data. In this example, the collection unit 52a collects the device values of device names Dev1, Dev2, and Dev10 from the device unit 34a for each scan cycle, adds the time information obtained from the collection count and the timer to create a data record, and stores the data record in the first buffer 37a. Note that the collection target can be the data stored in the buffer memory or device assigned to the analysis unit 4b. In this example, the first buffer 37a is a first-in, first-out (FIFO) type buffer. The collection count is the count value of a counter that increments by 1 each time a data record is collected. The collection count is sequentially numbered and is thus used to detect data record loss or compression.

[0085] For example, when the data obtained by the basic unit 3 and the data obtained by the expansion unit 4b are displayed in the dashboard in a comparable manner, time information such as a timestamp is useful. Generally, the collection timing in the basic unit 3 is not consistent with the collection timing in other expansion units. Therefore, comparing the operation of the basic unit 3 with the operation of other expansion units requires information for correlating the data of the basic unit 3 with the data of other expansion units. Generally, the time information can be synchronized between the basic unit 3 and other expansion units through inter-unit synchronization, etc. Therefore, the basic unit 3 and other expansion units each assign the time information at the time of collecting the data record to the data record, so that the data processing unit 73 can align multiple data records obtained by different units on the time axis. In addition, before analysis, the data processing unit 73 performs preprocessing for cutting out the feature quantity data of the camera image to be analyzed in association with the change in the device value of a predetermined device. For the preprocessing, it is assumed that the time-series data of the device value and the time-series data of the feature quantity data are synchronized.

[0086] In the case where the feature quantity data from the camera input unit 4a is also collected synchronously with the scan cycle, the feature quantity data can be collected synchronously with the collection counter in the same manner, or the feature quantity data can be collected intermittently in consideration of the image acquisition cycle of the camera sensor 10 and the memory resources of each unit. In addition, since the scan cycle is usually shorter than the image acquisition cycle of the camera, in most cases, a camera image corresponding to the scan cycle is not obtained. In this case, the feature quantity data indicating the feature quantity generated by using the most recently captured image is collected. This enables the analysis to be synchronized with the scan cycle. For example, the feature quantity data can be collected once every four counts of the collection count. This collection timing can be changed by setting. By collecting the feature quantity data of the camera image from the camera input unit 4a synchronously with the scan cycle, the device and the camera image can be correlated with each other and analyzed.

[0087] In the case where the feature quantity data is collected asynchronously with the scan cycle, the data is collected when there is a timing offset in the scan cycle. Similarly in this case, the counter generated by the basic unit 3 at the time of collection is recorded, so it is also possible to perform analysis in association with the counter generated by the basic unit 3 associated with each device value. In addition, the feature quantity data of the camera image can be analyzed separately, and in this case, it can be analyzed whether there is a change in the feature quantity in the region of interest between the feature quantity data of the previously captured image and the feature quantity data of the currently captured image. For example, when there is a change equal to or greater than a predetermined value, the change can be detected as a value different from the normal, or when there is a change that should not occur normally, or when there is a change in a cycle, the state can be detected as different from the normal. As described above, when a state different from the normal is detected when analyzing the feature quantity data separately, it can be considered that the above storage conditions are satisfied. As described above, a log can be left according to the storage conditions, and the data in the circular buffer can be stored in the memory card 44 of the basic unit 3 or in a memory card (not shown) provided in the analysis unit 4b. When the feature quantity data of the camera image is a value within a specific range or the same value as the analysis value obtained from the last analysis, there is no need to store the feature quantity data, and there is no need for the analysis of the analysis unit 4b. By judging the change of the feature quantity in this way in the first stage, an effective detection related to whether there is a change in the camera image can be performed. Although the device values of the predetermined device and the feature quantity data of the camera input unit 4a are monitored during the constant monitoring period, when a state different from the normal is detected, the data of each device in all devices and the image data of the camera can be stored as historical data of the operation record in the memory card 44. Thus, it is possible to easily and detailedly identify the abnormal device and the abnormal factor.

[0088] <Transfer Timing>

[0089] Figure 9 is a diagram for explaining the transfer timing of data records. As Figure 9 shown, PLC 1 repeatedly performs input / output refreshing, execution of the user program, and end processing. To reduce the extension of the scan cycle, transfer processing between the basic unit 3 and the expansion unit 4 is executed to avoid the time period for input / output refreshing. Similarly, to reduce the extension of the scan cycle, transfer processing is executed to avoid the execution time periods of UREAD and UWRIT. UREAD is a command to read data from the buffer memory allocated to the expansion unit 4 and is described in the user program. Therefore, during the execution time period of the user program, the basic unit 3 accesses the expansion unit 4 according to UREAD to obtain data from the buffer memory. UWRIT is a command to write data into the buffer memory allocated to the expansion unit 4 and is described in the user program. During the execution time period of the user program, the basic unit 3 accesses the expansion unit 4 according to UWRIT and writes the data into the buffer memory.

[0090] As Figure 9 shown, during the remaining transferable time periods other than input / output refreshing, UREAD, and UWRIT, transfer processing is executed on the expansion bus 90. For example, assume that it has been set by the collection setting 36a to execute transfer processing for five data records once. In this case, after accumulating the five data records in the first buffer 37a and at the timing of the first transferable time period or when the camera input unit 4a receives a transfer request, the transfer unit 53a executes the transfer processing. During the transferable time period, image data is transferred from the camera input unit 4a by using the time period when the transfer processing of data records is not performed.

[0091] <Configuration related to camera image analysis>

[0092] Figure 10 is a diagram showing the configuration related to camera image analysis in the PLC system. In Figure 10 it, components that are the same as those shown in Figure 4 showing the electrical configuration of the PLC system and components shown in Figures 5 to 7 showing the functional configuration of each unit are given the same reference numerals. Here, the configuration for analyzing the camera image captured by the camera sensor 10 and displaying the result will be described with reference to Figure 10 Units 3, 4a, and 4b each include bus communication units 50a, 50b, and 50c. The bus communication units 50a, 50b, and 50c can send and receive signals to and from each other via the expansion bus 90. Note that the expansion unit 4c also has these configurations, but for ease of description, its detailed configuration is omitted here.

[0093] First, the camera sensor 10 captures a moving image of the object range, generates image data, and inputs the image data into the camera input unit 45. Any communication component can be applied to the transmission of the image data, and for example, dedicated communication or communication with a general network camera such as the Open Network Video Interface Forum (ONVIF) can be applied. The camera input unit 45 receives the image data from the camera sensor 10 and transfers the image data to the feature quantity generation unit 54b. Various settings can be made via the PC 2a or PC 2b to capture an image by the camera sensor 10. Examples of the setting items include automatic exposure correction, automatic white balance correction, contrast, sharpness, and image rotation. The automatic exposure correction is a setting for whether to automatically adjust or fix the brightness. The automatic white balance correction is a setting for whether to automatically adjust or fix the color. The contrast is a setting for vividness. The sharpness is a setting for the contour intensity. The image rotation is a setting for the direction of the captured image. The image data to be acquired includes, for example, YUV, RGB, and JPEG. Of course, other parameters can also be set, and the setting items vary according to the performance of the camera sensor.

[0094] The feature quantity generation unit 54b generates feature quantities related to the received light quantity and brightness from the image data as the image data. For example, the feature quantity generation unit 54b extracts the average value of at least one feature quantity (such as a brightness component, a color difference (blue) component, a color difference (red) component, an R component, a G component, a B component, and their degree of change, etc.) within a predetermined area from the image data of a predetermined area of the camera image, extracts the average value of the gradient intensity and angle, etc. from the image data of the predetermined area, and the feature quantity generation unit 54b stores the average value as feature quantity data in the fourth buffer 37d. The range of the predetermined area can be set via the setting screen displayed on the PCs 2a and 2b. Details of the setting screen will be described later. Thereafter, the feature quantity data is collected from the camera input unit 4a to the basic unit 3 or the analysis unit 4b together with each device value synchronously with the scanning cycle via the bus communication unit 50b and is used for analysis in the analysis unit 4b. As described above, the feature quantity data can be collected asynchronously with respect to the scanning cycle. The bus communication unit 50b also transfers the image data of the camera image captured by the camera sensor 10 to the basic unit 3 or the analysis unit 4b via the extended bus 90 during the transmittable time period.

[0095] The device values of each device collected as operation records by the basic unit 3, the feature quantity data of the camera images, and the image data are transmitted to the analysis unit 4b via the expansion bus 90. When transmitting the data of these operation records respectively, or when acquiring the feature quantity data or device values during constant monitoring, the monitoring unit 76 of the analysis unit 4b monitors the time series data of these device values and the feature quantity data, monitors changes such as values different from normal, and detects anomalies. For the feature quantity data of the camera images, for example, when the feature quantity of the monitoring object is the luminance component, the monitoring unit 76 monitors whether the average value of the luminance in each period exceeds a predetermined threshold value and detects anomalies. The monitoring unit 76 preferably saves the monitoring results as log information to a memory card (not shown) connected to the memory 42b or the analysis unit 4b. Although the log information is preferably held in a non-volatile memory, a volatile memory can be used when monitoring is performed only during power-on.

[0096] The display control unit 74 generates a confirmation screen indicating the monitoring object of the monitoring unit 76 and displays the confirmation screen on the Web browsers of the PCs 2a, 2b or other devices. The confirmation screen can also be displayed on the display of the basic unit 3 or the expansion unit 4. The confirmation screen displays the time series data of the feature quantity as the monitoring object, the time series data of the device values synchronized with the time series data of the feature quantity, the current state of the camera images, and the monitoring results. The monitoring results include, for example, the number of NG occurrences indicating the number of times judged to be abnormal. These displays are selected according to the performance of the display for display. The details of the confirmation screen will be described later.

[0097] The PC 2a connected to the basic unit 3 can, via the setting unit 56a, reflect the user program executed by the execution engine 51a, the settings of the feature quantity generation unit 54b, and the settings of the monitoring unit 76 to each unit via the communication unit 33. For example, the setting unit 56a checks the flags of each setting content, determines each setting destination, and performs writing. Therefore, the settings are written from the basic unit 3 to each expansion unit 4. Note that the settings of the feature quantity generation unit 54b and the monitoring unit 76 can also be made via the PC 2b. The setting of the feature quantity generation unit 54b is provided in the camera input unit 4a, which is an expansion unit 4 different from the analysis unit 4b connected to the PC 2b, so that the already input setting content is notified via the expansion bus 90 serving as an internal bus. The data that can be set by the PC 2b is limited, and collection settings, acquisition settings, settings of each device (especially settings related to the camera sensor 10), and monitoring settings can be performed, but item data including the user program to be executed cannot be transmitted to each unit via the PC 2b. In order to be able to make settings from a Web server such as a PC connected to the expansion unit as described above, it is necessary to consider conflicts with the settings from the PC 2a connected to the basic unit 3. Therefore, in addition to the buffer memory, each expansion unit is preferably provided with a temporary write memory for temporarily holding the received setting content. The settings from the Web server are temporarily written to the temporary write memory and then rewritten to the buffer memory. At the time of rewriting, it is confirmed whether the setting content to be newly updated is data within the defined range or whether rewriting is not prohibited. The defined range has a fixed value in advance as the initial setting or system of the item data. In addition, by providing the temporary write memory, setting changes can be received even during the execution of the monitoring process. Data can also be temporarily held in the temporary write memory and written after waiting for the end of the monitoring process. Note that the present subject matter is not limited to such a configuration having a separate memory, and the setting data can be temporarily stored in the memory of the PC connected to the Web server, and it can be confirmed whether the data is within the defined range or whether rewriting is not prohibited, or the buffer memory can be divided into multiple parts and have a part for temporary writing as described above. Now, the differences between the PC 2a and the PC 2b will be described. There are differences in the operation authorities between the user of the PC 2a and the user of the PC 2b, and the operator of the PC 2b can only change some settings. The PC 2a is used at the initial startup of the PLC 1, and the PC 2b is used at the start of operation or during system operation. The PC 2a and the PC 2b can be tablet terminals or the like. Note that the present subject matter is not intended to limit these matters, and can be arbitrarily changed according to the installation environment, specifications, etc.

[0098] In this embodiment, a configuration in which the basic unit 3, the camera input unit 4a, the analysis unit 4b, and the extension unit 4c are separated is described. However, this is not intended to limit the subject matter, and for example, these units may be provided integrally. That is, the basic unit 3 may have the functions of the camera input unit 4a, the analysis unit 4b, and the extension unit 4c. Alternatively, the basic unit 3 may have partial functions of the camera input unit 4a, the analysis unit 4b, and the extension unit 4c. The camera input unit 4a, the analysis unit 4b, and the extension unit 4c may be provided integrally.

[0099] <Camera analysis processing flow>

[0100] Figure 11 Shows a processing flow for acquiring and analyzing a camera image and displaying the result according to this embodiment. The processing described below will be described as processing executed by the CPU 31 of the basic unit 3, the CPU 41a of the camera input unit 4a, and the CPU 41b of the analysis unit 4b. However, this is not intended to limit the subject matter, and some processing may be executed in other units, or all processing may be executed in one unit or may be executed by an external device (analysis device) communicatively connected to a programmable logic controller.

[0101] In S1, the CPU 31 (setting unit 56a) sets the collection setting 36a of the basic unit 3 according to the information input via the PC 2a. For example, the CPU 31 stores the collection setting 36a for the basic unit 3 and the collection setting 36a based on the collection settings and acquisition settings obtained from the camera input unit 4a, the analysis unit 4b, and other expansion units in the storage device 32. The collection setting stored in the storage device 32 is, for example, a setting that instructs the basic unit 3 which device data is to be collected and transmitted to the analysis unit 4b. The camera input unit 4a stores the acquisition setting 36b in the memory 42a. The analysis unit 4b stores the collection setting 36c in the memory 42b. These settings are settings that indicate which device data is to be collected. For example, settings are made such that during constant monitoring, the device values of a predetermined device are collected in the corresponding unit and transmitted to the analysis unit 4b. Although an example is described here in which the setting unit 56a of the basic unit 3 performs the setting independently, the present subject matter is not limited thereto, and the setting unit 71 of the analysis unit 4b can perform the setting independently. In this case, the setting unit 71 performs the setting according to the information input via the PC 2b and transmits the collection setting to other units. Additionally, in S1, the CPU 41b (setting unit 71) of the analysis unit 4b performs camera settings such as shooting conditions, image processing settings, monitoring item settings, etc. according to the information input via the setting screen displayed on the display unit 7b of the PC 2b. The setting screen for these will be described later. Note that the CPU 31 of the basic unit 3 can perform these settings according to the information input via the setting screen displayed on the PC 2a.

[0102] In S2, the CPU 41a of the camera input unit 4a acquires a camera image (image data) captured by the camera sensor 10 according to the content of the acquisition setting 36b set in S1 via the camera input unit 45. Here, the CPU 41a temporarily stores the image data of the acquired camera image in the memory 42a. Further, the CPU 41a (feature quantity generation unit 54b) extracts the feature quantity of a predetermined region of the image data held in the memory 42a, and the analysis unit sequentially collects and accumulates this feature quantity to generate time series feature quantity data. The feature quantity generation unit 54b extracts a specified feature quantity from a predetermined region corresponding to the setting content via the setting screen. Although the feature quantity of the type set via the setting screen is extracted here, basically, in addition to the feature quantity of the set type, extractable feature quantities are also extracted. The feature quantity generation unit 54b associates the generated feature quantity data with a counter (timestamp) and saves the associated data in the fourth buffer 37d. The detailed processing of S2 will be described below with reference to Figure 13 Describe the detailed processing of S2.

[0103] In S3, the CPU 31 (collection unit 52a) collects the device values of each device to be collected and the feature quantity data indicating the feature quantity of a predetermined area of the camera image captured by the camera sensor 10 according to the set collection setting 36a, and stores these in the first buffer 37a. Regarding the collection timing of the feature quantity data, the feature quantity data can be collected synchronously or asynchronously with the scan cycle for collecting the device values. On the other hand, during the constant monitoring period, the feature quantity data is not collected in the basic unit 3, but the feature quantity data is transmitted from each unit to the analysis unit 4b, and anomalies are detected in real time. In addition, when an anomaly such as a state different from normal is detected in the image data stored in the moving image memory 37e of the camera input unit 4a, the CPU 31 can use the transferable time period of the expansion bus 90 to collect the image data of the camera image and save the image data to the first buffer 37a and the memory card 44. In addition, in the collection process of S3, preferably, based on features such as cycle attributes, the device values of each device and the feature quantity data of the camera image (i.e., these respective collected time series data) are classified so that the analysis can be easily performed using predetermined parameters in subsequent analysis processing. Details of this process will be described later with reference to Figures 23 to 24B the details of this process.

[0104] In S4, the CPU 31 (transmission unit 53a) saves the data collected in the first buffer 37a in S3 to the memory card 44 in response to the satisfaction of the save condition, and transmits the time series data stored in the memory card 44 to the analysis unit 4b via the expansion bus 90. For example, when a predetermined amount of collected data is stored in S3, the transmission process can be performed. That is, whenever a sufficient amount of data (time series data of a predetermined number of scan cycles) required for the monitoring process of the analysis unit 4b accumulates in a storage unit such as a buffer or the memory card 44, the transmission process is executed. Although an example of temporarily saving the data collected from the first buffer 37a to the memory card 44 and transmitting it to the analysis unit 4b is described here, it is not intended to limit the subject matter, and the data can be saved to a non-volatile memory different from the memory card, or the data can be transmitted from the first buffer 37a to the analysis unit 4b without being saved to such a memory.

[0105] In S5, the CPU 41b (data processing unit 73 and monitoring unit 76) obtains the time-series data and feature quantity data of the device values of each device stored in the memory card in the operation record analysis, and performs preprocessing and analysis processing on the obtained feature quantity data. In addition, during the constant monitoring period, the CPU 41b also collects the device values of a predetermined device and the feature quantity data from the camera input unit 4a, and monitors changes different from normal. Here, the constant monitoring is basically performed according to the monitoring timing defined by the devices in the device memory, but can be implemented at various levels such as real time, continuous, and scan time level (scan cycle). "Abnormal" indicates that the feature quantity data of the device value and the camera image to be analyzed is different from the value in the normal state, that is, deviated from the normal data, and does not necessarily indicate an abnormality occurring in the programmable logic controller. Details of S5 will be described later with reference to Figures 14 to 16 Describe the details of S5.

[0106] In S6, the CPU 41b (display control unit 74) generates display data indicating the monitoring result based on the monitoring result of the monitoring performed by the monitoring unit 76, and displays the generated display data on the display unit 5 of the basic unit 3, or transmits the generated display data to the PC 2b or an external device via the communication unit 43 through the transmitting unit 79. For example, the monitoring result is displayed to the operator in the PC 2b. Note that the settings such as the monitoring items can be changed via the displayed confirmation screen, and the display screen can be changed in real time according to the setting change. Although the example of constantly collecting and constantly monitoring the predetermined device value and feature quantity data has been described as the monitoring form for performing constant monitoring, this is not intended to limit the subject matter. The following methods can be used: monitoring when the device value previously determined by the user changes in a predetermined manner; periodically monitoring the device value at the set cycle; or analyzing whether there is an abnormality in the device value, etc. based on the monitoring instruction from the user. Details of the processing of S6 and the confirmation screen will be described later.

[0107] <Feature quantity data>

[0108] Figure 12 It is a diagram showing an example of a method for extracting the feature quantity of a camera image according to the present embodiment. Here, reference will be made to Figure 12 A method for generating feature quantity data for various types of feature quantities is described. Note that the method shown here is an example, and any other method can be used in this subject according to the type of feature quantity. In this embodiment, an example of extracting the feature quantity in a predetermined region in a camera image will be described, but the feature quantity of the entire camera image can be extracted, or the feature quantity of multiple regions can be extracted. The region to be extracted can be changed for various types of feature quantities. Examples of feature quantity types include luminance component, color difference (blue) component, color difference (red) component, R component, G component, B component, and their degrees of change.

[0109] Reference numeral 1201 denotes a method for obtaining the average value of the feature quantity related to luminance for each pixel from the image data. First, the feature quantity generation unit 54b divides the camera image into one or more regions, and designates the region to be the object according to the set predetermined region. One region includes one or more pixel regions. In addition, the feature quantity generation unit 54b cuts out the designated predetermined region for each of the components R, G, and B, and obtains the average value of the luminance of each pixel included in the cut-out region. Reference numeral 1202 denotes a method for obtaining the gradient intensity and gradient angle for each region of the feature quantity related to luminance from the image data. Similarly to 1201, the feature quantity generation unit 54b divides the camera image into one or more regions, and designates the region to be the object according to the set predetermined region. In addition, the feature quantity generation unit 54b cuts out the designated predetermined region for each of the components R, G, and B, and obtains the intensity and angle of the horizontal and vertical differential values (gradients) of each region.

[0110] Reference numeral 1203 denotes a method for obtaining the change amount (degree of change) obtained by further differentiating the feature quantities shown in 1201 and 1202 in the time direction. Reference numeral 1231 denotes the time series data of the feature quantity extracted from the camera image. Reference numeral 1232 denotes a graph obtained by differentiating the time series data shown in 1231 and squaring the differential values. Therefore, as shown in 1232, the change amount can be confirmed more significantly.

[0111] Reference numerals 1201 and 1202 represent feature amounts in a predetermined region of an image, and reference numeral 1203 represents a feature amount in its time direction. Note that the present subject matter is not limited to these extraction methods, and various other extraction methods can be applied. Preferably, any extraction method is applied according to the characteristics of the monitoring object. For example, when it is necessary to measure the size of an object photographed as a feature amount, it is necessary to calculate a more advanced feature amount. In this case, it is preferable to perform edge processing on the image in a predetermined region specified by the user or at a specified timing to measure the size. Although an example of using one camera sensor 10 has been described in the present embodiment, multiple camera sensors can be used to measure distances and sizes in a three-dimensional space. Further, character recognition can be performed on a specified region of the photographed camera image by an optical character reader (OCR), and the extracted character string (numerical value) can be used as a feature amount. For example, in the case of a photographed image of a thermometer, a barometer, or various other measuring instruments, the numerical value obtained by character recognition can be used as a feature amount. Note that a recognized character string other than a numerical value can be used as a feature amount. As another feature amount, a neural network can be used to extract a feature amount. For example, in a neural network, image parameters related to the position, operation, and posture of a person or workpiece to be photographed can be learned as teacher data, and by using the learned model, coordinates of a person or workpiece or joint angles of a posture can be extracted as feature amounts.

[0112] <Camera Image Acquisition and Feature Amount Generation Processing Procedure>

[0113] Figure 13 is a flowchart showing a detailed processing procedure for acquiring a camera image and generating a feature amount in S2 above according to the present embodiment. The processing described below is executed by the CPU 41a of the camera input unit 4a.

[0114] First, in S21, the CPU 41a acquires the set setting information from the basic unit 3 or the analysis unit 4b. The setting information includes various parameters used when generating a feature amount as image processing settings. The parameters include, for example, parameters related to the set predetermined region (zone number, coordinates, etc.) and parameters indicating the type of feature amount to be extracted. In S22, the CPU 41a stores the latest acquired setting information in the memory 42a.

[0115] Next, in S23, the CPU 41a (feature quantity generation unit 54b) acquires, via the camera input unit 45, the image data of the camera image that has been captured by the camera sensor 10 and stored in the fourth buffer 37d of the memory 42a. Next, in S24, the CPU 41a (feature quantity generation unit 54b) divides the acquired camera image into regions of a predetermined size, and cuts out a predetermined region as the monitoring object in the camera image according to the setting information stored in S22. The cut-out region may have a rectangular shape or may have a shape other than a rectangular shape. When the cut-out region has a shape other than a rectangular shape, this specification is made by providing an operator with a setting screen (wherein the divided regions can be specified individually). Additionally, a masking region to be excluded from the object for generating the feature quantity can be set in the cut-out region. By providing such a specification method, it is possible to specify the removal of pixels or regions that do not change in the time direction, and it is possible to monitor the change in the feature quantity in the object region with higher accuracy.

[0116] In S25, the CPU 41a (feature quantity generation unit 54b) extracts a predetermined feature quantity from the cut-out predetermined region according to the setting information stored in S22, and generates feature quantity data based on the image data. When generating the feature quantity data, the CPU 41a (feature quantity generation unit 54b) stores the generated feature quantity data in the fourth buffer 37d of the memory 42a in S26. The generation of the feature quantity of the image data is performed for the camera images sequentially captured by the camera sensor 10. The stored feature quantity data is collected by the basic unit 3 and sequentially accumulated in S3 above. The sequentially accumulated data is transmitted to the analysis unit 4b as time series data in S4. Each feature quantity data in the time series data is stored in association with a common counter (e.g., timer value, time information, etc.) in the PLC 1, and is synchronized with the device value of other devices according to the counter. This can be achieved by setting the camera input unit 4a in the PLC layer. For example, in the case of using a network camera as an external device instead of the setting in the PLC layer, when synchronizing using the time information in each device, the accuracy of synchronization is reduced due to errors, and further, a delay occurs due to the data transmission / reception of the camera image via the network, and high-precision synchronization cannot be achieved. On the other hand, by setting the camera input unit in the PLC layer, each unit in the PLC layer can operate according to the operation clock of the basic unit, and high-precision synchronization can be easily achieved.

[0117] In S27, the CPU 41a (feature quantity generation unit 54b) determines whether a setting change has been made via a setting screen (setting screen 1700) described later. When there is no setting change, the process returns to S23 to acquire the next camera image. On the other hand, when a setting change has been made, the CPU 41a (feature quantity generation unit 54b) proceeds to S28, acquires the changed image processing settings (for example, change information of the region for generating feature quantities), returns the process to S22, and stores the acquired latest setting information in the memory 42a. Therefore, subsequent feature quantity generation can be performed using the latest setting information. As described in the reference Figure 12 Feature quantity data can be generated using a method corresponding to the type of feature quantity to be generated. An additional description of feature quantity generation will be given. The feature quantity generation unit 54b can derive a motion quantity from the generated feature quantity and binarize the motion quantity. For example, values indicating whether there is a change for each pixel can be indicated by 0 and 1, and the values in the monitoring region can be averaged. In this case, anomaly detection can be performed as a bit signal, and it is easy to detect a state different from normal. In addition, the feature quantity generation unit 54b can generate a feature quantity by averaging the values in a predetermined region in the camera image. Therefore, the time-series data of the feature quantity in the camera image does not have to be classified as an analog value in the classification of the collected data described later.

[0118] <Classification of Collected Data>

[0119] Figure 23 Examples of the types of time-series data to be collected are shown. Reference numerals 2301 to 2306 each represent time-series data of various types of device signals. Note that the types described below are only examples and are not intended to limit the present subject matter, and the present subject matter can be applied to other types. In the present subject matter, in order to improve the classification effect, the types used for classification preferably include at least two of these multiple types.

[0120] Reference numeral 2301 denotes a device signal of a device that operates in synchronization with an operation cycle of a device. Reference numeral 1210 denotes the operation cycle of the device. The type 2301 indicates a type in which the same change pattern (stable change pattern) occurs for each operation cycle of the device. In this type, for example, such a monitoring method is assigned, in which a stable change pattern is identified from time series data acquired during a time period that can be regarded as a normal state, and an abnormal device (a device different from normal) is identified based on a deviation from the stable change pattern identified from newly acquired time series data. Specifically, a time change related to a change point is measured for a change pattern of time series data for each of a plurality of cycles that can be regarded as a normal state, and a threshold value is set according to the measured change. Based on whether a change point of time series data acquired thereafter exceeds a threshold range set corresponding to the change point, a monitoring algorithm for detecting an abnormal device is assigned. For example, a plurality of waveforms for each operation cycle of the device are superimposed, and a reference value and a threshold value of the change point are set based on a change at a point where the device changes from off to on, so that these parameters can be used to identify an abnormal device. In each cycle of 2301, a relative time indicating a rising timing of a first signal or a phase in each cycle can be used as an evaluation variable, and a parameter corresponding to the evaluation variable can be determined based on a change of the evaluation variable. For example, when a phase in each cycle is used as an evaluation variable, an upper threshold value and a lower threshold value of the phase can be used as parameters.

[0121] On the other hand, reference numeral 2302 denotes a device signal of a device that operates not in synchronization with a scan cycle but in synchronization with a cycle other than the operation cycle of the device. Also for this type, for example, such a detection method is assigned, in which a stable change pattern is identified from time series data acquired during a time period that can be regarded as a normal state, and an abnormal device (a device different from normal) is identified based on a deviation from the stable change pattern identified from newly acquired time series data. Specifically, a time change related to a change point is measured for a change pattern of time series data for each of a plurality of cycles that can be regarded as a normal state, and a threshold value is set according to the measured change. Based on whether a change point of time series data acquired thereafter exceeds a threshold range set corresponding to the change point, a detection algorithm for detecting an abnormal device is assigned. In each cycle of 2302, a relative time indicating a rising timing of a second signal or a phase in each cycle can be used as an evaluation variable, and a parameter corresponding to the evaluation variable can be determined based on a change of the evaluation variable. For example, when a relative time in each cycle is used as an evaluation variable, an upper threshold value and a lower threshold value of the relative time can be used as parameters respectively.

[0122] Reference numeral 2303 denotes the device signal of a device that uses a constant value. In this type, for example, such a detection method is assigned, in which the normal value is identified from the time series data obtained during a time period that can be regarded as the normal state, and newly obtained time series data that is different from the value identified as normal is identified as an abnormal device (a device different from the usual). Specifically, a detection algorithm is assigned, in which the value of the device in the time series data that can be regarded as the normal state is set as the detection reference value, and an abnormal device is detected based on whether the value of the time series data obtained thereafter is different from the detection reference value set corresponding to the normal value. In other words, when the value (constant value) changes, the value can be judged as abnormal. Reference numeral 2304 denotes the device signal of a device that operates irregularly. This type is excluded from the data used when identifying an abnormal device. Reference numeral 2305 denotes the device signal of a device that uses an analog value. The image data obtained from the camera sensor 10 is classified into this type. In a device that uses an analog value, as described below, when the device value is not the device value of a bit device, the process enters Figure 24A the "No" of S31 in Figure 24B S37), and as Figure 24B shown, the algorithm in abnormal detection differs according to the method for changing data. As algorithms in abnormal detection of a device that uses an analog value, various methods such as the dynamic time warping method and the autoregressive model can be used. Reference numeral 2306 denotes the device signal of a monotonically increasing and monotonically decreasing device. In this type, the differential value and the integral value are used to identify an abnormal device. In addition, the relative time indicating the timing of monotonic increase or decrease or the phase in each cycle can be used as an evaluation variable. Note that the types described here are only examples, and it is also assumed that there are devices classified into other types. For example, some devices can take multiple states (such as a stepped device signal, etc.). When changing to a state different from the normal state, a device that takes multiple states can be judged as abnormal.

[0123] Each device includes a 0, 1-bit type device, and a word type device or a floating-point type device that uses an analog value. Note that a word device also includes an unsigned integer of 1 word (0 to 65535), a signed integer of 1 word (-32768 to 32767), an unsigned integer of 2 words (0 to 4294967295), and a signed integer of 2 words (-214783648 to 214783647). Considering this type, for example, when the change rate per scan cycle is equal to or greater than a predetermined value and the type is not a bit, the device can be judged as a device that uses an analog value.

[0124] <Classification processing>

[0125] Figure 24A and 24B It is a flowchart showing the processing procedure when classifying the data collected in S3 above. The processing described below is implemented by the CPU 31 of the basic unit 3. However, this is not intended to limit the subject matter, and a part of the processing may be executed in the expansion units 4a, 4b, 4c, or may be executed by an external device (analysis device) communicatively connected to the programmable logic controller. Note that the processing described below is an example of a learning process that is automatically performed when collecting device values, and this process can always be learned when collecting device values, but it can also be performed periodically or within a certain time period based on an instruction from the user.

[0126] In S31, the CPU 31 determines whether the time series data to be classified is a device value of a bit device. When the device value is a device value of a bit device, the process proceeds to S32; otherwise, the process proceeds to Figure 24B S37.

[0127] In S32, the CPU 31 determines whether there is a change in the value of the time series data to be classified. When there is a change, the process proceeds to S34, and when there is no change and a constant value is adopted, the process proceeds to S33. In S33, the CPU 31 classifies the time series data to be classified as a device (2303) that adopts a constant value, and ends the process. As classification information, information associated with the time series data is stored. For example, flag information indicating the classification type is stored in association with the time series data or discriminant information indicating the device corresponding to the time series data. As described herein, the flag information of the type classified at the time of data collection classification and the discriminant information of the device are associated and stored in advance, so that in the analysis process, based on which device the time series data to be analyzed is from, and based on the type associated with the discriminant information of the device, it is easy to select an evaluation variable or parameter of the detection algorithm for the abnormal device. That is, in the analysis process, the process of specifying which type the time series data to be verified corresponds to can be omitted. Although the flag information indicating the classification type has been shown as an example of the classification information, algorithm information indicating the algorithm for identifying the abnormal device can be used instead of the flag information. When the device corresponding to the time series data is a variable, needless to say, the discriminant information stored as the classification information is the discriminant information indicating the variable.

[0128] In S34, when there is a change in the value of the time series data, the CPU 31 determines whether the change is a stable change pattern in each cycle. When it is determined that the pattern is a stable change pattern, the process proceeds to S35, and when it is determined that the pattern is not a stable change pattern, the process proceeds to S36. Here, the stable change pattern indicates a pattern that changes in the same pattern in each cycle. In S35, the CPU 31 classifies the device as a device that operates at a predetermined cycle and ends the process. As described above, the devices that operate at a predetermined cycle include the device (2301) that operates in synchronization with the operation cycle of the equipment and the device (2302) that operates in synchronization with a cycle other than the operation cycle of the equipment, and the CPU 31 classifies the device as each device. On the other hand, in S36, the CPU 31 classifies the device as a device that operates irregularly (2304) and ends the process.

[0129] On the other hand, when it is determined in S31 that the device is not a bit device, the process proceeds to Figure 24B S37 shown, and the CPU 31 determines whether the time series data to be classified has changed by a predetermined value or more. Here, it is determined whether the extreme value of the change signal is equal to or greater than the predetermined value. When there is a change, the process proceeds to S38; otherwise, the process proceeds to S312.

[0130] In S38, the CPU 31 determines whether the value changes at a predetermined cycle, and when the value does not change at a predetermined cycle, the process proceeds to S39, and when the value changes, the process proceeds to S310. In S310, the CPU 31 determines whether the change value is monotonically increasing or monotonically decreasing. When the change value is monotonically increasing or monotonically decreasing, the process proceeds to S311; otherwise, the process proceeds to S39.

[0131] In S39, the CPU 31 classifies the device as a device with an analog value (2305) and ends the process. There are various types of devices with analog values, and separate correspondences are made according to the type of the device to be classified. On the other hand, in S311, the CPU 31 classifies the device as a device with a monotonically increasing or decreasing value (2306) and ends the process. In S312, the CPU 31 classifies the device as another device and ends the process. When it is determined that the device is another device, the characteristics of the time series data to be classified cannot be extracted, and the characteristic operation in the normal state cannot be recognized, so that the device is not used when identifying an abnormal device.

[0132] The CPU 31 preferably determines parameters for identifying abnormal devices based on the classified time series data. Here, this determination process is performed for each classification type, and the types and parameters of the devices corresponding to the time series data are stored in association with each other. When there are already determined parameters, the parameters can be updated with the parameters determined this time to replace the already determined parameters, or the already determined parameters and the parameters determined this time can be used to calculate and update new parameters. Various methods such as the average value can be applied to the calculation method, and a method suitable for each time series data is selected. The determined parameters can be changed by the operator. Therefore, even when determining extreme value parameters in the case of a short learning period, adjustments can be made.

[0133] Here, the parameters to be determined will be described. For Figure 23 types 2301 and 2302 (each device whose value changes in a stable mode synchronously with a predetermined period), for example, the CPU 41b calculates the time between arbitrary change points of the waveforms of each operation cycle of the device based on the change mode of the waveforms within the operation cycle of the device, and determines the threshold of the time between the change points as a parameter based on the time distribution between the change points. In addition, the parameters in this case can include the timing of the change in the device value of the time series data during normal times. The above-mentioned threshold of the time between the change points as a parameter can be determined, for example, by calculating the standard deviation and the average value based on the changes during learning, setting the average value + 3 × standard deviation as the upper threshold, and setting the average value - 3 × standard deviation as the lower threshold. For type 1203 (devices that adopt a constant value), the CPU 41a uses this constant value as a parameter. As described above, for devices whose values change in a stable mode synchronously with a predetermined period, the time interval between the change points, the timing of the change points, and the related thresholds are used as parameters.

[0134] For type 1205 (analog values), for example, the value of the time series data during learning and the threshold related to the distance from the time series data during learning can be used as parameters. The calculation method of the distance can be, for example, a method for obtaining the sum of the differences of each point. In this case, during evaluation, the distance between the time series data during learning and the time series data during evaluation is calculated, and it is judged whether the distance is within the range of the threshold. For type 1206 (monotonically increasing, monotonically decreasing), for example, the increased value during normal monotonic increase and the decreased value during normal monotonic decrease are determined as parameters.

[0135] As described above, according to the present embodiment, based on the periodic characteristics of the time series data of each device, each device is classified into one of a plurality of predetermined types, and a detection algorithm applicable to the abnormality (change different from normal) of each classification type is applied. That is, in the present embodiment, in the learning stage, each device is classified into one of a plurality of predetermined types according to the time series data in the normal state. In addition, a detection algorithm is determined according to each type, and evaluation variables and parameters corresponding to each detection algorithm are set. On the other hand, in the evaluation stage, for each device for which the detection algorithm has been determined in the learning stage, the detection algorithm determined according to the type of the evaluation target device is applied to the time series data of the device. Therefore, an abnormal device is identified by analyzing the time series data of the device using the detection algorithm of the abnormal device corresponding to the type of the device determined in the learning stage or the type of the device. Note that preferably, a device that is not classified into the predetermined type at the time of learning is not taken as a device to be collected at the time of evaluation because no evaluation variable or parameter is set. In other words, the above is to prevent the following situation: even if such data is collected, since no detection algorithm for identifying whether the device is an abnormal device is set, the processing load and the consumption of memory resources are unnecessarily increased.

[0136] <Monitoring process (preprocessing / analysis process)>

[0137] Figure 14 FIG. is a flowchart showing a detailed processing procedure of the monitoring process (preprocessing / analysis process) in S5 according to the present embodiment. The processing described below is executed by the CPU 41b of the analysis unit 4b. Here, as an example of preprocessing, a description will be given of a process of cutting out time series data of a predetermined section as a monitoring object by synchronizing the time series data of the device value with the time series data of the feature amount data. However, this is not intended to limit the subject matter. For example, the data can be monitored as it is without cutting out the data, or the data can be subjected to moving statistics in the time axis direction. For example, the data can be filtered with a filter for the average value or the maximum value for each 10-second section.

[0138] First, in S51, the CPU 41b acquires the time series data of the device value of each device transmitted from the basic unit 3 in S4 and the feature amount data of the camera image captured by the camera sensor 10. In addition, the CPU 41b analyzes the acquired time series data of the device value and the feature amount, classifies the data based on the characteristics of each device, and determines a detection algorithm. Specifically, when acquiring the data, the CPU 41b first executes a classification process. The classification process is related to the reference Figure 24A and 24B This is the same as the classification process in the described learning phase. In addition, the CPU 41b classifies the types of each data to be monitored, and then obtains various parameters (such as thresholds for analysis, etc.) held in association with that type. These parameters are used in the abnormality determination in S55 described later.

[0139] In S52, the CPU 41b (data processing unit 73) performs preprocessing on each time series data acquired in S51. In the preprocessing, an association is made so that the data can be processed as synchronous data, and a monitoring target interval is cut out from each of the time series data of the device values of the predetermined devices in synchronization and the time series data of the feature quantity data. That is, instead of monitoring all the data of the time series data, for example, a cut is made so that the time series data in a predetermined interval with a device value change is monitored, and improved monitoring is achieved while reducing the processing load. Reference will be made to Figure 15 and 16 describe the details of the preprocessing. Note that all time periods of the acquired time series data can be monitored without cutting out the time series data.

[0140] In S53, the CPU 41b (monitoring unit 76) determines whether there is a predetermined change in the time series data of the feature quantity that has undergone preprocessing in S52. Here, the predetermined change means a change that exceeds the threshold range set for each feature quantity included in the setting information set from the basic unit 3 or the analysis unit 4b. The threshold range is determined based on the separation from the data in the normal interval set by the user. For example, the standard deviation and the average value can be calculated, and the upper limit can be set to the average value + 3 × the standard deviation, and the lower limit can be set to the average value - 3 × the standard deviation. Alternatively, the threshold range can be determined based on the number of times a predetermined threshold has been exceeded. When there is no predetermined change, the process returns to S51, and the process proceeds to the next data processing. On the other hand, when it is determined that there is a predetermined change, the process proceeds to S54, and the CPU 41b (monitoring unit 76) saves the time series data of the feature quantity that has undergone preprocessing in S52 into the memory 42b. Here, the time series data is saved in association with the predetermined device value.

[0141] Next, in S55, the CPU 41b (monitoring unit 76) analyzes the time-series data of the device values and characteristic quantities based on the detection algorithms such as types and parameters obtained in the above S51, and determines whether a change different from the normal is detected. When an abnormality is detected, various abnormalities can be determined, for example, in the following cases: the degree of change in S53 is equal to or greater than a predetermined value; the number of changes detected in a predetermined interval is equal to or greater than a predetermined value; and the relationship with the changes in other device values is different from the normal. In addition, the relationships between different characteristic quantities can be analyzed, such as the difference between characteristic quantity A and characteristic quantity B, the correlation value of multiple characteristic quantities, and the Mahalanobis distance. Optionally, the number of changes in the device value of another device when the characteristic quantity of the camera is within a certain range can be determined. When there is no abnormality, the process returns to S51, and the process proceeds to the next data processing. On the other hand, when there is an abnormality, the process proceeds to S56, and the CPU 41b (monitoring unit 76) outputs the following as a log record: there is a change different from the normal. Note that when the user program satisfies the previously determined storage conditions, this log record can be saved. For the recording location, it can be stored in the storage device 32 of the basic unit 3 or in the memory 42b of the analysis unit 4b. As the information stored as a log record, for example, in addition to the time-series data of the characteristic quantities, the corresponding image data can also be saved. The synchronized time-series data of the device values can be saved. In addition, this save can be notified to the user via the PCs 2a and 2b. Optionally, an external device can be notified by email or the like. Subsequently, in S57, the CPU 41b (monitoring unit 76) saves the logs from the first buffer 37a of the basic unit 3 and the buffers 37b and 37c of the analysis unit 4b to the memory card 44 of the basic unit 3 and the memory card (not shown) of the analysis unit 4b.

[0142] In S58, the CPU 41b (monitoring unit 76) analyzes the anomalies by using the data recorded in S57, specifies whether a predetermined anomaly or a device anomaly has occurred and the factors of the anomaly, and ends the process. Even when a change different from normal is detected, it is not always necessary to specify that an emergency anomaly has occurred in S58. The degree of the anomaly can be specified. The analysis result is reflected in the result display in S6 above. For the specification of the anomaly factors, a learned model learned from various teacher data during the anomaly can be used, or preset anomaly factors preset according to feature quantities having changes different from normal can be specified. For the anomaly in the device value, the type of the signal can be classified according to the periodic property and continuity of the time series data, and judgment criteria can be provided according to each classified type to specify whether a device anomaly has occurred or its factors. The types to be classified include, for example, signals of devices operating synchronously with the scan cycle, devices operating asynchronously, devices adopting constant values, devices operating irregularly, devices increasing monotonically and decreasing monotonically, and devices adopting analog values. These classifications can also be performed by using a learned model learned from normal time series data as teacher data. These learned models can be learned by using data determined to be normal or data determined to be abnormal during the monitoring of the PLC 1.

[0143] <Cut-out of data>

[0144] In this embodiment, the designation of the reference device is received, and data in a predetermined interval is cut out from the time series data of the feature quantity data. For example, when device A is designated as the reference device, data in the corresponding interval is cut out from the time series data of the feature quantity data having a period from the timing when the device value of device A changes from off to on to the next timing when the device value changes from off to on. Hereinafter, a method of cutting out data by using two reference devices will be described. Figure 15 It is a diagram for explaining the data cut-out. Here, a method (period cut-out) of cutting out data in a predetermined period from the time series data of the feature quantity data of the camera image in association with predetermined reference devices (device A and device B) will be described.

[0145] Reference numeral 1501 denotes time series data of apparatus A, reference numeral 1502 denotes time series data of apparatus B, reference numeral 1503 denotes feature quantity data of a camera image, and the respective time series data are in a synchronized state. As shown at 1501, the apparatus values of apparatus A and apparatus B have a periodic property. Here, by using the portion from the change of apparatus A from off to on to the change of apparatus B from off to on as a separation section, the feature quantity data of the camera image is cut out as a monitoring object 1504, and other intervals are set as non-monitoring objects 1505. This makes it possible to exclude (mask) intervals that are not preferably used as monitoring objects. For example, it is possible to prevent a decrease in the accuracy of anomaly detection due to a pause during operation or a person's hand appearing in the camera image, thereby reducing false detection. Note that there are various cutting methods, and for example, an interval from the change of a predetermined apparatus from off to on to the change from on to off can be used as a cutting interval. Although the waveform of a predetermined interval is cut out here, the value of the feature quantity data of the camera image at the timing when the apparatus value of the reference apparatus changes can be obtained.

[0146] <Processing flow of preprocessing>

[0147] Figure 16 is a flowchart showing a detailed processing procedure of the preprocessing in S52 according to the present embodiment. The processing described below is executed by the CPU 41b of the analysis unit 4b. Here, the preprocessing in the case of using the cutting method described with reference Figure 15 will be described.

[0148] In S521, the CPU 41b (data processing unit 73) sequentially acquires the apparatus value of apparatus A as a first reference apparatus. In S522, the CPU 41b (data processing unit 73) determines whether the apparatus value acquired in S521 has changed from off to on. When the apparatus value changes, the process proceeds to S523, and when the apparatus value has not changed, the process returns to S521 to acquire the next apparatus value.

[0149] In S523, the CPU 41b (data processing unit 73) simultaneously acquires and accumulates the apparatus value of apparatus A acquired in S521 and the feature quantity of the feature quantity data. Next, in S524, the CPU 41b (data processing unit 73) acquires the apparatus value of apparatus B. In S525, the CPU 41b (data processing unit 73) determines whether the apparatus value acquired in S524 has changed from off to on. When the apparatus value changes, the process proceeds to S526, and when the apparatus value has not changed, the process returns to S523 to acquire the next apparatus value.

[0150] In S526, the CPU 41b (data processing unit 73) compares the waveform of the feature amount of the camera image accumulated in S523 with the waveform in the normal state. Here, the comparison result is temporarily held, and the held information is used for the determination in S53 described above. In this way, by performing period cutting of the waveform of the feature amount of the camera image in a predetermined section and comparing the cut-out waveform with the waveform in the normal state, changes can be detected with high precision. As a comparison method, for example, there are a method for obtaining the Euclidean distance of each point and calculating the sum, and a method for calculating the distance between two waveforms by dynamic time warping (DTW), etc. When a change is detected, an abnormality is detected as described above in S55.

[0151] <Monitoring item setting screen>

[0152] Figure 17 FIG. is a diagram showing an example of a monitoring item setting screen according to the present embodiment. The setting screen 1700 is displayed on the display unit 7b of the PC 2b at the above timing S1 or the like. Note that the screen 1700 may be displayed on the PC 2a or other external devices. The display timing is not limited to the timing of S1 described above, but is set to an arbitrary timing. For example, the display of the confirmation screen in S6 described above can be changed to the setting screen 1700. When displayed on the PC 2a, when the setting of the monitoring item is indicated on the setting screen of the item data, the setting screen 1700 can be changed or displayed as a pop-up window. In addition, the setting screen is related to the setting of abnormality detection, and there are cases where important parameters that affect the safety guaranteed by the PLC system are set and cases where the operator at the site inadvertently changes the parameters from the setting screen, so it is preferable to perform authentication processing using operation authority or the like when the setting screen is displayed.

[0153] The setting screen 1700 includes various displays 1701 to 1718. When there are multiple monitoring settings, the display 1701 indicates which monitoring setting is being used. When there are multiple monitoring areas, settings can be made by specifying each monitoring area. The display 1706 is a button for enabling or disabling the status of the monitoring operation. In addition to the settings for the setting screen 1700, it is also possible to select whether to enable monitoring based on the output status of the human-machine interface (HMI), physical buttons, or sensors. In this case, it is possible to determine whether the analysis unit 4b performs monitoring by changing the value of a specific area of the device memory. The display 1702 indicates the status of the monitoring operation. "Working" is displayed when monitoring is enabled, and "Stopped" is displayed when monitoring is disabled. The display 1703 indicates the monitoring status. In response to the result of the monitoring judgment, when the abnormal condition (monitoring condition) is not satisfied after the start of monitoring, "OK" is displayed, and when the condition is satisfied even once, "NG" is displayed until the abnormal detection is cleared. The abnormal detection indicates the detection of the above-mentioned change different from normal. The display 1704 indicates the number of times an abnormality has been detected, that is, the NG detection count. This indicates the cumulative number of times an abnormality has been detected since the monitoring status became "NG". By default, the display shows the total number of monitoring conditions 1 and 2 described later, but this is not intended to limit the subject matter and can be changed by specifications, etc. The display 1705 is a button for clearing the NG detection count. When cleared, the monitoring status of the display 1703 becomes "OK", and the NG detection count is reset.

[0154] The display 1721 shows a check box for automatically adjusting and displaying the time-series data of the characteristic quantity of the monitoring object in the display 1707. As a calibration (scale) adjustment method, the following can be applied: the case of setting the upper and lower limits of the display range based on the maximum and minimum values of the data acquired in the past; the method of setting the upper and lower limits of the display range based on the maximum and minimum values of the range currently displayed on the screen; etc. When the automatic calibration is turned off, the upper and lower limits to be displayed can be selected according to the user's input. The display 1720 is used to display the component to be monitored that is currently selected on the setting screen 1700. The display 1707 shows a real-time monitoring screen of the characteristic quantity to be monitored. In Figure 17 In the example, the object to be monitored is the luminance component, and a line graph of time-series data of all image information obtained for each frame based on the scanning period for the luminance component is displayed on the monitoring screen. The broken line indicated by the dotted line is the value of the previous period or the average value so far. The solid line indicates the value of the current period. By displaying the difference so as to be comparable with the value of the past period, the difference can be easily recognized by the operator. Display 1708 indicates a button for indicating magnification / reduction of the feature amount. "1x" is the magnification display. The magnification is decreased by the "-" button and increased by the "+" button. When the display is magnified, a scroll bar for moving and displaying the display area is displayed on the graph. Instead of the scroll bar, operation buttons for moving the display area in the left-right direction may be displayed. In addition to these buttons, a "stop button", a "resume button", and a "button for saving the current screen display" may be provided. Thus, the user can confirm the measurement state, such as the motion monitoring waveform, the camera image, and the feature amount, from the feature amount graph at a desired timing. Displays 1709 to 1711 show the "current value" indicating the current feature amount, the minimum value "MIN", and the maximum value "MAX" of the data obtained after the monitoring is enabled. The user can easily set the monitoring conditions by confirming these displays. Display 1712 shows a button for clearing the maximum value and the minimum value. When the operation button is operated, the maximum value and the minimum value are updated with newly obtained data.

[0155] Display 1713 displays a real-time image captured by the camera sensor. The data thinned out at intervals in consideration of the processing load may be displayed on the video, or the data may not be thinned out according to the processing performance of the PLC 1. In the camera input unit 4a, the image data itself is acquired for each frame and stored in the buffer, but since the image data itself is monitored via the Web server by communication, it is difficult to match the time precisely. Therefore, there is no particular problem in displaying the image data after thinning out at intervals. Figure 17 The image capture shown illustrates how to correct the skew of the workpiece being conveyed at a predetermined position when the workpiece is conveyed in the factory. Display 1713 shown within display 1719 is a frame for defining a predetermined area in which a feature amount is generated in the camera image captured by the camera sensor 10. For example, the size of the predetermined area can be adjusted by dragging and operating the square points using an indicating device or the like. Here, a state in which the position for correcting the skew of the workpiece is set as the predetermined area is shown. At this position, the workpiece being conveyed is supported by the member from above, and the skew of the workpiece is corrected from the right-hand direction of the image by the pusher member. The area defined by display 1719 is notified to the camera input unit 4a and applied to the generation of the feature amount data of the subsequently acquired camera image. In Figure 17 In the example, the predetermined area defined by the display 1719 is rectangular, but it does not necessarily have to be rectangular. For example, the divided areas can be specified individually. In addition, a masking area can be specified within the defined predetermined area. The masking area represents an area to be excluded from the monitoring area where the feature amount is generated and monitored.

[0156] The setting content will be described. The display 1720 indicates the currently selected monitoring object. In Figure 17 the luminance component is selected. When the change button is operated, the parameter selection screen 1900 shown in Figure 19 is displayed. In the parameter selection screen 1900, a list 1902 of the types of feature amounts that can be selected is displayed, and a display example of the time series data of each feature amount is shown together with the type. Each feature amount is extracted and saved from the image data by the camera input unit 4a when the camera image is input from the camera sensor 10, so these data can be used to display the feature amount. Therefore, the user can confirm which feature amount has changed and selectively monitor the feature amount. The user can select a desired feature amount by selecting the bar indicating the type of each feature amount and operating the OK button 1903. On the other hand, when the cancel button 1904 is operated, no feature amount is selected and the screen returns to the setting screen 1700. Multiple feature amounts can be made selectable. In this case, the multiple feature amount data are displayed on the display 1707. When the feature amount changes, the monitoring state 1703 and the NG detection count 1704 can be reset and displayed, and the NG detection count 1704 can be not reset and the cumulative count from the start can be displayed. In the case of reset, the information after the change to the feature amount setting can be seen, while in the case of the cumulative count, the number of times of detecting "a state different from normal" after the start of monitoring can be seen.

[0157] The description returns to Figure 17 On displays 1715 and 1716, setting areas for monitoring condition 1 and monitoring condition 2 are respectively displayed. At least one monitoring condition must be selected for monitoring. Of course, multiple monitoring conditions can be selected simultaneously. For example, when monitoring condition 1 and monitoring condition 2 are selected, a logical operation (default is OR) is performed on the NG detection results. When the monitoring condition is satisfied, NG detection counting is performed. Although two monitoring conditions are described here as an example, other monitoring conditions can be added, and the logical operation of multiple monitoring conditions can also be set. In each monitoring condition, for example, "peak value" and "on-off count" etc. as monitoring items, the threshold value of the monitoring item, "detect more or" and "detect less or" etc. as the judgment conditions of the threshold value can be set. When setting the threshold value of the peak value, the threshold value can be displayed on the monitoring of display 1707. Therefore, the user can intuitively set the threshold value. As another monitoring condition, a guard band, the distance relative to a reference signal, a specific timing such as Euclidean or DTW etc. can be set. The guard band judges whether the waveform to be monitored is within the defined area (for example, within a range with an upper limit and a lower limit). The distance relative to the reference signal judges the distance from the preset reference value. The specific timing judges whether the value after a defined time has elapsed from the start of the period falls within the specified range. The defined value of the monitoring condition can be a value related to the device value of other devices. When the characteristic quantity changes, it is changed so that the threshold value corresponding to the characteristic quantity can be set.

[0158] Display 1717 is a setting area for cycle cut-out. When "cycle cut-out" is enabled, the cycle is automatically cut out according to the conditions set on the cycle cut-out setting screens 2000 and 2010 described later, and a graph of the cut-out characteristic quantity data is displayed on the monitoring of display 1707. In "timeout", the time for one cycle when cycle cut-out is disabled can be set. One cycle is determined according to the time set in "timeout", and a graph of the cut-out characteristic quantity data is displayed on the monitoring of display 1707. When cycle cut-out is enabled, regardless of the set timeout time, the cycle is automatically cut out according to the information set in the cycle cut-out setting, but when an abnormality occurs, the end of the cycle may not be cut out. In this case, the cycle is cut out according to the time set in timeout. Time is adopted as "timeout" here, but this is not intended to limit the subject matter, and for example, the definition can be such that when a certain amount of data is accumulated in the buffer, it is judged that one cycle has passed. For example, when cycle cut-out is enabled and no timeout time is set, it can be judged that the cycle has ended when the data size that can be buffered is reached. Optionally, by rewriting the buffer, the process can be performed infinitely until other cycle end conditions are met. The cycle cut-out setting screen will be referred to later Figure 20 and 21 describe the cycle cut-out setting screen.

[0159] Display 1718 indicates the confirmation button for the set monitoring items. When the confirmation button is operated, a monitoring operation is performed based on the setting information. After the confirmation button is operated, display 1713 is switched so that changes such as changes in the use of the predetermined area of display 1719 and changes in the setting of monitoring conditions cannot be made on the displayed monitoring screen.

[0160] <Monitoring status confirmation screen>

[0161] Figure 18 It is a diagram showing an example of a confirmation screen for confirming the monitoring status according to the present embodiment. The confirmation screen 1800 is displayed on the display unit 7b of the PC 2b at the above timing S6. Note that the screen 1800 may be displayed on the PC 2a or other external devices. The confirmation screen 1800 is a screen for confirming the monitoring status among the contents set on the setting screen 1700, and is different from the setting screen 1700 in that monitoring items cannot be set. Therefore, the same display as the setting screen 1700 is given the same reference numeral and its description is omitted.

[0162] In displays 1707 and 1713, the feature amount data and the real-time image captured by the camera sensor 10 are displayed in real time according to the setting information set on the setting screen 1700. In display 1707, the time series data of the device values of other related devices can be displayed in a synchronized state. Therefore, the user can check the changing state in more detail.

[0163] As shown in displays 1802 to 1804, the setting contents of the monitoring conditions 1, 2, and cycle cutout cannot be changed on the confirmation screen 1800, and the setting contents set on the setting screen 1700 are displayed. On the other hand, when the setting change button of display 1801 is operated, the display changes to the setting screen 1700, and settings can be made again. In this way, according to the present embodiment, settings can be changed even during monitoring, and when the confirmation button is operated on the setting screen 1700, subsequent monitoring is performed according to the set monitoring conditions.

[0164] In addition, on the confirmation screen 1800, the normal camera image and feature amount data, as well as the abnormal camera image and feature amount data, can be displayed so as to be comparable to each other. In this case, it is preferable to emphasize and display, for example, the abnormal positions in the camera image and the feature amount data. For these displays, a slider bar can be displayed to display the camera image and the feature amount data at a specified time. The time series data of the device values of other corresponding devices can be displayed in association with each other. This can be achieved by holding the camera image, its feature amount data, and the time series data of other corresponding devices at the timing when a change different from the normal is detected. Although examples of displaying the result image have been described here, a report including the display and its analysis result can be created and stored as a log or sent to a predetermined destination. Therefore, even when the operator does not confirm the analysis result in real time, the analysis result at the time of abnormality can be confirmed later.

[0165] <Cut-out setting screen>

[0166] Figure 20 FIG. is a diagram showing an example of a periodic cut-out setting screen according to the present embodiment. For example, the periodic cut-out setting screen 2000 is a screen displayed by a pop-up screen or the like when periodic cut-out is selected in the display 1717 of the setting screen 1700. The periodic cut-out setting screen 2010 is a screen that is transitioned when "Perform detailed setting" of the display 2003 is selected in the periodic cut-out setting screen 2000.

[0167] In the periodic cut-out setting screen 2000, the display 2001 is a display capable of selecting a reference device. As described above, the reference device is a device that serves as a reference for cut-out when monitoring target intervals of feature amount data are cut out from the camera image during preprocessing and corresponds to other device values. Here, the state of selecting Device A is shown. The display 2002 is a display for setting the signal change used as the cut-out reference. As an example, Figure 20 the states of being able to select signal rise (from off to on), signal fall (from on to off), and the interval where the signal value is on are shown. Other signal changes may be selectable. Although binary signal values are shown here, it can be set corresponding to the signal values of multi-value (analog value) device values. In this case, a predetermined threshold can be set, and the case where the value exceeds the threshold or the case where the value drops below the threshold can be set as the cut-out reference. When "Perform detailed setting" of the display 2003 is operated, it transitions to the periodic cut-out setting screen 2010.

[0168] In the cycle cut-out setting screen 2010, display 2011 is for setting the cut-out start timing. Here, by default, device A, which is the reference device set on the cycle cut-out setting screen 2000, and its cut-out reference are displayed. Note that each item among them can be changed on this screen. The cut-out reference can be changed by selecting the position of the signal change as the reference position using an indicating device or the like. Display 2012 is for setting the cut-out end timing. Here, the device having the signal value used as the reference for the cut-out end timing and its cut-out reference can be set. In the example shown in Figure 20 , device B is displayed as the reference device for the cut-out end timing, but usually, by default, device A, which is set as the reference device on the cycle cut-out setting screen 2000, is displayed, and it can be changed to other devices by pulling down. Display 2013 is for setting the interval to be masked as a non-monitoring object. For user operations, for example, the start position of the masking interval can be defined by dragging the object signal value using an indicating device or the like, the range of the masking interval can be defined by moving the object signal value in the dragged state, and the end of the masking interval can be defined by releasing the object signal value. In the example shown in Figure 20 , the device having the signal value to be used as the reference and its masking interval 2014 can be set. Among the setting contents set in the cycle cut-out setting screen 2010, the interval from the rise of the signal of device A to the second fall of the signal of device B is set as the cycle cut-out interval, and the interval from the second fall of the signal of device B to the rise of the signal of the next device A is set as the masking interval 2014. Note that the number of setting items can be reduced by making the cycle start condition and the cycle end condition part common. For example, when a specific device is turned on, the object devices for the cycle start condition and the cycle end condition are the same, which is equivalent to setting the rise and fall separately, and the number of setting items can be reduced.

[0169] As described above, the programmable logic controller (PLC) according to the present embodiment includes: an execution engine for repeatedly executing a user program; a device memory for storing device values of devices accessed by the execution engine according to the user program; and a camera input interface for connecting a camera sensor, the camera sensor sequentially capturing images according to shooting conditions and sequentially generating image data of the captured camera images. The PLC sequentially obtains the image data of the camera images from the camera sensor via the camera input interface, and generates feature quantity data indicating the feature quantity of the image data in a preset monitoring area in the obtained image data of the camera images. In addition, the PLC sequentially collects the generated feature quantity data to obtain time series data of the feature quantity, and monitors the obtained time series data of the current feature quantity according to the monitoring timing defined by the devices in the device memory. Therefore, according to the present embodiment, the camera images for monitoring are appropriately obtained and analyzed, and the monitoring object can be constantly monitored with high accuracy. For constant monitoring, the feature quantity data of the camera images can be collected synchronously with the scan cycle for collecting the device values of each device, or can be associated with the device values of a predetermined device based on the timestamps of the collected feature quantity data. These controls can be easily implemented because the unit as the PLC is configured to obtain camera images and generate the feature quantity of the images.

[0170] In addition, the PLC performs settings related to the image data of the camera images captured by the camera sensor via a setting screen capable of receiving user input. The PLC generates and outputs a confirmation screen for displaying the monitoring result. Optionally, the PLC can create a report and send it to a predetermined destination. It is also possible to transition from the confirmation screen to the setting screen and change the settings related to the monitoring object during monitoring. When the settings are changed, the monitoring result changes according to the changed setting content.

[0171] <Modification Example>

[0172] Here, reference will be made to Figure 22 describe a modification example of the setting screen 1700 for display. Figure 22 Shows the state of calling the setting screen 1700 for performing monitoring settings etc. from the setting screen 2200 for project data. The setting screen 2200 can perform various settings for project data, which are settings related to the user program, settings related to unit configuration, collection settings, monitoring settings, camera settings, etc. The setting screen 2200 is a setting screen displayed on the PC 2a connected to the basic unit 3, but the setting screen 1700 for performing monitoring settings can also be read from the setting screen 2200, and an example of displaying the setting screen 1700 on the setting screen 2200 is shown here. This is not restricted, and when monitoring settings are indicated on the setting screen 2200, it is possible to transition to the setting screen 1700 as a separate screen. As described above, the setting screen 1700 can be displayed on the PC 2a connected to the basic unit 3, or can be displayed on a PC (Web server) connected to an expansion unit such as the analysis unit 4b.

[0173] <Second Embodiment>

[0174] Hereinafter, a second embodiment of the present subject matter will be described. In the first embodiment, an example in which the PLC layer includes the basic unit 3, the camera input unit 4a, and the analysis unit 4b was described. However, the present subject matter is not limited to such a configuration, and a part of the functions of each unit can be implemented by other units. For example, the camera input unit 4a and the analysis unit 4b can be provided integrally, or all the units can be provided integrally. Further, in the present embodiment, the analysis unit 4b can be provided as an external device other than the PLC layer, and the form in such a case will be described in the present embodiment.

[0175] Figure 21 Is a diagram showing a configuration in which the analysis unit according to the present embodiment is provided outside the PLC layer. The same reference numerals are given to the same components as in the first embodiment, and their description is omitted. As Figure 21 shown, in the present embodiment, the communication unit 4c as an expansion unit is connected instead of the analysis unit 4b in the first embodiment. Further, the analysis device 2100 as an external device is communicably connected via the communication unit 4c. The analysis device 2100 can be connected to the upper layer of the PLC layer to which the camera input unit is connected. By separating the analysis unit from the PLC layer in this way, space savings can be achieved, and high-level calculations such as PC and cloud computing can be performed. On the other hand, although it is necessary to transmit the data to be monitored to the analysis device 2100, which increases the communication load, the communication volume can be reduced by transmitting not the image data itself but the feature amount data obtained by extracting the feature amount.

[0176] The device values of each device and the feature quantity data of the camera image collected by the basic unit 3 are transmitted to the analysis device 2100 via the communication unit 4c. More specifically, the communication unit 4c receives the transmission data from the basic unit 3 via the expansion bus 90 and transmits the transmission data to the analysis device 2100 via the communication unit 43a. The communication unit 43a also transmits image data (image data). When the image data is used for monitoring, the transmission frequency of the image data may be lower than that of the device value and the feature quantity data. Note that it is preferable to add the time to the feature quantity data. Thus, when the analysis device 2100 detects an abnormality, the occurrence time can be displayed.

[0177] The analysis device 2100 receives the transmission data in the communication unit 43b through a network 2101 such as a local area network (LAN) and performs a monitoring process based on the received transmission data. For the monitoring process, the same control is performed with the same configuration as the analysis unit 4b of the first embodiment, so the description thereof is omitted. The monitoring result is displayed on the display 2102 of the analysis device 2100. The display control unit 74 displays the setting screen 1700 and the confirmation screen 1800 on the display 2102. The monitoring result can be transmitted to the PLC 1 via the communication unit 43b. The analysis device can be provided integrally with the PC 2b. In this case, the confirmation screen 1800 and the setting screen 1700 can be displayed on the display of the analysis device.

[0178] <Third Embodiment>

[0179] Hereinafter, a third embodiment of the present subject will be described. In the first embodiment, an example of performing constant monitoring has been described, but the present subject is not limited to a configuration for performing constant monitoring. For example, instead of performing the monitoring process in real time or in addition to performing the monitoring process in real time, the remaining information in the memory card 44 or the like can be used as log information for later analysis.

[0180] As described above, when the save condition is satisfied, the device values of each device and the feature quantity data are saved to the memory card 44 of the basic unit 3 or the memory card (not shown) of the analysis unit 4b. For example, when an abnormality is detected in the captured image of the camera sensor 10, when an abnormality occurs in the PLC, when the user pre-specifies a device to be used as a save trigger and there is a predetermined change in the device, when the user issues an analysis command, when the monitoring cycle for periodic monitoring has elapsed, when an analysis command is issued from an expansion unit connected to the basic unit 3, and in some other cases, the save condition is satisfied.

[0181] In this way, when the analysis process is performed later based on the log information stored in a memory card or the like, the abnormal device and the abnormal factor can be specified by an external device having high processing capabilities, and the analysis can be performed more efficiently. Further, as described above, in the case of constant monitoring, the device value of a predetermined device and the feature amount data of the camera image are the monitoring objects, but the device values of all devices can be stored as log information. Therefore, a method can be adopted in which constant monitoring is performed until an abnormality occurs in a predetermined device, and when an abnormality occurs in the predetermined device, a more detailed analysis is performed based on the log information of each device in all devices.

[0182] The present subject matter is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the subject matter.< / plc>

Claims

1. A programmable logic controller, comprising: An execution engine for repeatedly executing a user program; A device memory for storing device values of devices to be accessed by the execution engine according to the user program; A camera input interface for connecting a camera sensor that sequentially captures images according to shooting conditions and sequentially generates image data of the captured camera images; An image processing unit for sequentially acquiring image data of camera images from the camera sensor via the camera input interface and generating feature quantity data indicating feature quantities of image data in a preset monitoring area in the acquired image data of the camera images; A time series data acquisition unit for sequentially collecting the feature quantity data from the image processing unit and acquiring time series data of the feature quantities; And A monitoring unit for monitoring the time series data of the current feature quantities acquired by the time series data acquisition unit according to a monitoring timing defined by a device in the device memory, wherein the time series data acquisition unit also sequentially collects device values from the device memory to acquire time series data of the device values, and the monitoring unit monitors the time series data of the current feature quantities acquired by the time series data acquisition unit and the time series data of the current device values acquired by the time series data acquisition unit according to a monitoring timing defined by a device in the device memory.

2. The programmable logic controller according to claim 1, further comprising: A collection unit for collecting device values of each device stored in the device memory at a predetermined collection period to accumulate the collected device values in a first buffer memory, collecting the feature quantity data generated by the image processing unit at the predetermined collection period to accumulate the collected feature quantity data in a second buffer memory, and saving the time series data of the device values of each device accumulated in the first buffer memory and the time series data of the feature quantities accumulated in the second buffer memory to a storage memory according to a saving condition, wherein the monitoring unit also performs monitoring using the time series data of the device values of a predetermined device saved to the storage memory by the collection unit and the time series data of the feature quantity data obtained from the image data acquired from the camera input interface.

3. The programmable logic controller according to claim 1, further comprising: A collection unit for collecting device values of each device stored in the device memory at a predetermined collection period to accumulate the collected device values in a first buffer memory, collecting the image data of the camera images sequentially acquired from the camera sensor via the camera input interface at the predetermined collection period to accumulate the collected image data in a second buffer memory, and saving the time series data of the device values of each device accumulated in the first buffer memory and a plurality of image data accumulated in the second buffer memory to a storage memory according to a saving condition, Among them, the monitoring unit also performs monitoring using the time-series data of the device values of a predetermined device stored by the collection unit in the storage memory and the time-series data of the feature quantity data obtained from the image data stored in the storage memory.

4. The programmable logic controller according to claim 2, wherein, the monitoring unit monitors whether the feature quantity data is abnormal, and when the abnormality of the feature quantity data is set as the storage condition and the monitoring unit detects the abnormality, the collection unit stores the time-series data of the device values of each device accumulated in the first buffer memory and the time-series data of the feature quantity or a plurality of image data accumulated in the second buffer memory in the storage memory.

5. The programmable logic controller according to claim 2, wherein, The predetermined collection period is the scan period during which the execution engine accesses the device memory.

6. The programmable logic controller according to claim 2, wherein, the collection unit stores the device values of each device stored in the device memory together with the time stamp at the time of acquisition in the first buffer memory, and the monitoring unit performs monitoring using the time-series data of the device values of a predetermined device collected and accumulated by the collection unit and the time-series data of the feature quantity data corresponding to the time stamp.

7. The programmable logic controller according to claim 6, wherein, the feature quantity includes at least one of a luminance component, a blue color difference component, a red color difference component, an R component, a G component, a B component, and the degree of change in these components, and the image processing unit extracts the average value of at least one of the feature quantity, the gradient intensity, and the gradient angle in the monitoring area.

8. The programmable logic controller according to claim 7, further comprising: a setting unit configured to perform settings related to monitoring, including the position of the monitoring area and the type of feature quantity, wherein the image processing unit generates feature quantity data indicating the feature quantity of the image data in the monitoring area according to the position of the monitoring area and the type of feature quantity received by the setting unit.

9. The programmable logic controller according to claim 7, further comprising: a setting unit configured to receive settings related to monitoring, including the positions of a plurality of monitoring areas, wherein the image processing unit generates feature quantity data indicating the feature quantity of the image data in each monitoring area according to the positions of the plurality of monitoring areas received by the setting unit, the time-series data acquisition unit sequentially collects the feature quantity data for each monitoring area from the image processing unit, and acquires the time-series data of the feature quantity for each monitoring area, and the monitoring unit monitors the time-series data of the current feature quantity and the time-series data of the current device value for each monitoring area acquired by the time-series data acquisition unit according to the monitoring timing defined by the device in the device memory.

10. The programmable logic controller according to claim 8, wherein, The setting unit performs settings related to the image data of the camera image captured by the camera sensor via a setting screen capable of receiving user input.

11. The programmable logic controller according to claim 10, wherein, The setting screen displays the camera image captured by the camera sensor and enables setting of the monitoring area on this display.

12. The programmable logic controller according to claim 10, wherein, Components to be extracted as feature quantities and thresholds for detecting abnormalities can be set on the setting screen.

13. A programmable logic controller, comprising: An execution engine for repeatedly executing a user program; A device memory for storing device values of devices that the execution engine is to access according to the user program; A camera input interface for connecting a camera sensor that sequentially captures images according to shooting conditions and sequentially generates image data of the captured camera images; An image processing unit for sequentially acquiring image data of the camera images from the camera sensor via the camera input interface and generating feature quantity data indicating feature quantities of the image data in a preset monitoring area in the acquired image data of the camera images; A time series data acquisition unit for sequentially collecting the feature quantity data from the image processing unit and acquiring time series data of the feature quantities; A monitoring unit for monitoring the time series data of the current feature quantities acquired by the time series data acquisition unit according to a monitoring timing defined by a device in the device memory; And An output unit for outputting a monitoring result for confirming the monitoring state of the monitoring unit, wherein the output unit generates and outputs a confirmation screen for confirming the monitoring state, and wherein a camera image in which an abnormality is detected and an abnormal position in the feature quantity data are highlighted and displayed on the confirmation screen.

14. The programmable logic controller according to claim 13, wherein, It is possible to transition from the confirmation screen to a setting screen for making settings related to the image data of the camera image captured by the camera sensor.

15. A programmable logic controller, comprising: An execution engine for repeatedly executing a user program; A device memory for storing device values of devices that the execution engine is to access according to the user program; A camera input interface for connecting a camera sensor that sequentially captures images according to shooting conditions and sequentially generates image data of the captured camera images; An image processing unit for sequentially acquiring image data of the camera images from the camera sensor via the camera input interface and generating feature quantity data indicating feature quantities of the image data in a preset monitoring area in the acquired image data of the camera images; A time series data acquisition unit for sequentially collecting the feature quantity data from the image processing unit and acquiring time series data of the feature quantities; And A monitoring unit for monitoring the time series data of the current feature quantities acquired by the time series data acquisition unit according to a monitoring timing defined by a device in the device memory, wherein the execution engine and the device memory are provided in a basic unit of the programmable logic controller, the camera input interface and the image processing unit are provided as a first expansion unit to which the camera sensor is connected, and The monitoring unit is provided as a second expansion unit different from the first expansion unit, and the second expansion unit collects device values and characteristic quantity data of a predetermined device during a constant monitoring period.

16. The programmable logic controller according to claim 15, wherein, The first expansion unit and the second expansion unit are integrally provided.

17. The programmable logic controller according to claim 15, wherein The second expansion unit is a communication unit capable of communicating with an analysis device, the analysis device being capable of communicating via a network, and the second expansion unit sends data to and receives data from the analysis device via the network to implement the function of the monitoring unit.

Images