Information processing system, information processing method, and program

Abstract

To provide an information processing system capable of appropriately supporting searching for the cause of abnormal operation of a monitoring target.SOLUTION: An information processing system performs information processing using time-series data on values of one or more variables representing the state of a monitoring target. The information processing system includes a simulator that simulates the behavior during a target period of a virtual model corresponding to the monitoring target in virtual space using the values of one or more variables indicated by the time-series data in the target period to generate a virtual moving image obtained by capturing of a virtual camera arranged in the virtual space. The information processing system also includes an identification module for identifying abnormality occurrence time at which a difference between a first position of a moving object in an actual moving image obtained by capturing the monitoring target in the target period and a second position of a virtual object corresponding to the moving object in the virtual moving image exceeds a reference.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] A device has been developed that supports the investigation of the cause of abnormal behavior of a monitored object such as production equipment. JP 2022-99341 A (Patent Document 1) discloses a technology for displaying video data obtained by filming the monitored object and simulated behavior reproduction data that reproduces the behavior of a virtual monitored object on a display unit. The simulated behavior reproduction data is generated based on log data that records, in chronological order, the state of the monitored object and input / output data of control signals between the monitored object and a control device that controls the monitored object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-99341 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-97426 Summary of the Invention [Problem to be solved by the invention]

[0004] According to the technology disclosed in Patent Document 1, a user finds the time when an abnormality occurs (hereinafter referred to as the "abnormal time") in real video shown by video data or virtual video shown by operation reproduction simulation data, and checks the state of the monitored object before and after that time. However, if the shooting time is long or if there are many cameras, it takes time to visually find the abnormal time in the real video or virtual video. Therefore, it takes time to find the cause of the abnormal operation of the monitored object.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an information processing system, an information processing method, and a program that can appropriately assist in searching for the cause of abnormal behavior of a monitored object. [Means for solving the problem]

[0006] According to an example of the present disclosure, an information processing system performs information processing using time-series data of values ​​of one or more variables representing a state of a monitored object. The information processing system includes a simulator and an identification module. The simulator simulates the behavior of a virtual model corresponding to the monitored object in a virtual space during a target period indicated by the time-series data, and generates a virtual video obtained by capturing images with a virtual camera placed in the virtual space. The identification module identifies an abnormal time at which a difference between a first position of a moving object in a real video obtained by capturing images of the monitored object during the target period and a second position of a virtual object corresponding to the moving object in the virtual video exceeds a reference value.

[0007] According to this disclosure, a user can easily search for the cause of the abnormal behavior of a monitored object by investigating in detail the state of the monitored object at the time of the abnormality identified by a specific module and before and after the time. In this way, the information processing system according to the present disclosure can appropriately support the search for the cause of the abnormal behavior of a monitored object.

[0008] In the above disclosure, the information processing system further includes an output module that outputs data showing a list of abnormal times. According to this disclosure, a user can easily recognize abnormal times using the list.

[0009] In the above disclosures, the monitored object includes a marker placed at a fixed position. The simulator sets the position and orientation of the virtual camera in the virtual space so that the position of the marker in the real video coincides with the position of a virtual marker corresponding to the marker in the virtual video. According to these disclosures, positional deviation between the virtual video and the real video is suppressed.

[0010] According to another example of the present disclosure, an information processing method uses time-series data of values ​​of one or more variables representing a state of a monitored object. The information processing method includes: a processor simulating, in a virtual space, the behavior of a virtual model corresponding to the monitored object during a target period indicated by the time-series data, and generating a virtual video obtained by capturing images with a virtual camera disposed in the virtual space. The information processing method further includes the processor identifying an abnormal time at which a difference between a first position of a moving object in the real video obtained by capturing images of the monitored object during the target period and a second position of a virtual object corresponding to the moving object in the virtual video exceeds a reference value.

[0011] According to yet another example of the present disclosure, a program causes a computer to execute the above information processing method.

[0012] These disclosures also enable the information processing method and program to appropriately assist in identifying the cause of abnormal behavior of a monitored object. [Effects of the Invention]

[0013] According to the present disclosure, an information processing system, an information processing method, and a program can appropriately assist in searching for the cause of abnormal behavior of a monitored object. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a diagram illustrating an example of a manufacturing system to which an information processing system according to an embodiment is applied; [Figure 2] 1 is a schematic diagram illustrating an example of the overall configuration of a manufacturing system according to an embodiment. [Figure 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a control device according to an embodiment. [Figure 4] FIG. 1 is a diagram illustrating a specific example of a configuration of a monitoring target. [Figure 5] FIG. 2 is a diagram illustrating processing performed by a visual sensor. [Figure 6] 10 is a diagram showing an example of variables indicating the measurement results of the visual sensor 330. FIG. [Figure 7] FIG. 10 is a diagram illustrating an example of a software configuration of a PC related to an analysis process of an abnormal time. [Figure 8] FIG. 10 is a diagram illustrating an example of correspondence relationship information. [Figure 9] FIG. 10 is a diagram illustrating an example of initial information. [Figure 10] FIG. 10 is a diagram illustrating a method for setting the position of a virtual camera. [Figure 11] FIG. 10 is a diagram illustrating a method for identifying an abnormal time by the identification module. [Figure 12] 10 is a flowchart showing the first half of the flow of an abnormal time analysis process in a PC. [Figure 13] 10 is a flowchart showing the second half of the flow of the abnormal time analysis process in the PC. [Figure 14] FIG. 10 is a diagram showing an example of a window displayed on the display in step S20. [Figure 15] FIG. 10 is a diagram illustrating another method for generating a virtual work. [Figure 16] FIG. 10 is a diagram illustrating an example of the software configuration of a PC related to the visualization process of data for a target period. [Figure 17] FIG. 10 is a diagram illustrating an example of an operation window. [Figure 18] FIG. 10 is a diagram showing an example of a screen displayed on a display. DETAILED DESCRIPTION OF THE INVENTION

[0015] Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals, and the description thereof will not be repeated. In this specification, unless otherwise specified, the term "or" includes the meaning of "and" in addition to the meaning of "or."

[0016] §1 Application Examples 1 is a diagram illustrating an example of a manufacturing system to which an information processing system according to an embodiment is applied. As shown in FIG. 1, the manufacturing system 1 includes an information processing system 100, a control device 200, a monitored object 300, and a camera 400.

[0017] The control device 200 is typically a programmable logic controller (PLC). The control device 200 is connected to the monitored object 300, and controls the monitored object 300 in accordance with a control program created in advance.

[0018] The monitored object 300 may include an actuator that exerts some physical effect on a production facility or production line (hereinafter referred to as the "field"), various sensors installed in the field, or an input / output device that exchanges information with the field.

[0019] The control device 200 holds the values ​​of one or more variables that represent the state of the monitored object, and controls the monitored object 300 by performing calculations using the values ​​of the one or more variables. The one or more variables may include various variables acquired from the monitored object 300 and various variables output to the monitored object 300.

[0020] The values ​​of the one or more variables are updated every control cycle. The control device 200 records the values ​​of the one or more variables every control cycle and generates time-series data 11 of the values ​​of the one or more variables during a target period in which some abnormality occurs. The time-series data 11 is transferred from the control device 200 to the information processing system 100.

[0021] The one or more cameras 400 are installed so as to capture at least a portion of the monitoring target 300. Actual video data 12 obtained by capturing images with the one or more cameras 400 is transferred to the information processing system 100. Note that the number of cameras 400 is not particularly limited. An embodiment in which the manufacturing system 1 includes one camera 400 will be described below.

[0022] The information processing system 100 performs information processing using time-series data 11 and real video data 12. The information processing system 100 is configured with one or more computers. The information processing system 100 includes a database 10, a simulator 20, and a specific module 30.

[0023] The database 10 stores time-series data 11 of values ​​of one or more variables that represent the state of the monitoring target 300 during a target period. Furthermore, the database 10 stores real video data 12 obtained by capturing images with a camera 400. The real video data 12 represents a real video 12a obtained by capturing images of the monitoring target 300 during a target period.

[0024] The simulator 20 simulates the behavior of a virtual model corresponding to the monitored object 300 in a virtual space during a target period, using values ​​of one or more variables during the target period indicated by the time-series data 11. Furthermore, the simulator 20 generates a virtual video 28a obtained by capturing images with a virtual camera placed in the virtual space.

[0025] The identification module 30 identifies an abnormal time when the difference between the first position of the workpiece 3 in the real video 12a and the second position of the virtual workpiece 3M corresponding to the workpiece 3 in the virtual video 28a exceeds a reference value. The workpiece 3 is an example of a "moving object" in the present disclosure. The virtual workpiece 3M is an example of a "virtual object" in the present disclosure.

[0026] According to this embodiment, the user can easily search for the cause of the abnormal behavior of the monitored object 300 by investigating in detail the state of the monitored object 300 at the time of the abnormality identified by the identification module 30 and before and after.

[0027] In this way, the information processing system 100 according to this embodiment can appropriately assist in searching for the cause of abnormal behavior of the monitored object 300.

[0028] §2 Specific examples <Overall configuration of the manufacturing system> 2 is a schematic diagram showing an example of the overall configuration of a manufacturing system according to an embodiment. As shown in FIG. 2, the manufacturing system 1 includes a personal computer (hereinafter referred to as "PC 100A") constituting an information processing system 100, a control device 200, a monitored object 300, a camera 400, and a storage medium 500. The PC 100A, the control device 200, and the camera 400 can communicate with each other using, for example, a general-purpose Ethernet (registered trademark). The PC 100A and the control device 200 may also communicate with each other using another communication means (for example, a Universal Serial Bus (USB)).

[0029] The storage medium 500 includes a volatile storage medium, a nonvolatile storage medium, a general-purpose semiconductor storage device such as a CF (Compact Flash) or an SD (Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory). The storage medium 500 is, for example, an SD memory card.

[0030] The PC 100A is typically a computer having a general-purpose architecture. As shown in Fig. 2, the PC 100A includes a processor 101 such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a memory 102 configured as a volatile storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), a storage 103, an input interface 104, a display controller 105, a communication interface 106, and a reader / writer 107. These components are connected to each other via a bus so as to be able to communicate data with each other.

[0031] The processor 101 reads out a program stored in the storage 103 and loads it into the memory 102. The processor 101 executes the loaded program.

[0032] The storage 103 is configured by a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), or a flash memory. The storage 103 stores the program executed by the processor 101 and the database 10.

[0033] The programs executed by the processor 101 include a development program 131, a viewer program 132, a data reproduction program 133, and a simulation program 134.

[0034] The development program 131 mainly includes a set of instructions for supporting the development of a control program to be used by the control device 200. Furthermore, the development program 131 provides a setting function related to the generation of the time-series data 11 and a function for monitoring the execution status of the control program by the control device 200 (hereinafter referred to as a "program monitoring function").

[0035] A control program developed using the development program 131 is installed in the control device 200. Furthermore, monitoring setting data generated using the setting function of the development program 131 is transferred to the control device 200.

[0036] The viewer program 132 includes a set of commands for visualizing various types of information and providing it to the user. The viewer program 132 provides a function for displaying a graph showing the transition of the values ​​of one or more variables (hereinafter referred to as a "variable graph display function") and a function for playing video.

[0037] The data reproduction program 133 includes a group of commands for reproducing past data, and includes a function for managing time and a function for requesting data to be reproduced.

[0038] The simulation program 134 includes a group of commands for simulating the behavior of a virtual model of the monitored object 300 in a virtual space. The simulation program 134 includes a function for reproducing the behavior of the virtual model corresponding to the monitored object 300, a script for moving a virtual workpiece corresponding to a moving object not controlled by the control device 200, a data management function, and an analysis function.

[0039] The script is written in, for example, C# format and includes a start method and a render method. The start method is executed only once when the script starts to run. The render method is executed periodically.

[0040] The database 10 stores time-series data 11. The time-series data 11 is transferred from the control device 200 and indicates actual values ​​at each time of one or more variables that represent the state of the monitored object 300 during a target period. Furthermore, the database 10 stores actual video data 12 transferred from the camera 400.

[0041] The input interface 104 mediates data transmission between the processor 101 and an input device 110 such as a keyboard, a mouse, or a touch panel. The display controller 105 is connected to a display 120 and displays on the display 120 a screen represented by screen data generated by the processor 101. The communication interface 106 communicates with the control device 200 via a communication network. The reader / writer 107 mediates data transmission between the processor 101 and an external storage medium (which may include, for example, storage medium 500).

[0042] Each of the development program 131, viewer program 132, data reproduction program 133, and simulation program 134 may be provided as a part of an arbitrary program rather than as a standalone program. In this case, the processing according to this embodiment is realized in cooperation with the arbitrary program. Furthermore, some or all of the functions provided by the development program 131, viewer program 132, data reproduction program 133, and simulation program 134 may be realized by dedicated hardware circuits (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array)).

[0043] <Control device hardware configuration> 3 is a schematic diagram showing an example of a hardware configuration of a control device according to an embodiment. As shown in FIG. 3, control device 200 includes, as main components, a power supply circuit 201 that supplies power PW to each unit of control device 200, a CPU 202, a chipset 204, a memory 206, a storage 208, a USB controller 250, a field network controller 252, a network controller 254, a timer 260, and a reader / writer 270.

[0044] The CPU 202 reads the program stored in the storage 208 and loads it into the memory 206. The CPU 202 executes the loaded program.

[0045] The memory 206 is configured with a volatile storage device such as a DRAM or an SRAM. The memory 206 holds the values ​​of one or more variables. The values ​​of the one or more variables held in the memory 206 are updated according to a control period by the CPU 202 executing a program stored in the storage 208.

[0046] The values ​​of one or more variables are stored in a buffer area 207 of a memory 206 for a certain period of time.

[0047] The storage 208 is configured, for example, by a non-volatile storage device such as an HDD, an SSD, or a flash memory, but a portion of the storage device may also be configured by a volatile storage device.

[0048] The chipset 204 mediates the exchange of data between the CPU 202 and each component, thereby realizing the processing of the control device 200 as a whole.

[0049] The storage 208 stores a system program 211 including an OS (Operating System) 212 and a scheduler program 213 for implementing the basic functions of the control device 200.

[0050] Furthermore, the storage 208 stores a program data set 214 created by the PC 100A. The program data set 214 includes a control program and sampling setting data.

[0051] The control program defines a process for collecting the values ​​of various variables from the monitored object 300 and a process for performing calculations based on the collected variable values ​​to generate the values ​​of various variables to be output to the monitored object 300. The values ​​of the various variables are temporarily stored in memory 206. By executing the control program, CPU 202 updates the values ​​of the variables stored in memory 206 at each control cycle.

[0052] The sampling setting data defines the conditions for generating the time series data 11. The conditions for generating the time series data 11 indicate the trigger conditions for sampling, the variables to be sampled, the sampling time, and the like.

[0053] The trigger condition includes, for example, a condition that the value of a variable related to an abnormality in the monitored object 300 has changed to a specific value. The CPU 202 reads, from the buffer area 207, time-series data 11 indicating the values ​​of one or more variables during a target period including the timing at which the trigger condition is satisfied, and controls the reader / writer 270 to write the time-series data 11 to the storage medium 500. The target period includes, for example, a first predetermined period before the timing at which the trigger condition is satisfied and a second predetermined period after the timing at which the trigger condition is satisfied. The lengths of the first predetermined period and the second predetermined period are defined by the sampling setting data.

[0054] The USB controller 250 is responsible for transferring data to and from external devices (which may include the PC 100A) via a USB connection.

[0055] The field network controller 252 has a connector 252a that connects to the field network, and controls the exchange of data with the monitored object 300. The field network controller 252 has an internal buffer that stores input data received from the field network and output data output from the control device 200 to the field network.

[0056] The network controller 254 has a connector 254a for connecting to a higher-level network, and controls the exchange of data between the control device 200 and an external device (which may include the PC 100A).

[0057] The timer 260 is configured to include a clock circuit or a counter circuit that measures time, but is not limited to such circuits and may be configured by a software module executed by the CPU 202.

[0058] The reader / writer 270 mediates data transmission between the CPU 202 and the storage medium 500. The CPU 202 copies the time series data 11 generated according to the generation conditions indicated by the sampling setting data from the buffer area 207 to the storage medium 500. The time series data 11 stored in the storage medium 500 is transferred to the PC 100A via the network controller 254 at a predetermined timing. Alternatively, the user may connect the storage medium 500 to the PC 100A and copy or move the time series data 11 from the storage medium 500 to the PC 100A.

[0059] <Specific examples of monitoring targets> 4 is a diagram showing a specific example of the configuration of a monitored object 300A. As shown in FIG. 4, a monitored object 300A includes a transport system 310, a robot 320, and a visual sensor 330.

[0060] The transport system 310 transports a workpiece 3, which is an example of a moving object of the present disclosure. The transport system 310 includes an encoder 312. A variable indicating the measurement result of the encoder 312 is collected by the control device 200. The variable value indicating the measurement result of the encoder 312 is referred to as an "encoder value."

[0061] The robot 320 performs processing on the workpiece 3. The robot 320 operates in accordance with various variable values ​​(command values) received from the control device 200. Variable values ​​(state values) representing the current state of the robot 320 are collected by the control device 200.

[0062] The visual sensor 330 performs image measurement processing such as pattern matching on an image acquired from a camera 332 installed so as to capture an image of a partial area 313 of the transport system 310 .

[0063] FIG. 5 is a diagram explaining the processing content of the visual sensor. When the camera 332 captures an image, the visual sensor 330 outputs an on-state image capture flag to the control device 200. The image capture flag is one of the variables collected by the control device 200. The visual sensor 330 acquires an image from the camera 332. The image is obtained by capturing an image of a partial area 313 of the conveyance system 310. The visual sensor 330 measures the number of workpieces 3 captured in the image and the position of each workpiece 3. Furthermore, the visual sensor 330 may determine the type of workpiece 3 captured in the image. The values ​​of the variables indicating the measurement results of the visual sensor 330 are collected by the control device 200.

[0064] FIG. 6 is a diagram showing an example of variables indicating the measurement results of the visual sensor 330. As shown in FIG. 6, the values ​​of the variable "controller name," the variable "detection flag of workpiece 3," the variable "number of detected workpieces 3," the variable "X coordinate of workpiece 3," the variable "Y coordinate of workpiece 3," the variable "rotation angle of workpiece 3," and the variable "type of workpiece 3" are collected from the visual sensor 330. The values ​​of the variable "X coordinate of workpiece 3," the variable "Y coordinate of workpiece 3," the variable "rotation angle of workpiece 3," and the variable "type of workpiece 3" are measured for each detected workpiece 3. Therefore, the variable "X coordinate of workpiece 3," the variable "Y coordinate of workpiece 3," the variable "rotation angle of workpiece 3," and the variable "type of workpiece 3" constitute array variables.

[0065] As shown in FIG. 4, the camera 400 is installed so as to capture an image of an area 410 downstream of an area 313 of the transport system 310.

[0066] The monitored object 300A includes three or more markers 390 arranged at fixed positions. The three or more markers 390 are arranged within an area 410. Therefore, the three or more markers 390 appear in each frame of the actual video 12a captured by the camera 400.

[0067] <PC software configuration for analyzing abnormal times> 7 is a diagram showing an example of the software configuration of a PC related to the analysis processing of abnormal times. As shown in FIG. 7, the PC 100A includes a data reproduction module 14, an operation setting module 21, an initial information setting module 22, a calibration module 23, an operation reproduction module 24, a virtual work generation module 25, a script execution module 26, a data management module 27, a virtual video generation module 28, an identification module 30, and an output module 32.

[0068] 2 executes a data reproduction program 133. The operation setting module 21, the initial information setting module 22, the calibration module 23, the operation reproduction module 24, the virtual work generation module 25, the script execution module 26, the data management module 27, the virtual moving image generation module 28, the identification module 30, and the output module 32 are implemented by the processor 101 executing a simulation program 134.

[0069] The operation setting module 21, the initial information setting module 22, the calibration module 23, the operation reproduction module 24, the virtual work generation module 25, the script execution module 26, the data management module 27, and the virtual moving image generation module 28 constitute the simulator 20.

[0070] The data reproduction module 14 reproduces data at a time included in the target period using the database 10. The data reproduction module 14 includes a time control module 15 and a data request module 16.

[0071] The time control module 15 manages a counter value that indicates the time within a target period. The counter value is the value of a cycle counter for a control period. Therefore, the counter value can indicate the elapsed time from the start of the target period.

[0072] The time control module 15 updates the counter value. Specifically, the time control module 15 sets the initial counter value to "0" and then adds 1 to the counter value every time a specified time elapses.

[0073] The data request module 16 generates a query according to the specified time represented by the counter value. The specified time is the time that has elapsed from the start time of the target period by the product of a specified time and the counter value. The data request module 16 uses the generated query to obtain the value of each variable at the specified time from the time-series data 11 stored in the database 10. The data request module 16 outputs the obtained value of each variable to the behavior reproduction module 24, the virtual work generation module 25, and the script execution module 26.

[0074] Furthermore, the data request module 16 generates a query requesting a real image frame corresponding to a specified time from among the multiple real image frames indicated by the real video data 12. The data request module 16 uses the generated query to acquire a real image frame at the specified time from the real video data 12 stored in the database 10. The data request module 16 outputs the acquired real image frame to the identification module 30. The shooting time of the real image frame is added to the real image frame.

[0075] In general, the frame interval in the actual moving image data 12 is longer than the control cycle, so the actual image frame corresponding to the specified time is the frame captured at the time closest to the specified time and before the specified time.

[0076] The operation setting module 21 sets correspondence relationship information indicating the correspondence relationship between the associated variables and the operation of each movable part that constitutes the monitoring target 300 A. The operation setting module 21 sets the correspondence relationship information in accordance with a user input.

[0077] FIG. 8 is a diagram showing an example of correspondence information. As shown in FIG. 8, correspondence information 21a indicates the name of a moving part, a variable associated with the movement of the moving part, a vector of translational movement (translation vector), and information on rotational movement (rotation information). The translational vector defines the translational movement of the moving part according to the value of the variable. The rotational information defines the rotational movement of the moving part according to the value of the variable. The rotational information includes information defining the rotation axis or information defining the amount of rotation.

[0078] 7, the initial information setting module 22 sets initial information at the start time of the target period for each component of the monitoring target 300A and each of the objects (including the workpiece 3) surrounding the monitoring target 300A. The objects surrounding the monitoring target 300A may include, in addition to the workpiece 3, for example, a tray or a fixed stand. The initial information setting module 22 sets the initial information in accordance with a user input.

[0079] FIG. 9 is a diagram showing an example of initial information. As shown in FIG. 9, the initial information 22a includes an object name, a mesh shape of the outer surface, coordinates defining a position, and dependency information. The dependency information indicates the dependency destination (including components of the monitored object 300A and objects surrounding the monitored object 300A) and the relative positional relationship between the dependency destination and the dependency destination. If there is no dependency destination, the dependency information is set to Null. A first object being subordinate to a second object means that the first object follows the second object while maintaining a relative positional relationship with the second object.

[0080] The mesh shape of the outer surface may be set from pre-imported CAD (Computer-Aided Design) data. Furthermore, the workpiece 3 may not exist at the start time of the target period. Therefore, coordinates defining the position of the workpiece 3 are not set. As will be described later, the initial position of the workpiece 3 is set by the script execution module 26.

[0081] Returning to FIG. 7, the calibration module 23 sets the position of the virtual camera in the virtual space.

[0082] Fig. 10 is a diagram illustrating a method for setting the position of a virtual camera. Fig. 10 shows a real image frame 12b of a real video 12a captured by a camera 400. As described above, three or more markers 390 appear in the real image frame 12b. Fig. 10 also shows a virtual image frame 28b of a virtual video 28a captured by a virtual camera 400M placed in the virtual space 5.

[0083] The calibration module 23 sets the position and orientation of the virtual camera 400M in the virtual space 5 so that the position of the marker 390 in the real image frame 12b coincides with the position of the virtual marker 390M in the virtual image frame 28b.

[0084] Returning to FIG. 7, the behavior reproduction module 24 simulates the behavior of the virtual model of the monitored object 300A during a period of interest based on the values ​​of one or more variables.

[0085] Specifically, the action reproduction module 24 acquires the values ​​of one or more variables for each time in the target period from the data reproduction module 14 in chronological order. The action reproduction module 24 generates data (1) indicating the state of the virtual model based on the values ​​of one or more variables at each time. The data (1) indicates the position of each part of the virtual model. Hereinafter, the data (1) corresponding to time tk, which is k times the control period after the start time t0 of the target period, will be referred to as "kth data (1)."

[0086] When the action reproduction module 24 acquires the values ​​of one or more variables at time tk, it simulates the action of the part of the virtual model corresponding to each moving part at time tk based on the acquired values ​​and the correspondence information 21a. The action reproduction module 24 integrates the actions of each part to generate k-th data (1) indicating the state of the virtual model at time tk. The action reproduction module 24 outputs the k-th data (1) to the data management module 27.

[0087] The virtual work generation module 25 generates a virtual work 3M (see FIG. 1) corresponding to the work 3 based on the values ​​of one or more variables. The virtual work generation module 25 acquires the values ​​of one or more variables for each time in the target period in chronological order from the data reproduction module 14. The virtual work generation module 25 generates a virtual work 3M in response to the fact that the acquired values ​​of one or more variables include a value of a variable indicating the measurement result of the visual sensor 330.

[0088] Specifically, the virtual workpiece generation module 25 sets the initial position of the virtual workpiece 3M based on the value of a variable indicating the position of the workpiece 3 measured by the visual sensor 330, and generates the virtual workpiece 3M at the initial position.

[0089] It is preferable that the virtual workpiece generation module 25 corrects the initial position by taking into consideration the time lag between when the camera 332 captures an image and when the visual sensor 330 measures the position of the workpiece 3 based on that image. For example, the virtual workpiece generation module 25 uses the technology described in Japanese Patent Laid-Open No. 2017-97426 (Patent Document 2) to correct the initial position using the value of a variable (encoder value) indicating the measurement result of the encoder 312 and an imaging flag output from the visual sensor 330.

[0090] Specifically, when the virtual work generation module 25 receives a photography flag indicating an on state from the data reproduction module 14, it records the time tp corresponding to the photography flag and the encoder value at that time tp. If the value of a variable indicating the measurement result of the visual sensor 330 is included among the values ​​of one or more variables corresponding to time tq, the virtual work generation module 25 extracts the encoder value at time tq from the values ​​of the one or more variables. Time tq is after time tp. The virtual work generation module 25 calculates the movement amount of the transport system 310 between time tp and time tq based on the difference between the encoder value at time tp and the encoder value at time tq. The virtual work generation module 25 corrects the initial position by that movement amount.

[0091] When the virtual workpiece generation module 25 generates the new virtual workpiece 3M, it outputs data (2) indicating the position of the new virtual workpiece 3M to the data management module 27.

[0092] The script execution module 26 simulates the movement of the virtual workpiece 3M during the target period and updates data (2) according to the simulation results. The script execution module 26 acquires the values ​​of one or more variables for each time during the target period from the data reproduction module 14 in chronological order. The script execution module 26 simulates the movement of the part of the virtual model corresponding to the dependent part of the workpiece 3 based on the values ​​of one or more variables at time tk. The simulation method is the same as the simulation method used by the action reproduction module 24. The script execution module 26 simulates the movement of the virtual workpiece 3M at time tk based on the relative positional relationship indicated by the dependency information included in the initial information 22a of the workpiece 3 and the position of the part of the virtual model corresponding to the dependent part. The script execution module 26 generates data (2) indicating the position of the virtual workpiece 3M at time tk based on the simulation results and data (2) corresponding to time tk-1. The action reproduction module 24 outputs data (2) corresponding to time tk to the data management module 27.

[0093] When a plurality of virtual works 3M exist in the virtual space 5, the script execution module 26 updates the data (2) for each of the plurality of virtual works 3M.

[0094] The script execution module 26 may simulate the motion of the virtual workpiece 3M at time tk based on the k-th data (1) generated by the action reproduction module 24 and the relative positional relationship indicated by the subordinate information.

[0095] The data management module 27 manages data (1) indicating the state of the virtual model 300M and data (2) indicating the state of the virtual work 3M. When the data management module 27 acquires the kth data (1), it holds the k-1st data (1) and the kth data (1) and deletes the k-2nd data (1). When the data management module 27 acquires the data (2) corresponding to time tk, it holds the data (2) corresponding to time tk-1 and the data (2) corresponding to time tk, and deletes the data (2) corresponding to time tk-2.

[0096] Virtual moving image generation module 28 generates virtual moving image 28a based on data (1) and data (2). Specifically, virtual moving image generation module 28 places virtual model 300M in virtual space 5 so as to be in the state indicated by data (1), and places virtual workpiece 3M at the position indicated by data (2). Furthermore, virtual moving image generation module 28 places virtual camera 400M in virtual space 5 at the position set by calibration module 23. Virtual moving image generation module 28 generates virtual image frames 28b obtained by shooting with virtual camera 400M at each time, thereby generating virtual moving image 28a.

[0097] The identification module 30 acquires real image frames 12b for each time in the target period from the data reproduction module 14. The real image frames 12b correspond to designated times in the real moving images 12a represented by the real moving image data 12. Upon acquiring the real image frame 12b corresponding to time tk, the identification module 30 acquires, from the virtual moving image generation module 28, a virtual image frame 28b generated based on data (1) and data (2) corresponding to the time tk.

[0098] 11 is a diagram illustrating a method for identifying an abnormal time by the identification module. The identification module 30 identifies a first position of the workpiece 3 in the actual image frame 12b. For example, the identification module 30 uses a known object recognition technique to recognize a pixel group in which each workpiece 3 appears from the actual image frame 12b, and identifies the center of gravity of the pixel group as the first position.

[0099] Furthermore, the identification module 30 identifies a second position of the virtual workpiece 3M in the virtual image frame 28b. The identification module 30 recognizes a group of pixels in which each virtual workpiece 3M appears from the virtual image frame 28b using a known object recognition technique, and identifies the center of gravity of the group of pixels as the second position.

[0100] The identification module 30 identifies the time when the difference between the first position and the second position exceeds a predetermined standard as an abnormal time. That is, the identification module 30 identifies the time tk as an abnormal time in response to the difference between the first position identified from the real image frame 12b corresponding to the time tk and the second position identified from the virtual image frame 28b corresponding to the time tk exceeding the standard.

[0101] 7, the output module 32 generates data showing a list of abnormal times and outputs the data to the display controller 105. As a result, the list of abnormal times is displayed on the display 120.

[0102] <Analysis process flow for abnormal times> Fig. 12 is a flowchart showing the first half of the flow of the anomalous time analysis processing in a PC, and Fig. 13 is a flowchart showing the second half of the flow of the anomalous time analysis processing in a PC.

[0103] First, in step S1, processor 101 of PC 100A performs a simulation initialization process. Specifically, processor 101 executes an initialization function of a script included in simulation program 134.

[0104] Next, in step S2, the processor 101 sets the start time t0 of the target period as the time t. Specifically, the processor 101 sets the counter value to 0. Furthermore, the processor 101 sets the ordinal number i to 0.

[0105] In the next step S3, the processor 101 obtains the values ​​of one or more variables at time t from the time series data 11.

[0106] In the next step S4, the processor 101 simulates the behavior of the virtual model corresponding to the monitored object 300A based on the values ​​of one or more variables. Furthermore, in step S5, if a virtual work 3M has already been placed in the virtual space 5, the processor 101 moves the existing virtual work 3M in accordance with the behavior of the virtual model.

[0107] In the next step S6, the processor 101 determines whether or not the workpiece 3 has been detected based on the value of the variable indicating the measurement result of the visual sensor 330. If the workpiece 3 has been detected (YES in step S6), in step S7 the processor 101 generates a virtual workpiece 3M in the virtual space 5 at an initial position according to the measurement result of the visual sensor 330.

[0108] After step S7, or if step S6 is NO, the analysis process proceeds to step S8. In step S8, the processor 101 generates a virtual image frame 28b corresponding to time t. The virtual image frame 28b constitutes the virtual moving image 28a. In the next step S9, the processor 101 identifies a second position of the virtual workpiece 3M in the virtual image frame 28b. Specifically, the processor 101 identifies the pixel coordinates of the second position.

[0109] In the next step S10, the processor 101 acquires an actual image frame 12b corresponding to time t from the actual video data 12. In the next step S11, the processor 101 identifies a first position of the workpiece 3 in the actual image frame 12b. Specifically, the processor 101 identifies the pixel coordinates of the first position.

[0110] In the next step S12, the processor 101 determines whether the difference between the first position and the second position exceeds a reference value. For example, the processor 101 determines whether the error between the pixel coordinates of the first position and the pixel coordinates of the second position exceeds a threshold value.

[0111] If the difference between the first position and the second position exceeds the reference value (YES in step S12), the processor 101 sets the time t as the abnormal time t_b[i] in step S13. Furthermore, the processor 101 adds 1 to the ordinal number i in step S14.

[0112] After step S14, or if the difference between the first position and the second position does not exceed the reference value (NO in step S12), the analysis process proceeds to step S15. In step S15, the processor 101 determines whether the time t is before the end time T of the target period. If the time t is before the end time T (YES in step S15), in step S16, the processor 101 advances the time t by one control cycle. Specifically, the processor 101 advances the counter value by 1. After step S16, the analysis process returns to step S3. If the time t is not before the end time T (NO in step S15), in step S17, the processor 101 generates data indicating a list of abnormal times t_b and outputs the data to the display controller 105. As a result, the list of abnormal times t_b is displayed on the display 120. After step S17, the analysis process ends.

[0113] <Screen example> Fig. 14 is a diagram showing an example of a window displayed on the display in step S20. The data of the window shown in Fig. 14 is generated by the processor 101. As shown in Fig. 14, a window 50 shows a list of abnormal times t_b.

[0114] <Modification of analysis process> In the above description, the virtual workpiece generation module 25 generates the virtual workpiece 3M in accordance with the value of the variable indicating the measurement result of the visual sensor 330. However, the method of generating the virtual workpiece 3M is not limited to this.

[0115] For example, the virtual work generation module 25 may generate a virtual work 3M based on an image captured by the camera 332. For example, the image captured by the camera 332 is stored in the database 10. If an image corresponding to a specified time is stored in the database 10, the data reproduction module 14 outputs the image to the virtual work generation module 25 along with the values ​​of one or more variables. Upon receiving the image from the data reproduction module 14, the virtual work generation module 25 performs the same processing on the image as the visual sensor 330, thereby identifying the position of the work 3 appearing in the image. The virtual work generation module 25 may set the initial position of the virtual work 3M based on the identified position. Then, the virtual work generation module 25 generates the virtual work 3M at the initial position in the virtual space 5.

[0116] Fig. 15 is a diagram illustrating another method for generating a virtual work. As shown in Fig. 15, workpieces 3 are transported in a line in a transport system 310. An object detection sensor 345 is disposed upstream of an area 410 (see Fig. 4) photographed by a camera 400, and detects whether or not a workpiece 3 is present in a location 314.

[0117] The one or more variables include a specific variable that indicates the detection result of the object detection sensor 345. The virtual work generation module 25 may generate a virtual work 3M at a position corresponding to the location 314 in the virtual space 5 based on the value of the specific variable indicating the presence of the work 3.

[0118] <Visualization of data for the target period> 16 is a diagram showing an example of the software configuration of a PC related to the visualization process of data for a target period. As shown in FIG. 16, PC 100A includes a time designation module 13, a data reproduction module 14, a waveform display module 41, a program display module 42, and a video playback module 43. The waveform display module 41 and the video playback module 43 are realized by the processor 101 executing a viewer program 132. The program display module 42 is realized by the processor 101 executing a development program 131.

[0119] The time designation module 13 sets a playback start counter value and playback conditions, and outputs the set playback start counter value and playback conditions to the data reproduction module 14. The time designation module 13 sets the playback start counter value and playback conditions in response to a user input to an operation window.

[0120] Fig. 17 is a diagram showing an example of an operation window. The operation window 60 shown in Fig. 17 is generated by the time designation module 13 and displayed on the display 120 shown in Fig. 2. The operation window 60 includes a cursor 61, input fields 62 and 66, a play button 63, a frame rewind button 64, and a frame forward button 65.

[0121] The cursor 61 represents a time within the target period. The time designation module 13 accepts a user's operation to move the cursor 61 and moves the position of the cursor 61 in accordance with the movement operation. Furthermore, when the counter value is updated by the data reproduction module 14, the time designation module 13 moves the cursor 61 to a position corresponding to the counter value.

[0122] The input field 62 displays a counter value corresponding to the position of the cursor 61. The counter value represents the ordinal number of the control period in the target period. The user may input a counter value into the input field 62. When the counter value is input into the input field 62, the time designation module 13 moves the cursor 61 to the position corresponding to the input counter value. Furthermore, when the counter value is updated by the data reproduction module 14, the time designation module 13 updates the numerical value displayed in the input field 62.

[0123] The input field 66 is used for inputting the playback speed. The time designation module 13 displays a list of multiple playback speeds in response to an operation of the pull-down button in the input field 66. The time designation module 13 displays the playback speed selected by the user from the list in the input field 66. In the example shown in FIG. 17, the playback speed is expressed by a magnification. Therefore, hereinafter, the "playback speed" will be expressed as a "playback magnification."

[0124] The play button 63, the frame rewind button 64, and the frame advance button 65 are used to input operations related to data reproduction by the data reproduction module 14. In response to clicking any of the play button 63, the frame rewind button 64, and the frame advance button 65, the time designation module 13 sets a playback start counter value and playback conditions, and outputs the set playback start counter value and playback conditions to the data reproduction module 14.

[0125] The play button 63 is used to instruct the data reproduction module 14 to start data reproduction. When the play button 63 is clicked, the time designation module 13 sets a playback start counter value and playback conditions, and outputs the set playback start counter value and playback conditions to the data reproduction module 14. Specifically, the time designation module 13 sets the counter value corresponding to the position of the cursor 61 as the playback start counter value. Furthermore, the time designation module 13 sets the playback conditions in accordance with the playback multiplier entered in the input field 66.

[0126] The frame rewind button 64 is used to move the time back by one control cycle. When the frame rewind button 64 is clicked, the time designation module 13 subtracts 1 from the counter value corresponding to the position of the cursor 61 and sets the result as the playback start counter value. Furthermore, the time designation module 13 sets a playback condition indicating a playback magnification of "0". Note that if the time t is the start time t0 of the target period, operation of the frame rewind button 64 is invalid.

[0127] The frame advance button 65 is used to advance the time by one control cycle. When the frame advance button 65 is clicked, the time designation module 13 adds 1 to the counter value corresponding to the position of the cursor 61 and sets the counter value as the playback start counter value. Furthermore, the time designation module 13 sets a playback condition indicating a playback magnification of "0". Note that if the time t is the end time T of the target period, operation of the frame advance button 65 is invalid.

[0128] Returning to FIG. 16, the data reproduction module 14 reproduces data based on the reproduction start counter value and the reproduction conditions. The time control module 15 of the data reproduction module 14 sets the reproduction start counter value as the initial counter value, and thereafter adds a value obtained by multiplying a specified value (e.g., 50 or 100) by the reproduction magnification to the counter value each time a specified time has elapsed. The specified time is the control period multiplied by the specified value. If the reproduction magnification "×1.0" is set as the reproduction condition, the time control module 15 adds the specified value to the counter value each time the specified time has elapsed. If the reproduction magnification "×2.0" is set as the reproduction condition, the time control module 15 adds a value twice the specified value to the counter value each time the specified time has elapsed. Note that if the reproduction magnification "0" is set as the reproduction condition, the time control module 15 does not update the count value.

[0129] The data request module 16 generates a query corresponding to the specified time represented by the counter value. The data request module 16 uses the query to acquire the values ​​of one or more variables at the specified time from the time-series data 11. Furthermore, the data request module 16 uses the query to acquire actual image frames 12b corresponding to the specified time from the actual video data 12. The data request module 16 outputs the acquired values ​​of the one or more variables to the waveform display module 41 and the program display module 42. The data request module 16 outputs the acquired actual image frames 12b to the video playback module 43.

[0130] The waveform display module 41 generates data for a window including a graph showing the transition of the variable values ​​over the target period based on the values ​​of one or more variables received from the data reproduction module 14, and outputs the data to the display controller 105. As a result, a window including a graph showing the transition of the variable values ​​over the target period is displayed on the display 120.

[0131] The program display module 42 generates data for a window showing the execution status of the control program during the target period based on the values ​​of one or more variables received from the data reproduction module 14, and outputs the data to the display controller 105. As a result, a window showing the execution status of the control program during the target period is displayed on the display 120.

[0132] The video playback module 43 generates data for a window in which the actual video captured during the target period is played, based on the actual image frames 12b received from the data reproduction module 14. The video playback module 43 outputs the data to the display controller 105. As a result, a window in which the actual video captured during the target period is played is displayed on the display 120.

[0133] Fig. 18 is a diagram showing an example of a screen displayed on the display 120. As shown in Fig. 18, the display 120 displays an operation window 60, a window 81 including a graph showing the transition of the value of a variable, a window 82 showing the execution status of the control program, and a window 83 in which a video is played, side by side.

[0134] The user can check the transition of variable values, the execution status of the control program, and video around the abnormal time t_b identified by the analysis process by operating the operation window 60. This makes it easier for the user to search for the cause of the abnormal operation of the monitored object 300.

[0135] <Modification> In the above description, the information processing system 100 is configured by one PC 100A. However, the information processing system 100 may be configured by multiple computers. That is, the information processing system 100 may include multiple processors. Alternatively, the information processing system 100 may include a hardware environment other than a general-purpose PC, such as a virtual machine on a cloud.

[0136] §3 Supplementary Note As described above, the present embodiment includes the following disclosures.

[0137] (Configuration 1) An information processing system (100, 100A) that performs information processing using time series data (11) of values ​​of one or more variables that represent the state of a monitoring target (300), a simulator (20) that simulates the behavior of a virtual model corresponding to the monitoring target in a virtual space (5) during a target period using values ​​of the one or more variables during the target period indicated by the time-series data (11), and generates a virtual video (28a) obtained by capturing images with a virtual camera (400M) placed in the virtual space (5); An information processing system (100, 100A) comprising: an identification module (30) that identifies an abnormal time when a difference between a first position of a moving object (3) in a real video (12a) obtained by photographing the monitored object (300) during the target period and a second position of a virtual object (3M) corresponding to the moving object (3) in the virtual video (28a) exceeds a standard.

[0138] (Configuration 2) The information processing system (100, 100A) according to configuration 1 further comprises an output module (32) that outputs data indicating the list of abnormal times.

[0139] (Configuration 3) The monitored object (300) includes a marker (390) disposed at a fixed position; The information processing system (100, 100A) according to configuration 1 or 2, wherein the simulator (20) sets the position and orientation of the virtual camera (400M) in the virtual space (5) so that the position of the marker (390) in the real video (12a) coincides with the position of a virtual marker (390M) corresponding to the marker (390) in the virtual video (28a).

[0140] (Configuration 4) An information processing method using time series data (11) of values ​​of one or more variables representing the state of a monitoring target (300), A processor (101) simulates behavior of a virtual model corresponding to the monitoring target (300) in a virtual space (5) using values ​​of the one or more variables for a target period indicated by the time-series data (11), and generates a virtual video (28a) obtained by capturing images with a virtual camera (400M) placed in the virtual space (5) (S4 to S8); The information processing method includes the processor (101) identifying (S9 to S13) an abnormal time at which a difference between a first position of a moving object (3) in a real video (12a) obtained by photographing the monitored object (300) during the target period and a second position of a virtual object (3M) corresponding to the moving object (3) in the virtual video (28a) exceeds a standard.

[0141] (Configuration 5) A program for causing a computer to execute an information processing method using time series data (11) of values ​​of one or more variables representing the state of a monitoring target (300), The information processing method includes: Using values ​​of the one or more variables for a target period indicated by the time-series data (11), a behavior of a virtual model corresponding to the monitored object (300) in a virtual space (5) is simulated for the target period, and a virtual video (28a) obtained by capturing images with a virtual camera (400M) arranged in the virtual space (5) is generated (S4 to S8); and identifying (S9 to S13) an abnormal time at which a difference between a first position of a moving object (3) in a real video (12a) obtained by photographing the monitored object (300) during the target period and a second position of a virtual object (3M) corresponding to the moving object (3) in the virtual video (28a) exceeds a standard.

[0142] Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0143] 1 Manufacturing system, 3 Workpiece, 3M virtual workpiece, 5 Virtual space, 10 Database, 11 Time series data, 12 Real video data, 12a Real video, 12b Real image frame, 13 Time specification module, 14 Data reproduction module, 15 Time control module, 16 Data request module, 20 Simulator, 21 Operation setting module, 21a Correspondence information, 22 Initial information setting module, 22a Initial information, 23 Calibration module, 24 Operation reproduction module, 25 Virtual workpiece generation module, 26 Script execution module, 27 Data management module, 28 Virtual video generation module, 28a Virtual video, 28b Virtual image frame, 30 Identification module, 32 Output module, 41 Waveform display module, 42 Program display module, 43 Video playback module, 50, 81, 82, 83 Window, 60 Operation window, 61 Cursor, 62, 66 Input field, 63 Playback button, 64 Frame rewind button, 65 Frame forward button, 100 Information processing system, 100A PC, 101 Processor, 102, 206 Memory, 103, 208 Storage, 104 Input interface, 105 Display controller, 106 Communication interface, 107, 270 Reader / writer, 110 Input device, 120 Display, 131 Development program, 132 Viewer program, 133 Data reproduction program, 134 Simulation program, 200 Control device, 201 Power supply circuit, 202 CPU, 204 Chipset, 207 Buffer area, 211 System program, 213 Scheduler program, 214 Program data set, 250 USB controller, 252 Field network controller, 252a, 254a Connector, 254 Network controller, 260 Timer, 300, 300A Monitoring target, 300M Virtual model, 310 Conveyor system, 312 Encoder, 313,410 area, 314 location, 320 robot, 330 vision sensor, 332,400 camera, 345 object detection sensor, 390 marker, 390M virtual marker, 400M virtual camera, 500 storage medium.

Claims

1. An information processing system that performs information processing using time series data of values ​​of one or more variables that represent a state of a monitoring target, comprising: a simulator that simulates behavior of a virtual model corresponding to the monitoring target in a virtual space using values ​​of the one or more variables in a target period indicated by the time-series data, and generates a virtual video obtained by shooting with a virtual camera arranged in the virtual space; and an identification module that identifies an abnormal time at which a difference between a first position of a moving object in a real video obtained by photographing the monitored object during the target period and a second position of a virtual object corresponding to the moving object in the virtual video exceeds a standard.

2. The information processing system according to claim 1 , further comprising an output module that outputs data indicating the list of abnormal times.

3. The monitored object includes a marker disposed at a fixed position; The information processing system according to claim 1 or 2, wherein the simulator sets a position and orientation of the virtual camera in the virtual space so that a position of the marker in the real video coincides with a position of a virtual marker corresponding to the marker in the virtual video.

4. An information processing method using time series data of values ​​of one or more variables representing a state of a monitored object, comprising: A processor simulates behavior of a virtual model corresponding to the monitoring target in a virtual space using values ​​of the one or more variables in a target period indicated by the time-series data, and generates a virtual video obtained by shooting with a virtual camera arranged in the virtual space; An information processing method comprising: the processor identifying an abnormal time at which a difference between a first position of a moving object in a real video obtained by photographing the monitored object during the target period and a second position of a virtual object corresponding to the moving object in the virtual video exceeds a standard.

5. A program for causing a computer to execute an information processing method using time series data of values ​​of one or more variables representing a state of a monitoring target, The information processing method includes: simulating behavior of a virtual model corresponding to the monitoring target in a virtual space using values ​​of the one or more variables in a target period indicated by the time-series data, and generating a virtual video obtained by shooting with a virtual camera arranged in the virtual space; and identifying an abnormal time at which a difference between a first position of a moving object in a real video obtained by shooting the monitoring target during the target period and a second position of a virtual object corresponding to the moving object in the virtual video exceeds a standard.

Images