Information processing system, information processing method, and program

Abstract

To provide an information processing system capable of appropriately supporting the investigation of a cause of abnormal operation of production facilities.SOLUTION: An information processing system includes a database that stores first time-series data of values of one or more variables that represent a control state of a control device during a target period. The information processing system simulates the motion of a first virtual model of an object that is not controlled by the control device, and stores second time-series data of a position attitude of the first virtual model during the target period in the database. The information processing system generates a virtual image including the first virtual model and a second virtual model of a controlled object. In the virtual image, the first virtual model is visualized to have a position attitude at a specified time indicated by the second time series data, and the second virtual model is visualized to have a state of the controlled object corresponding to the values of one or more variables at the specified time indicated by the first time series data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] A device has been developed to assist in the investigation of the cause of abnormal operation of production equipment. JP 2022-99341 A (Patent Document 1) discloses a technology for generating operation reproduction simulation data that reproduces the operation of a virtual control target based on log data that records the state of an operating part and input / output data of a control signal, and displaying the operation reproduction simulation data on a display unit. The technology disclosed in Patent Document 1 further reproduces the state of the work at a specified time based on the initial state of the work and the log data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-99341 A [Non-patent literature]

[0004] [Non-Patent Document 1] "Pixel2Mesh++", [online], [Retrieved June 5, 2023], Internet〈URL:https: / / github.com / walsvid / Pixel2MeshPlusPlus〉 [Non-Patent Document 2] Rana Hanocka, Gal Metzer, Raja Giryes, Daniel Cohen-Or, "Point2Mesh A Self-Prior for Deformable Meshes", [online], [Retrieved June 5, 2023], Internet〈URL:https: / / ranahanocka.github.io / point2mesh / 〉 Summary of the Invention [Problem to be solved by the invention]

[0005] According to the technology disclosed in Patent Document 1, the state of the workpiece is reproduced on the premise that the workpiece and the moving part are in contact with each other. If an abnormality occurs in the production equipment, the workpiece may separate from the moving part. For example, the workpiece may separate from the robot hand and fall freely. The technology disclosed in Patent Document 1 does not reproduce such a state of the workpiece.

[0006] A physical simulation is known that simulates a physical phenomenon such as the free fall of an object. However, in a physical simulation, only the future motion of the object is simulated. That is, in a physical simulation, time cannot be turned back. Therefore, when investigating the cause of an abnormal operation while performing a physical simulation, a user cannot check the state of the object by turning back the reproduction time.

[0007] The present disclosure has been made in consideration of the above problems, and has as its objective to provide an information processing system, an information processing method, and a program that can appropriately assist in investigating the cause of abnormal operation of production equipment. [Means for solving the problem]

[0008] According to an example of the present disclosure, an information processing system includes one or more databases, a simulator, a data acquisition module, and a first visualization module. The one or more databases store first time series data of values ​​of one or more variables representing a control state of a control device in a target period. The simulator simulates the motion of a first virtual model of an object in a target period based on the first time series data, a dependency relationship between a control target of the control device and an object not controlled by the control device in the target period, and physical properties of the object. The data acquisition module acquires second time series data of a position and orientation of the first virtual model in the target period from a simulation result by the simulator, and stores the second time series data in one or more databases. The first visualization module generates a virtual image in which the first virtual model and a second virtual model of the control target are visualized in an overlapping manner. The first visualization module visualizes the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time series data, and visualizes the second virtual model so as to have a state of the control target corresponding to the values ​​of one or more variables at a specified time indicated by the first time series data.

[0009] According to this disclosure, when the first virtual model is visualized in the virtual image, the motion of the first virtual model is not simulated by the simulator. Therefore, even if the time is turned back, the first visualization module can visualize the first virtual model in the virtual image so as to assume the position and orientation at the time after the time is turned back. This allows the user to check the virtual image including the first virtual model of the object in the target period and the second virtual model of the control target while turning the time back and forward. As a result, the user can easily investigate the cause of the abnormality of the production equipment in the target period. In other words, the information processing system can appropriately support the investigation of the cause of the abnormal operation of the production equipment.

[0010] In the above disclosure, the information processing system includes a shape analysis module that analyzes a change in shape of an object during a target period based on a video obtained by photographing the object. The data acquisition module acquires third time series data of the shape of the object during the target period from the analysis result by the shape analysis module and stores the third time series data in one or more databases. The first visualization module visualizes the first virtual model in the virtual image so that it has a shape at a specified time indicated by the third time series data.

[0011] According to this disclosure, the shape of the first virtual model in the virtual image changes in accordance with the change in the shape of the object, which makes it easier for the user to grasp the change in the shape of the object during the target period.

[0012] In the above disclosure, the simulator simulates the operation of the second virtual model based on the values ​​of one or more variables represented by the first time-series data during a first section of the target period, and simulates the movement of the first virtual model to follow the second virtual model. The simulator simulates the movement of the first virtual model based on physical properties during a second section of the target period other than the first section.

[0013] According to this disclosure, the motion of the first virtual model is appropriately simulated according to the dependent relationship between the object and the control target.

[0014] In the above disclosure, the information processing system further includes one or more second visualization modules that generate data for a first window in which values ​​of one or more variables at a specified time are visualized, and a display controller that displays the first window and a second window including a virtual image side by side on a display.

[0015] For example, the one or more second visualization modules include at least one of a waveform display module that visualizes a graph showing the changes in the values ​​of one or more variables, and a program display module that visualizes the execution status of a control program for controlling a control object in the control device.

[0016] According to this disclosure, a user can simultaneously view a second window containing a virtual image and a first window in which the values ​​of one or more variables are visualized.

[0017] In the above disclosure, the information processing system further includes a video playback module that generates data of a first window including a frame at a specified time from a video obtained by shooting a control target, and a display controller that displays the first window and a second window including a virtual image side by side on a display. According to this disclosure, a user can check the virtual image and the real video at the same time.

[0018] According to another example of the present disclosure, an information processing method uses one or more databases that store first time series data of values ​​of one or more variables representing a control state of a control device in a target period. The information processing method includes one or more processors simulating a motion of a first virtual model of an object in the target period based on the first time series data, a dependency relationship between a control target of the control device and an object not controlled by the control device in the target period, and physical properties of the object. The information processing method includes one or more processors acquiring second time series data of a position and orientation of the first virtual model in the target period from a simulation result, and storing the second time series data in one or more databases. The information processing method includes one or more processors generating a virtual image in which the first virtual model and a second virtual model of the control target are visualized in an overlapping manner. Generating the virtual image includes one or more processors visualizing the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time series data, and one or more processors visualizing the second virtual model in the virtual image so as to have a state of the control target corresponding to the values ​​of one or more variables at a specified time indicated by the first time series data.

[0019] According to yet another example of the present disclosure, a program causes a computer to execute the above information processing method.

[0020] These disclosures also enable the information processing method and program to appropriately assist in investigating the cause of abnormal operation of production equipment. Effect of the Invention

[0021] According to the present disclosure, an information processing system, an information processing method, and a program can appropriately assist in investigating the cause of abnormal operation of production equipment. [Brief description of the drawings]

[0022] [Figure 1] 1 is a diagram illustrating an example of a manufacturing system to which an information processing system according to an embodiment is applied; [Diagram 2] 1 is a schematic diagram showing an example of an overall configuration of a manufacturing system according to an embodiment; [Diagram 3] FIG. 2 is a schematic diagram illustrating an example of a hardware configuration of a control device according to an embodiment. [Figure 4] FIG. 2 is a block diagram showing an example of a software configuration related to advance preparation processing of a PC. [Diagram 5] FIG. 11 is a diagram illustrating an example of correspondence relationship information. [Figure 6] FIG. 11 is a diagram illustrating an example of initial information. [Figure 7] FIG. 13 is a diagram illustrating an example of a method for generating a mesh. [Figure 8] FIG. 2 is a diagram illustrating an example of a data structure of a database. [Figure 9] FIG. 2 is a block diagram showing an example of a software configuration related to information provision processing of a PC. [Figure 10] FIG. 13 illustrates an example of an operation window. [Figure 11] 13 is a flowchart showing an example of the flow of an initial advance preparation process. [Figure 12] 13 is a flowchart showing an example of the flow of the advance preparation process from the second time onward. [Figure 13] 11 is a flowchart showing an example of the flow of an information providing process when the play button shown in FIG. 10 is clicked. [Figure 14]11 is a flowchart showing an example of the flow of an information providing process when a frame rewind button or a frame forward button shown in FIG. 10 is clicked. [Figure 15] 15 is a flowchart showing an example of the process flow of a subroutine of step S27 shown in FIG. 13 or 14. [Figure 16] FIG. 13 is a diagram showing an example of a screen displayed in an operation phase. [Figure 17] FIG. 13 is a diagram showing another example of a screen displayed in the operation phase. [Figure 18] 13 is a diagram showing an example of a conversion table referred to by a first query generating unit; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and the description thereof will not be repeated. In this specification, unless otherwise specified, the term "or" includes the meaning of "and" in addition to the meaning of "or".

[0024] §1 Examples of application 1 is a diagram illustrating an example of a manufacturing system to which an information processing system according to an embodiment is applied. As shown in FIG. 1, the manufacturing system 1 includes an information processing system 100, a control device 200, and a controlled object 300.

[0025] The control device 200 is typically a programmable logic controller (PLC). The control device 200 is connected to a control target 300, and controls the control target 300 according to a control program created in advance.

[0026] The controlled object 300 may include an actuator that exerts some physical action on a production facility or a production line (also referred to as a "field"), various sensors installed in the field, or an input / output device that exchanges information with the field. In the example shown in FIG. 1, the controlled object 300 includes a robot that processes and transports a workpiece 400. The workpiece 400 is an example of an object that is not controlled by the control device 200.

[0027] The control device 200 holds one or more variable values ​​representing a control state, and controls the control target 300 by performing calculations using the one or more variable values. The one or more variables include various variables indicating the state of the control target 300 and various variables to be output to the control target 300.

[0028] The control device 200 updates the values ​​of the variables held in one or more control periods on the order of several hundreds of μsec to several tens of msec. The control period is set in advance according to the type of variable. For example, the control device 200 updates the values ​​of some of the variables every 1 msec, and updates the values ​​of other some of the variables every 2 msec. Hereinafter, the shortest control period of the one or more control periods is referred to as the "shortest control period."

[0029] The control device 200 generates first time series data 11 of values ​​of one or more variables held during a target period including the timing when some abnormality occurred in the controlled object 300. Specifically, the control device 200 generates the first time series data 11 by recording the value of each variable for each shortest control period during the target period. The first time series data 11 is transferred to the information processing system 100 at a predetermined timing.

[0030] The information processing system 100 includes one or more computers. The information processing system 100 includes a database 10, a simulator 20, a data acquisition module 30, and a first visualization module 40.

[0031] The database 10 stores first time-series data 11 of values ​​of one or more variables representing the control state of the control device 200 during a target period.

[0032] The simulator 20 simulates the motion of the first virtual model and the operation of the second virtual model in a virtual space. The first virtual model is a model that represents the three-dimensional shape of the workpiece 400. The second virtual model is a model that represents the three-dimensional shape of the control target 300.

[0033] In this embodiment, the simulator 20 simulates the motion of the first virtual model of the workpiece 400 during the target period based on the first time-series data 11, the subordinate relationship between the control object 300 and the workpiece 400 during the target period, and physical characteristics of the workpiece 400. The subordinate relationship of the workpiece 400 to the control object 300 means that the workpiece 400 follows the motion of the control object 300.

[0034] For example, the simulator 20 simulates the operation of the second virtual model based on the values ​​of one or more variables indicated by the first time-series data 11 in a first section in which the workpiece 400 is subordinate to the control target 300. Furthermore, the simulator 20 simulates the motion of the first virtual model so as to follow the second virtual model.

[0035] The simulator 20 simulates the motion of the first virtual model based on the physical characteristics of the workpiece 400 in the second section where the workpiece 400 is not subordinate to the control object 300.

[0036] The data acquisition module 30 acquires second time-series data 12 of the position and orientation of the first virtual model during the target period from the simulation results of the simulator 20, and stores the second time-series data 12 in the database 10.

[0037] The first visualization module 40 generates a virtual image in which a first virtual model of the workpiece 400 and a second virtual model of the control object 300 are visualized in an overlapping manner. The first visualization module 40 visualizes the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time-series data 12. Furthermore, the first visualization module 40 visualizes the second virtual model in the virtual image so as to have a state of the control object 300 corresponding to values ​​of one or more variables at a specified time indicated by the first time-series data 11.

[0038] According to this embodiment, the motion of the first virtual model of the work 400 during the target period is simulated based on the first time-series data 11, the subordinate relationship between the control object 300 and the work 400 during the target period, and the physical characteristics of the work 400. Therefore, even if the work 400 is not subordinate to the control object 300, the motion of the first virtual model of the work 400 during the target period is simulated based on the physical characteristics of the work 400.

[0039] Once the simulation is performed by the simulator 20, second time-series data 12 of the position and orientation of the first virtual model in the target period is acquired from the simulation result, and the second time-series data 12 is stored in the database 10. The first visualization module 40 visualizes the first virtual model in a virtual image based on the second time-series data 12. That is, when the first virtual model is visualized in the virtual image, the motion of the first virtual model is not simulated by the simulator 20. Therefore, even if the time is turned back, the first visualization module 40 can visualize the first virtual model in the virtual image so that it has the position and orientation at the time after the time is turned back. This allows the user to check the virtual image including the first virtual model of the workpiece 400 and the second virtual model of the control target 300 in the target period while turning the time back and forward. As a result, it becomes easier for the user to investigate the cause of the abnormality of the production equipment in the target period.

[0040] In this way, according to this embodiment, the information processing system 100 can appropriately assist in investigating the cause of abnormal operation of production equipment.

[0041] §2 Specific examples <Overall configuration of manufacturing system> Fig. 2 is a schematic diagram showing an example of the overall configuration of a manufacturing system according to an embodiment. As shown in Fig. 2, the manufacturing system 1 includes a personal computer (hereinafter referred to as "PC100A") constituting an information processing system 100, a control device 200, a control target 300, one or more network cameras 500, and a storage medium 700. The PC100A, the control device 200, and the one or more network cameras 500 can communicate with each other using, for example, a general-purpose Ethernet (registered trademark). The PC100A and the control device 200 may communicate with each other using another communication means (for example, Universal Serial Bus (USB)).

[0042] One or more network cameras 500 are installed in the field and capture at least a part of the control target 300. Video data 14 generated by the one or more network cameras 500 is transferred to the PC 100A. Alternatively, the video data 14 may be stored in a network storage (not shown). The video data 14 indicates frames (frame images) for each capture time.

[0043] The storage medium 700 includes a volatile storage medium, a non-volatile storage medium, a general-purpose semiconductor storage device such as a Compact Flash (CF) or a Secure Digital (SD), a magnetic storage medium such as a Flexible Disk, or an optical storage medium such as a Compact Disk Read Only Memory (CD-ROM). The storage medium 700 is, for example, an SD memory card.

[0044] The PC 100A is typically a computer having a general-purpose architecture. As shown in Fig. 2, the PC 100A includes a processor 101 such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), a memory 102 configured as a volatile storage device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory), a storage 103, an input interface 104, a display controller 105, a communication interface 106, and a reader / writer 107. These components are connected to each other via a bus so as to be able to communicate data with each other.

[0045] The processor 101 reads out a program stored in the storage 103, and loads the program in the memory 102. The processor 101 executes the loaded program.

[0046] The storage 103 is configured by a non-volatile storage device such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage 103 stores the program executed by the processor 101 and the database 10.

[0047] The programs executed by the processor 101 include a development program 131 , a viewer program 132 , a data reproduction program 133 , a simulation program 134 , a shape analysis program 135 , and a database construction program 136 .

[0048] The development program 131 mainly includes a set of commands for supporting the development of a control program used by the control device 200. The development program 131 has a function of setting the names of one or more variables used by the control program (hereinafter referred to as "variable names") in response to a user input. Furthermore, the development program 131 provides a setting function related to the generation of the first time-series data 11, and a function of monitoring the execution status of the control program by the control device 200 (hereinafter referred to as "program monitor function").

[0049] A control program developed using the development program 131 is installed in the control device 200. Furthermore, monitoring setting data generated using the setting function of the development program 131 is transferred to the control device 200.

[0050] The viewer program 132 includes a set of commands for visualizing various types of information and providing it to the user. The viewer program 132 provides a function for displaying a graph showing the transition of the values ​​of one or more variables (hereinafter referred to as a "variable graph display function") and a function for playing back a video represented by the video data 14.

[0051] The data reproduction program 133 includes a set of commands for reproducing past data. The data reproduction program 133 includes a function for managing time and a function for requesting data to be reproduced (hereinafter referred to as a "data request function"). The data request function includes a function for generating a query and a function for acquiring target data from the database 10 using the query.

[0052] The simulation program 134 includes a set of instructions for simulating the motion of the first virtual model of the workpiece 400 in the virtual space and the motion of the second virtual model of the control target 300. The simulation program 134 includes a function for reproducing the motion of the second virtual model (hereinafter referred to as the "motion reproduction function"), a script for moving an object such as the first virtual model, a physical simulation function for simulating a physical phenomenon of an object, a data management function, and a model display function.

[0053] The script is written in, for example, C# format and includes a start method and a render method. The start method is executed only once when the script starts to be executed. The render method is executed periodically.

[0054] The shape analysis program 135 includes a group of instructions for analyzing changes in the shape of an object shown in a video based on the video.

[0055] The database construction program 136 includes a set of instructions for storing various types of data in the database 10 .

[0056] The database 10 stores first time series data 11, second time series data 12, third time series data 13, and video data 14. The first time series data 11 is transferred from the control device 200, and indicates the values ​​at each time of one or more variables that indicate the control state of the control device 200 during the target period. The second time series data 12 is generated in response to execution of a script or a physical simulation function of the simulation program 134, and indicates the position and orientation at each time of the first virtual model during the target period. The third time series data 13 is generated in response to execution of the shape analysis program 135, and indicates the shape of the workpiece 400 at each time during the target period. The video data 14 is generated by shooting with the network camera 500.

[0057] The input interface 104 mediates data transmission between the processor 101 and an input device 110 such as a keyboard, a mouse, or a touch panel. The display controller 105 is connected to a display 120 and displays on the display 120 a screen indicated by screen data generated by the processor 101. The communication interface 106 communicates with the control device 200 and the network camera 500 via a communication network. The reader / writer 107 mediates data transmission between the processor 101 and an external storage medium (which may include, for example, a storage medium 700).

[0058] Each of the development program 131, the viewer program 132, the data reproduction program 133, the simulation program 134, the shape analysis program 135, and the database construction program 136 may be provided not as a standalone program but as part of an arbitrary program. In this case, the processing according to the present embodiment is realized in cooperation with the arbitrary program. In addition, some or all of the functions provided by the development program 131, the viewer program 132, the data reproduction program 133, the simulation program 134, the shape analysis program 135, and the database construction program 136 may be realized by a dedicated hardware circuit (for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0059] <Control device hardware configuration> Fig. 3 is a schematic diagram showing an example of a hardware configuration of a control device according to an embodiment. As shown in Fig. 3, the control device 200 includes, as main components, a power supply circuit 201 that supplies power PW to each unit of the control device 200, a CPU 202, a chipset 204, a memory 206, a storage 208, a USB controller 250, a field network controller 252, a network controller 254, a timer 260, and a reader / writer 270.

[0060] The CPU 202 reads out a program stored in the storage 208 and loads it in the memory 206. The CPU 202 executes the loaded program.

[0061] The memory 206 is configured with a volatile storage device such as a DRAM or an SRAM. The memory 206 holds the values ​​of various variables. The values ​​of the various variables held in the memory 206 are updated according to a control period preset according to the type of the variable, by the CPU 202 executing a program stored in the storage 208.

[0062] The actual values ​​of various variables are accumulated in a buffer area 207 of a memory 206 for a certain period of time.

[0063] The storage 208 is configured, for example, by a non-volatile storage device such as an HDD, an SSD, or a flash memory, but a portion of the storage device may be configured by a volatile storage device.

[0064] The chipset 204 mediates data exchange between the CPU 202 and each component, thereby achieving processing of the control device 200 as a whole.

[0065] The storage 208 stores an OS (Operating System) 212 for implementing the basic functions of the control device 200 and a system program 211 having a scheduler program 213 .

[0066] Furthermore, the storage 208 stores a program data set 214 created by the PC 100A. The program data set 214 includes a control program and sampling setting data.

[0067] The control program defines a process for collecting the values ​​of variables indicating the state of the control target 300, and a process for performing calculations based on the collected variable values ​​to generate variable values ​​related to commands to the control target 300. The values ​​of various variables are temporarily stored in memory 206. CPU 202 executes the control program to update the variable values ​​stored in memory 206 for each control cycle.

[0068] The sampling setting data defines the generation conditions of the first time series data 11. The generation conditions of the first time series data 11 indicate the trigger conditions for sampling, the variables to be sampled, the sampling time, and the like.

[0069] The trigger condition includes, for example, a condition that a value of a specific variable related to an abnormality of the control target 300 has changed to a specific value. The CPU 202 reads out the first time series data 11 indicating values ​​of one or more variables in a target period including the timing at which the trigger condition is satisfied from the buffer area 207, and controls the reader / writer 270 to write the first time series data 11 to the storage medium 700. The target period includes, for example, a first predetermined period before the timing at which the trigger condition is satisfied and a second predetermined period after the timing at which the trigger condition is satisfied. The lengths of the first and second predetermined periods are defined by the sampling setting data.

[0070] The USB controller 250 is responsible for sending and receiving data to and from an external device (eg, PC 100A) via a USB connection.

[0071] The field network controller 252 has a connector 252a that connects to the field network, and controls data exchange with the controlled object 300. The field network controller 252 has an internal buffer that stores input data received from the field network and output data output from the control device 200 to the field network.

[0072] The network controller 254 has a connector 254a for connecting to a higher-level network, and controls data exchange between the control device 200 and an external device (for example, the PC 100A, the network camera 500, or a network storage).

[0073] The timer 260 includes a clock circuit or a counter circuit that measures time, but is not limited to such circuits and may be configured by a software module executed by the CPU 202.

[0074] The reader / writer 270 mediates data transmission between the CPU 202 and the storage medium 700. The CPU 202 copies the first time series data 11 generated according to the generation conditions indicated by the sampling setting data from the buffer area 207 to the storage medium 700. The first time series data 11 stored in the storage medium 700 is transferred to the PC 100A via the network controller 254 at a predetermined timing. Alternatively, the user may connect the storage medium 700 to the PC 100A and copy or move the first time series data 11 from the storage medium 700 to the PC 100A.

[0075] 3 shows an example of a configuration in which the CPU 202 executes a program to provide necessary functions, but some or all of these provided functions may be implemented using a dedicated hardware circuit (e.g., ASIC or FPGA). Alternatively, the main part of the control device 200 may be realized using hardware that conforms to a general-purpose architecture (e.g., an industrial PC based on a general-purpose PC). In this case, multi-core technology may be applied to execute processes in parallel. Alternatively, virtualization technology may be used to execute multiple OSs with different uses in parallel, and necessary applications may be executed on each OS.

[0076] <Software configuration> The software configuration of the PC 100A will be described with reference to Figs. 4 to 10. The PC 100A acquires from the control device 200 first time-series data 11 of values ​​of one or more variables in a target period including the timing when some abnormality occurs in the control target 300. The PC 100A supports the user in investigating the cause of an abnormal operation of the production equipment by providing information indicating the state of the control target 300 and one or more objects (including the workpiece 400) not controlled by the control device 200 in the target period using the first time-series data 11. The processing of the PC 100A includes a preparation process executed in the preparation phase and an information provision process executed in the operation phase. The preparation phase is a phase before the operation phase. However, the user may move the PC 100A to the preparation phase again after checking the information provided in the operation phase.

[0077] (Software configuration for advance preparation processing) Fig. 4 is a block diagram showing an example of a software configuration related to the advance preparation process of a PC. As shown in Fig. 4, the PC 100A includes a data reproduction module 16, an operation setting module 21, an initial information setting module 22, an operation reproduction module 23, a script execution module 24, a physical simulation execution module 25, a data management module 26, a shape analysis module 50, a coordinate acquisition module 31, and a shape acquisition module 32 as software configuration related to the advance preparation process.

[0078] The data reproduction module 16 is realized by the processor 101 shown in Fig. 2 executing a data reproduction program 133. The action setting module 21, the initial information setting module 22, the action reproduction module 23, the script execution module 24, the physical simulation execution module 25, and the data management module 26 are realized by the processor 101 executing a simulation program 134. The shape analysis module 50 is realized by the processor 101 executing a shape analysis program 135. The coordinate acquisition module 31 and the shape acquisition module 32 are realized by the processor 101 executing a database construction program 136.

[0079] The operation setting module 21, the initial information setting module 22, the operation reproduction module 23, the script execution module 24, the physical simulation execution module 25, and the data management module 26 constitute the simulator 20 shown in Fig. 1. The coordinate acquisition module 31 and the shape acquisition module 32 constitute the data acquisition module 30 shown in Fig. 1.

[0080] The data reproduction module 16 reproduces data at a specified time included in a target period by using the database 10. The data reproduction module 16 includes a time control module 17 and a data request module .

[0081] The time control module 17 manages a counter value that indicates the time within the target period. The counter value is the value of the period counter of the shortest control period. Therefore, the counter value can indicate the elapsed time from the start of the target period.

[0082] The time control module 17 updates the counter value. Specifically, the time control module 17 sets an initial counter value as the counter value, and then adds 1 to the counter value every time a specified time elapses. In the advance preparation phase, the time control module 17 uses "0" as the initial counter value. The specified time is set in advance taking into account the processing time by the simulator 20. The time control module 17 increments the counter value by 1 every time the specified time elapses.

[0083] The data request module 18 generates a query according to the specified time represented by the counter value. The specified time is the time that has elapsed from the start time of the target period by the product of a specified time and the counter value. The data request module 18 uses the generated query to obtain the target data from the database 10. As a result, the data at the specified time included in the target period is reproduced.

[0084] Data request module 18 includes a first query generating unit 18a and a second query generating unit 18b.

[0085] First query generation unit 18a generates a query requesting the value of each variable at a specified time. First query generation unit 18a uses the generated query to acquire the value of each variable at the specified time from first time-series data 11 stored in database 10. In the advance preparation phase, first query generation unit 18a outputs the acquired value of each variable to action reproduction module 23 and script execution module 24. The designated time is added to the value of each variable.

[0086] Second query generation unit 18b generates a query requesting a frame corresponding to a specified time among multiple frames represented by video data 14. Second query generation unit 18b uses the generated query to acquire a frame at the specified time from video data 14 stored in database 10. In the advance preparation phase, second query generation unit 18b outputs the acquired frame to shape analysis module 50. The shooting time is added to the frame.

[0087] In addition, the frame interval in video data 14 is generally longer than the shortest control period. Therefore, if there is no frame corresponding to the specified time, second query generation unit 18b does not acquire a frame from database 10. A frame corresponding to the specified time is a frame whose shooting time is the time difference between the specified time and the shortest control period or less.

[0088] The operation setting module 21 sets correspondence relationship information indicating the correspondence relationship between related variables and the operation of each movable part constituting the control target 300. The operation setting module 21 sets the correspondence relationship information in response to a user input.

[0089] Fig. 5 is a diagram showing an example of the correspondence information. As shown in Fig. 5, the correspondence information 21a indicates the name of a moving part, a variable associated with the movement of the moving part, a vector of a translation movement (translation vector), and information of a rotation movement (rotation information). The translation vector defines the translation movement of the moving part according to the value of the variable. The rotation information defines the rotation movement of the moving part according to the value of the variable. The rotation information includes information defining the rotation axis or information defining the amount of rotation.

[0090] 4, the initial information setting module 22 sets initial information at the start time of the target period for each component of the control target 300 and one or more objects not controlled by the control device 200. The one or more objects not controlled by the control device 200 may include, for example, a tray or a fixed table in addition to the workpiece 400. The initial information setting module 22 sets initial information in response to a user input.

[0091] FIG. 6 is a diagram showing an example of the initial information. As shown in FIG. 6, the initial information 22a includes three layers of information 22b, 22c, and 22d. The information 22b indicates an object name for identifying an object. The information 22c indicates a mesh shape of an outer surface and coordinates for defining a position and orientation. Furthermore, the information 22c includes subordinate information. The subordinate information indicates a subordinate (including a part or object of the control target 300) and a relative positional relationship with the subordinate. If there is no subordinate, the subordinate information is set to Null. The information 22d indicates a physical simulation flag indicating whether or not it can be a target of physical simulation, and physical properties. The physical properties include, for example, mass, material, density, friction coefficient, Young's modulus, Poisson's ratio, elastic damping, or damping scale.

[0092] The mesh shape of the outer surface may be set from CAD (Computer-Aided Design) data imported in advance. The workpiece 400 may not exist at the start time of the target period. Therefore, coordinates that define the position and orientation of the workpiece 400 are not set. As described later, the initial position and orientation of the workpiece 400 is set by the script execution module 24.

[0093] In the following, a case will be described where the one or more objects not controlled by the control device 200 include only the workpiece 400. When the one or more objects not controlled by the control device 200 include an object other than the workpiece 400, the same processing as for the workpiece 400 is executed for the object.

[0094] Returning to FIG. 4, the operation reproduction module 23 simulates the operation of the second virtual model of the control target 300 in a target period based on the first time-series data 11.

[0095] In the advance preparation phase, the action reproducing module 23 acquires the values ​​of one or more variables for each time in the target period from the data reproducing module 16 in chronological order. The action reproducing module 23 generates data (1) indicating the state of the second virtual model based on the values ​​of one or more variables at each time. The data (1) indicates the position and orientation of each part of the second virtual model. Hereinafter, the data (1) corresponding to the time tk that is k times the shortest control period after the start time t0 of the target period is referred to as the "kth data (1)."

[0096] When the action reproducing module 23 acquires the values ​​of one or more variables at time tk, it simulates the action of the part of the second virtual model corresponding to each movable part at time tk based on the acquired values ​​and the correspondence information 21a. The action reproducing module 23 generates k-th data (1) indicating the state of the second virtual model at time tk by integrating the actions of each part. The action reproducing module 23 outputs the k-th data (1) to the data management module 26. The data management module 26 holds the k-th data (1).

[0097] The script execution module 24 generates a first virtual model corresponding to the workpiece 400 in a virtual space based on the values ​​of one or more variables at each time in a target period. The generation conditions of the first virtual model are defined by a script. For example, the script execution module 24 generates a first virtual model in a virtual space based on the values ​​of variables indicating the detection result of an object detection sensor that detects the presence of the workpiece 400. At this time, the script execution module 24 sets a position and orientation corresponding to the detection target area of ​​the object detection sensor in the virtual space as the initial position and orientation of the first virtual model. In addition, when the script execution module 24 generates the first virtual model based on the values ​​of one or more variables at time tk, the script execution module 24 sets a physical simulation flag for the first virtual model based on the initial information 22a.

[0098] The script execution module 24 and the physical simulation execution module 25 simulate the movement of the first virtual model of the workpiece 400 during the target period. When the script execution module 24 and the physical simulation execution module 25 simulate the movement of the first virtual model at each time during the target period, they generate data (2) indicating the position and orientation of the first virtual model. Hereinafter, the data (2) corresponding to the time tk during the target period is referred to as the "kth data (2)". That is, the script execution module 24 or the physical simulation execution module 25 generates the kth data (2) indicating the position and orientation of the first virtual model at the time tk. The script execution module 24 or the physical simulation execution module 25 outputs the kth data (2) to the data management module 26. The data management module 26 holds the kth data (2). The kth data (2) is added with the time tk.

[0099] In the advance preparation phase, the script execution module 24 acquires the values ​​of one or more variables for each time in the target period from the data reproduction module 16 in chronological order. The script execution module 24 updates a physics simulation flag for the first virtual model based on the values ​​of one or more variables at each time. The physics simulation flag is used to determine an entity that generates data (2). For example, when the physics simulation flag corresponding to time tk-1 is "0", the script execution module 24 simulates the movement of the first virtual model at time tk to generate the k-th data (2). When the physics simulation flag corresponding to time tk-1 is "1", the physics simulation execution module 25 simulates the movement of the first virtual model at time tk to generate the k-th data (2).

[0100] The method of updating the physical simulation flag is defined by a script. Typically, the script execution module 24 sets the physical simulation flag according to the subordinate relationship between the control object 300 and the work 400. As a result, the script execution module 24 simulates the motion of the first virtual model in a first section during the target period in which the work 400 is subordinate to the control object 300. The physical simulation execution module 25 simulates the motion of the first virtual model in a second section during the target period in which the work 400 is not subordinate to the control object 300.

[0101] For example, the script execution module 24 updates the physics simulation flag at time tk to "1" in response to the fact that the physics simulation flag at time tk-1 is "0" and the values ​​of one or more variables at time tk satisfy a predetermined blocking condition. The blocking condition is, for example, a condition that the distance between two fingers of the hand of the control target 300 calculated from the values ​​of the variables exceeds a specified value.

[0102] The script execution module 24 maintains the physics simulation flag at time tk at "0" in response to the fact that the physics simulation flag at time tk-1 is "0" and the blocking condition is not satisfied.

[0103] The script execution module 24 updates the physics simulation flag at time tk to "0" in response to the fact that the physics simulation flag at time tk-1 is "1" and the values ​​of one or more variables at time tk satisfy a predetermined dependent condition. The dependent condition is, for example, a condition that the difference between the midpoint position between two fingers of the hand of the control object 300 calculated from the values ​​of the variables and the position of the first virtual model indicated by the k-1th data (2) is within a specified range and the distance between the two fingers of the hand calculated from the values ​​of the variables is equal to or smaller than a specified value.

[0104] The script execution module 24 maintains the physics simulation flag at time tk at "1" in response to the fact that the physics simulation flag at time tk-1 is "1" and the dependent condition is not satisfied.

[0105] When the physical simulation flag at time tk-1 is "0", the script execution module 24 generates k-th data (2) indicating the state of the first virtual model at time tk. Specifically, the script execution module 24 simulates the operation of a portion of the second virtual model corresponding to the subordinate part of the workpiece 400 based on the values ​​of one or more variables at time tk. The method of simulation is the same as the method of simulation in the operation reproduction module 23. The script execution module 24 simulates the position and posture of the first virtual model at time tk based on the relative positional relationship indicated by the subordinate information included in the initial information 22a of the workpiece 400 and the position and posture of the second virtual model corresponding to the subordinate part.

[0106] The script execution module 24 may simulate the position and orientation of the first virtual model at time tk based on the k-th data (1) generated by the action reproduction module 23 and the relative positional relationship indicated by the dependent information.

[0107] When the physical simulation flag at time tk-1 is "1", the physical simulation execution module 25 generates k-th data (2) indicating the state of the first virtual model at time tk. Specifically, the physical simulation execution module 25 generates the k-th data (2) based on the k-1th data (2) held by the data management module 26 and the physical characteristics of the workpiece 400. The physical simulation execution module 25 acquires the physical characteristics of the workpiece 400 from the data management module 26. The data management module 26 acquires the physical characteristics of the workpiece 400 from the initial information setting module 22.

[0108] For example, when the state of the first virtual model indicated by the k-1th data (2) is not in contact with the second virtual model and the virtual model corresponding to the surrounding object, the physical simulation execution module 25 assumes that the first virtual model is falling freely due to gravity, and simulates the movement of the first virtual model.

[0109] The data management module 26 manages data (1) indicating the state of the second virtual model and data (2) indicating the state of the first virtual model. When the data management module 26 acquires the kth data (1), it holds the k-1st data (1) and the kth data (1), and deletes the k-2nd data (1). When the data management module 26 acquires the kth data (2), it holds the k-1st data (2) and the kth data (2), and deletes the k-2nd data (2).

[0110] Furthermore, the data management module 26 manages data (3) representing the shape of the workpiece 400. The data (3) is generated by the shape analysis module 50.

[0111] The shape analysis module 50 includes a video analysis module 51 and a mesh generation module 52 .

[0112] In the advance preparation phase, the video analysis module 51 acquires frames of the video data 14 during a target period in chronological order from the data reproduction module 16. The video analysis module 51 uses a known object recognition technique to extract an area in which the workpiece 400 appears from the acquired frames.

[0113] The mesh generation module 52 generates a mesh that represents the shape of the workpiece 400 based on the area in the frame in which the workpiece 400 appears.

[0114] FIG. 7 is a diagram for explaining an example of a method for generating a mesh. In the example shown in FIG. 7, the video analysis module 51 extracts an area in which the workpiece 400 appears for each of a plurality of frames acquired from a plurality of network cameras 500. The mesh generation module 52 generates a mesh from the areas extracted from the plurality of frames. For example, the mesh generation module 52 generates a mesh using Pixel2Mesh++, which is a deep learning library. Details of Pixel2Mesh++ are disclosed, for example, in "Pixel2Mesh++", [online], [searched June 5, 2023], Internet <URL: https: / / github.com / walsvid / Pixel2MeshPlusPlus>" (Non-Patent Document 1).

[0115] Alternatively, if one or more of the network cameras 500 include a 3D camera, the mesh generation module 52 may generate point cloud data based on an image obtained from the 3D camera, and generate a mesh from the point cloud data using Point2Mesh. Details of Point2Mesh are disclosed in, for example, "Rana Hanocka, Gal Metzer, Raja Giryes, Daniel Cohen-Or, "Point2Mesh A Self-Prior for Deformable Meshes", [online], [searched June 5, 2023], Internet <URL: https: / / ranahanocka.github.io / point2mesh / >" (Non-Patent Document 2).

[0116] The mesh generation module 52 outputs data (3) indicating the generated mesh to the data management module 26. The same time as the shooting time of the frame is added to the data (3). The data management module 26 holds the data (3) with the time added.

[0117] The shape analysis module 50 may perform the process of generating the data (3) only when an activation instruction is received. For example, the user inputs an activation instruction for the shape analysis module 50 when the workpiece 400 is a flexible object. The user inputs an inactivation instruction for the shape analysis module 50 when the workpiece 400 is not a flexible object.

[0118] The coordinate acquisition module 31 generates second time-series data 12 of the position and orientation of the first virtual model during the target period based on the data (2) managed by the data management module 26, and stores the second time-series data 12 in the database 10.

[0119] The shape acquisition module 32 generates third time series data 13 of the shape of the workpiece 400 in the target period based on the data (3) managed by the data management module 26, and stores the third time series data 13 in the database 10.

[0120] Fig. 8 is a diagram showing an example of the data structure of the database. As shown in Fig. 8, the database 10 includes, for each time, a data fragment 11a indicating a combination of a variable ID and a variable value, a data fragment 12a indicating coordinates defining the position and orientation of the first virtual model, and a data fragment 13a indicating the shape (mesh) of the workpiece 400. The data fragments 11a, 12a, and 13a constitute the first time series data 11, the second time series data 12, and the third time series data 13, respectively.

[0121] (Software configuration for information provision processing) Fig. 9 is a block diagram showing an example of a software configuration related to information provision processing of a PC. As shown in Fig. 9, PC 100A includes a time designation module 15, a data reproduction module 16, an operation setting module 21, an initial information setting module 22, an operation reproduction module 23, a model generation module 41, a model display module 42, a waveform display module 43, a program display module 44, and a video playback module 45 as software configuration related to information provision processing.

[0122] The model generation module 41 and the model display module 42 are realized by the processor 101 executing the simulation program 134. The waveform display module 43 and the video playback module 45 are realized by the processor 101 executing the viewer program 132. The program display module 44 is realized by the processor 101 executing the development program 131.

[0123] The model generation module 41 and the model display module 42 constitute the first visualization module 40 shown in Fig. 1. The waveform display module 43 and the program display module 44 are an example of a "second visualization module" in the present disclosure.

[0124] The time designation module 15 sets a playback start counter value and playback conditions in response to a user input to the operation window, and outputs the set playback start counter value and playback conditions to the data reproduction module 16 .

[0125] Fig. 10 is a diagram showing an example of an operation window. The operation window 60 shown in Fig. 10 is generated by the time designation module 15 and is displayed on the display 120 shown in Fig. 2. The operation window 60 includes a cursor 61, input fields 62 and 66, a play button 63, a frame rewind button 64, and a frame forward button 65.

[0126] The cursor 61 indicates a time within the target period. The time designation module 15 accepts a user's operation to move the cursor 61, and moves the position of the cursor 61 in response to the movement operation. Furthermore, when the counter value is updated by the data reproduction module 16, the time designation module 15 moves the cursor 61 to a position corresponding to the counter value.

[0127] A counter value corresponding to the position of the cursor 61 is displayed in the input field 62. The counter value represents the ordinal number of the minimum control period in the target period. The user may input a counter value in the input field 62. When the counter value is input in the input field 62, the time designation module 15 moves the cursor 61 to a position corresponding to the input counter value. Furthermore, when the counter value is updated by the data reproduction module 16, the time designation module 15 updates the numerical value displayed in the input field 62.

[0128] The input field 66 is used for inputting the playback speed. The time designation module 15 displays a list of multiple playback speeds in response to the operation of the pull-down button in the input field 66. The time designation module 15 displays the playback speed selected by the user from the list in the input field 66. In the example shown in FIG. 10, the playback speed is represented by a magnification. Therefore, hereinafter, the "playback speed" will be expressed as a "playback magnification."

[0129] The play button 63, the frame rewind button 64, and the frame forward button 65 are used to input operations related to data reproduction by the data reproduction module 16. In response to clicking any of the play button 63, the frame rewind button 64, and the frame forward button 65, the time designation module 15 sets a playback start counter value and playback conditions, and outputs the set playback start counter value and playback conditions to the data reproduction module 16.

[0130] The play button 63 is used to instruct the data reproduction module 16 to start reproducing data. When the play button 63 is clicked, the time designation module 15 sets a reproduction start counter value and reproduction conditions, and outputs the set reproduction start counter value and reproduction conditions to the data reproduction module 16. Specifically, the time designation module 15 sets the counter value corresponding to the position of the cursor 61 as the reproduction start counter value. Furthermore, the time designation module 15 sets the reproduction conditions according to the reproduction magnification input in the input field 66. In this way, the data reproduction module 16 reproduces data for each reproduction cycle, as described below. The reproduction cycle is a time obtained by multiplying a specified value (for example, 50 or 100) by the reproduction magnification.

[0131] The frame rewind button 64 is used to move the time back by one control cycle. When the frame rewind button 64 is clicked, the time designation module 15 subtracts 1 from the counter value corresponding to the position of the cursor 61 and sets the result as the playback start counter value. Furthermore, the time designation module 15 sets a playback condition indicating a playback magnification of "0".

[0132] The frame advance button 65 is used to advance the time by one control cycle. When the frame advance button 65 is clicked, the time designation module 15 adds 1 to the counter value corresponding to the position of the cursor 61 and sets the counter value as the playback start counter value. Furthermore, the time designation module 15 sets a playback condition indicating a playback magnification of "0".

[0133] Returning to FIG. 9, the data reproduction module 16 performs processing according to the operation phase. That is, the time control module 17 of the data reproduction module 16 sets the reproduction start counter value as the initial counter value, and thereafter, adds a value obtained by multiplying a specified value (e.g., 50 or 100) by the reproduction magnification to the counter value every time a specified time has elapsed. The specified time is the shortest control period multiplied by the specified value. When the reproduction magnification "x1.0" is set as the reproduction condition, the time control module 17 adds the specified value to the counter value every time the specified time has elapsed. When the reproduction magnification "x2.0" is set as the reproduction condition, the time control module 17 adds a value twice the specified value to the counter value every time the specified time has elapsed. Note that when the reproduction magnification "0" is set as the reproduction condition, the time control module 17 does not update the count value.

[0134] Data request module 18 generates a query according to the designated time represented by the counter value. In the operation phase, data request module 18 includes third query generation unit 18c in addition to first query generation unit 18a and second query generation unit 18b.

[0135] First query generation unit 18a and second query generation unit 18b perform the same processes in the operation phase as in the advance preparation phase. However, in the operation phase, first query generation unit 18a outputs the acquired values ​​of each variable to action reproduction module 23, waveform display module 43, and program display module 44. In the operation phase, second query generation unit 18b outputs the acquired frames to video playback module 45.

[0136] The third query generation unit 18c generates a query requesting coordinates at a specified time (coordinates that define the position and orientation of the first virtual model). The third query generation unit 18c uses the generated query to acquire the coordinates at the specified time from the second time-series data 12 stored in the database 10. The third query generation unit 18c outputs the acquired coordinates to the model generation module 41.

[0137] Furthermore, when the third time series data 13 is stored in the database 10, the third query generation unit 18c generates a query requesting a mesh indicating the shape of the workpiece 400 at the specified time. The third query generation unit 18c uses the generated query to acquire the mesh at the specified time from the third time series data 13 stored in the database 10. The third query generation unit 18c outputs the acquired mesh to the model generation module 41.

[0138] In the operation phase, the action reproducing module 23 performs the same processing as in the advance preparation phase. That is, the action reproducing module 23 generates data (1) indicating the state of the second virtual model based on the values ​​of one or more variables received from the data reproducing module 16. Furthermore, the action reproducing module 23 places the second virtual model in the virtual space so that it is in the state indicated by the generated data (1).

[0139] The model generation module 41 places the first virtual model in the virtual space so as to assume the position and orientation indicated by the coordinates received from the data reproduction module 16. Furthermore, when the model generation module 41 receives a mesh from the data reproduction module 16, it changes the outer surface of the first virtual model to the mesh.

[0140] The model display module 42 generates a virtual image that visualizes the first virtual model and the second virtual model in the virtual space. The model display module 42 generates a virtual image as viewed from a virtual viewpoint specified in response to a user input. The model display module 42 generates data of a window including the virtual image, and outputs the data to the display controller 105. As a result, the window including the virtual image is displayed on the display 120. Note that the window including the virtual image is an example of a "second window" in the present disclosure.

[0141] The waveform display module 43 generates data for a window including a graph showing the transition of the variable values ​​in the target period based on the values ​​of one or more variables received from the data reproduction module 16, and outputs the data to the display controller 105. As a result, a window including a graph showing the transition of the variable values ​​in the target period is displayed on the display 120. Note that the window including the graph showing the transition of the variable values ​​in the target period is an example of the "first window" in the present disclosure.

[0142] The program display module 44 generates data for a window showing the execution status of the control program in the target period based on the values ​​of one or more variables received from the data reproduction module 16, and outputs the data to the display controller 105. As a result, a window showing the execution status of the control program in the target period is displayed on the display 120. The window showing the execution status of the control program in the target period is an example of the "first window" in the present disclosure.

[0143] The video playback module 45 generates data for a window in which the video captured during the target period is played, based on the frames received from the data reproduction module 16. The video playback module 45 outputs the data to the display controller 105. As a result, the window in which the video captured during the target period is played is displayed on the display 120. Note that the window in which the video captured during the target period is played is an example of the "first window" of the present disclosure.

[0144] Each of the waveform display module 43, the program display module 44, and the video playback module 45 may output window data only when an enabling instruction is received.

[0145] <Preparation process flow> Fig. 11 is a flow chart showing an example of the flow of the initial advance preparation process. The flow shown in Fig. 11 is executed after the correspondence information 21a (see Fig. 5) of each movable part constituting the control target 300 and the initial information 22a of each of one or more objects (including the workpiece 400) not controlled by the control device 200 are set. The flow shown in Fig. 11 is executed in response to, for example, a user input.

[0146] First, in step S1, the processor 101 sets the start time t0 of the target period as a designated time (hereinafter referred to as "time t"). Specifically, the processor 101 sets a counter value to 0.

[0147] In the next step S2, the processor 101 performs an initialization process for the simulation. Specifically, the processor 101 executes an initialization function of a script included in the simulation program 134.

[0148] In the next step S3, the processor 101 acquires values ​​of one or more variables at time t from the first time-series data 11 stored in the database 10. Furthermore, in step S4, the processor 101 acquires a frame corresponding to time t from the video data 14 stored in the database 10.

[0149] If a first virtual model corresponding to the workpiece 400 has been generated in the virtual space, the following steps S5 to S10 are performed. If a first virtual model has not been generated in the virtual space, steps S5 to S10 are omitted. In step S5, the processor 101 determines whether or not the physical simulation flag is "0". Specifically, the processor 101 determines whether or not the physical simulation flag corresponding to counter value k-1, which is 1 less than the counter value k corresponding to time t, is "0".

[0150] If the physical simulation flag is "0" (YES in step S5), in step S6, the processor 101 executes the script to simulate the motion of the first virtual model so as to follow the motion of the second virtual model of the control object 300. The motion of the second virtual model is determined based on the values ​​of one or more variables acquired in step S4.

[0151] If the physical simulation flag is "1" (NO in step S5), in step S7, the processor 101 executes a physical simulation to simulate the movement of the first virtual model.

[0152] In step S8 after step S6 or step S7, the processor 101 analyzes the frame acquired in step S4. In the next step S9, the processor 101 generates a mesh indicating the shape of the workpiece 400 based on the analysis result of the frame.

[0153] In the next step S10, the processor 101 records in the database 10 the coordinates defining the position and orientation of the first virtual model simulated in step S7 or step S8, and the mesh generated in step S10.

[0154] In the next step S11, the processor 101 determines whether or not the time t is before the end time T of the target period. If the time t is before the end time T (YES in step S11), in step S12, the processor 101 advances the time t by the shortest control period. Specifically, the processor 101 advances the counter value by 1. After step S12, the advance preparation process returns to step S3. If the time t is not before the end time T (NO in step S11), the advance preparation process ends.

[0155] By repeatedly executing steps S3 to S10 until time t reaches end time T, second time series data 12 of the position and posture of the first virtual model during the target period and third time series data 13 of the shape of the work 400 during the target period are stored in the database 10.

[0156] It should be noted that if the shape analysis module 50 is disabled, steps S4, S8, and S9 are omitted.

[0157] 12 is a flowchart showing an example of the flow of the second or subsequent advance preparation process. For example, the user may wish to modify the initial information 22a of the workpiece 400 after the first advance preparation phase. If the physical characteristics of the workpiece 400 are modified, the simulation result of the motion of the first virtual model may change. In such a case, the second or subsequent advance preparation process is executed.

[0158] The flow shown in Fig. 12 differs from the flow shown in Fig. 11 in that it includes step S15. In step S15, the processor 101 determines whether or not it is necessary to update the second time series data 12 in response to a user input. For example, if the user has changed a parameter that affects the physical simulation (e.g., the position, mass, material, etc. of the workpiece 400), the user inputs that it is necessary to update the second time series data 12. If the user has changed only a parameter that does not affect the physical simulation, the user inputs that it is not necessary to update the second time series data 12.

[0159] If the determination in step S15 is YES, steps S1 to S12 are executed, whereas if the determination in step S15 is NO, the advance preparation process ends.

[0160] <Information provision process flow> Fig. 13 is a flowchart showing an example of the flow of the information provision process when the play button shown in Fig. 10 is clicked. First, in step S21, the processor 101 sets a time according to the position of the cursor 61 in the operation window 60 shown in Fig. 10 as a designated time (hereinafter referred to as "time t").

[0161] In the next step S22, the processor 101 acquires, from the first time series data 11 stored in the database 10, values ​​of one or more variables at time t.

[0162] In the next step S23, the processor 101 places the second virtual model in the virtual space so as to have a state of the control object 300 corresponding to the acquired values ​​of the one or more variables.

[0163] In the next step S24, the processor 101 acquires coordinates at time t from the second time series data 12 stored in the database 10. Furthermore, when the third time series data 13 is stored in the database 10, the processor 101 acquires a mesh at time t from the third time series data 13.

[0164] In the next step S25, the processor 101 places the first virtual model in the virtual space so as to assume a position and orientation defined by the coordinates acquired in step S24. Furthermore, if a mesh has been acquired in step S4, the processor 101 changes the outer surface of the first virtual model to the acquired mesh.

[0165] In the next step S26, the processor 101 outputs data of a window including a virtual image that visualizes the first virtual model and the second virtual model in the virtual space to the display controller 105. As a result, the window including the virtual image is displayed on the display 120.

[0166] In the next step S27, the processor 101 executes another visualization process. Step S27 will be described later in detail.

[0167] In the next step S28, the processor 101 determines whether or not the time t is before the end time T of the target period.

[0168] If the time t is before the end time T (YES in step S28), in step S29, when a specified time has elapsed since step S21 or the previous step S29, the processor 101 advances the time t by the playback cycle. Specifically, the processor 101 advances the counter value by a value obtained by multiplying a specified value (for example, 50 or 100) by the playback magnification. After step S29, the information provision process returns to step S22.

[0169] If the time t is not before the end time T (NO in step S28), the information provision process ends.

[0170] 13 is executed, a virtual moving image that imitates the movement of the workpiece 400 and the operation of the control target 300 during the target period is displayed on the display 120.

[0171] Fig. 14 is a flow chart showing an example of the flow of information provision processing when the frame rewind button or frame forward button shown in Fig. 10 is clicked. When time t is the start time t0 of the target period, the operation of the frame rewind button 64 is invalidated. When time t is the end time T of the target period, the operation of the frame forward button 65 is invalidated. The flow shown in Fig. 14 differs from the flow shown in Fig. 13 in that steps S28 and S29 are omitted. By executing the flow shown in Fig. 14, the user can check a virtual image at a desired timing within the target period while frame rewinding or forwarding a specified time.

[0172] Fig. 15 is a flowchart showing an example of the process flow of the subroutine of step S27 shown in Fig. 13 or 14. As shown in Fig. 15, in step S27, the processes of steps S31 and S32, the processes of steps S33 and S34, and the processes of steps S35 to S37 are carried out in parallel.

[0173] In step S31, the processor 101 determines whether or not the program display function is enabled. If the program display function is enabled (YES in step S31), in step S32, the processor 101 outputs data of a window showing the execution status of the control program to the display controller 105 based on the values ​​of one or more variables acquired in step S22. As a result, a window showing the execution status of the control program in the target period is displayed on the display 120. If the program display function is disabled (NO in step S31), step S32 is omitted.

[0174] In step S33, processor 101 determines whether or not the waveform display function is enabled. If the waveform display function is enabled (YES in step S33), in step S34, processor 101 outputs data of a window including a graph showing the transition of one or more variables based on the values ​​of one or more variables acquired in step S22 to display controller 105. As a result, a window including a graph showing the transition of the variable values ​​in the target period is displayed on display 120. If the waveform display function is disabled (NO in step S33), step S34 is omitted.

[0175] In step S35, the processor 101 determines whether or not the video playback function is enabled. If the video playback function is enabled (YES in step S35), in step S36, the processor 101 acquires a frame at time t from the video data 14 stored in the database 10. In the next step S37, the processor 101 outputs data of a window in which the acquired frame is played to the display controller 105. As a result, a window in which the video captured during the target period is played is displayed on the display 120. If the video playback function is disabled (NO in step S35), steps S36 and S37 are omitted.

[0176] <Example of information provision screen> Fig. 16 is a diagram showing an example of a screen displayed in the operation phase. Fig. 17 is a diagram showing another example of a screen displayed in the operation phase. The screen shown in Fig. 16 is displayed when the program display function and the video playback function are enabled. The screen shown in Fig. 17 is displayed when the program display function, the waveform display function, and the video playback function are enabled.

[0177] As shown in Fig. 16, an operation window 60, a window 80 including a virtual image, a window 82 showing the execution status of a control program, and a window 84 in which a moving image is played are displayed side by side on the display 120. As shown in Fig. 17, an operation window 60, a window 80 including a virtual image, a window 82 showing the execution status of a control program, a window 84 in which a moving image is played, and a window 86 including a graph showing the transition of the value of a variable are displayed side by side on the display 120. In the window 86, the graph includes a line L1 representing a specified time.

[0178] Based on the screen shown in Figure 16 or Figure 17, the user can check the execution status of the control program, the changes in the values ​​of variables, or actual video along with virtual images that visualize the first virtual model and the second virtual model that mimic the work 400 and the control object 300 during the target period.

[0179] The window 80 includes a group of icons 80a for changing the scale of the virtual image, the virtual viewpoint, etc. The processor 101 updates the virtual image in response to an operation on the group of icons 80a. This allows the user to check the state of a part in the virtual image that cannot be checked in the real image.

[0180] <Modification> In the above description, the information processing system 100 is configured by one PC 100A. However, the information processing system 100 may be configured by multiple computers. That is, the information processing system 100 may include multiple processors. Alternatively, the information processing system 100 may include a hardware environment other than a general-purpose PC, such as a virtual machine on a cloud.

[0181] The information processing system 100 may include a plurality of databases. In this case, the first time series data 11, the second time series data 12, and the third time series data 13 may be stored in different databases.

[0182] The first query generating unit 18a may acquire values ​​of all variables from the first time-series data 11, or may acquire values ​​of some variables. The variables to be acquired may be different from each other in the action reproduction module 23, the waveform display module 43, and the program display module 44. The variables to be acquired may be specified by the user. When developing a control program, the variable name of each variable used by the control program is set according to a user input. Therefore, the variables to be acquired are specified by the variable name. On the other hand, the first time-series data 11 indicates a variable ID instead of a variable name for each variable (see FIG. 8). Therefore, the first query generating unit 18a may convert the variable name to be acquired into a variable ID using a conversion table.

[0183] Fig. 18 is a diagram showing an example of a conversion table referred to by the first query generating unit. As shown in Fig. 18, conversion table 181 associates a variable name with a variable ID. Conversion table 181 can be generated before constructing first time series data 11 or when constructing first time series data 11.

[0184] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0185] (Configuration 1) An information processing system (100, 100A), One or more databases (10) storing first time series data (11) of values ​​of one or more variables representing a control state of a control device (200) during a target period; a simulator (20, 24, 25, 101) that simulates the motion of a first virtual model of the object (400) during the target period based on the first time-series data (11), a dependency relationship between a control target (300) of the control device (200) and an object (400) not controlled by the control device (200) during the target period, and physical properties of the object (400); a data acquisition module (30, 101) that acquires second time-series data (12) of the position and orientation of the first virtual model during the target period from a simulation result by the simulator (20, 24, 25) and stores the second time-series data (12) in the one or more databases (10); a first visualization module (40, 101) that generates a virtual image in which the first virtual model and the second virtual model of the control target are visualized in an overlapping manner; The first visualization module (40, 101) visualizes, in the virtual image, visualizing the first virtual model so as to assume a position and orientation at a specified time indicated by the second time-series data (12); An information processing system (100, 100A) that visualizes the second virtual model so as to have a state of the controlled object (300) corresponding to the values ​​of the one or more variables at the specified time indicated by the first time series data (11).

[0186] (Configuration 2) a shape analysis module (50, 101) that analyzes a change in shape of the object (400) during the target period based on a moving image obtained by photographing the object (400); the data acquisition module (30, 101) acquires third time series data (13) of the shape of the object (400) during the target period from the analysis result by the shape analysis module (50, 101), and stores the third time series data (13) in the one or more databases (10); The information processing system (100, 100A) of configuration 1, wherein the first visualization module (40, 101) visualizes the first virtual model in the virtual image so that it has a shape at the specified time indicated by the third time series data (13).

[0187] (Configuration 3) The simulator (20, 24, 25, 101) simulating an operation of the second virtual model based on values ​​of the one or more variables represented by the first time-series data (11) during a first section of the target period, and simulating a motion of the first virtual model so as to follow the motion of the second virtual model; The information processing system (100, 100A) according to configuration 1 or 2, which simulates the motion of the first virtual model based on the physical characteristics in a second section other than the first section of the target period.

[0188] (Configuration 4) one or more second visualization modules (43, 44, 101) that generate data for a first window (82, 86) in which values ​​of the one or more variables at the specified time are visualized; An information processing system (100, 100A) according to any one of configurations 1 to 3, further comprising a display controller (105) that displays the first window (82, 86) and a second window (80) including the virtual image side-by-side on a display (120).

[0189] (Configuration 5) The information processing system (100, 100A) according to configuration 4, wherein the one or more second visualization modules (43, 44, 101) include at least one of a waveform display module (43, 101) that visualizes a graph showing the transition of values ​​of the one or more variables, and a program display module (44, 101) that visualizes the execution status of a control program for controlling the control target in the control device.

[0190] (Configuration 6) a video playback module (45, 101) that generates data of a first window (84) including a frame at the specified time from among the video images obtained by shooting the control target (300); An information processing system (100, 100A) according to any one of configurations 1 to 3, further comprising a display controller (105) that displays the first window (84) and a second window (80) including the virtual image side-by-side on a display (120).

[0191] (Configuration 7) An information processing method using one or more databases (10) storing first time series data (11) of values ​​of one or more variables representing a control state of a control device (200) during a target period, comprising: One or more processors (101) simulate (S6, S7) a motion of a first virtual model of the object (400) during the target period based on the first time-series data (11), a dependency relationship between a control target (300) of the control device (200) and an object (400) not controlled by the control device (200) during the target period, and physical properties of the object (400); The one or more processors (101) acquire second time-series data (12) of the position and orientation of the first virtual model during the target period from a simulation result, and store the second time-series data (12) in the one or more databases (10) (S10); The one or more processors (101) generate a virtual image in which the first virtual model and a second virtual model of the control target (300) are visualized in an overlapping manner (S22 to S26); The generating of the virtual image (S22 to S26) includes: The one or more processors (101) visualize (S25) the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time-series data (12); The one or more processors (101) visualize (S23) the second virtual model in the virtual image so as to have a state of the control object (300) corresponding to the values ​​of the one or more variables at the specified time indicated by the first time series data (11).

[0192] (Configuration 8) A program (133, 134, 136) for causing a computer (100A) to execute an information processing method using one or more databases (10) storing first time series data (11) of values ​​of one or more variables representing a control state of a control device (200) during a target period, The information processing method includes: simulating (S6, S7) the motion of a first virtual model of the object (400) during the target period based on the first time-series data (11), a dependency relationship between a control target (300) of the control device (200) and an object (400) not controlled by the control device (200) during the target period, and physical properties of the object (400); acquiring second time series data (12) of the position and orientation of the first virtual model during the target period from a simulation result, and storing the second time series data (12) in the one or more databases (10) (S10); generating a virtual image in which the first virtual model and a second virtual model of the control object (300) are visualized in an overlapping manner (S22 to S26); The generating of the virtual image (S22 to S26) includes: Visualizing the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time-series data (12) (S25); and visualizing (S23) the second virtual model in the virtual image so as to have a state of the control object (300) corresponding to the value of the one or more variables at the specified time indicated by the first time series data (11).

[0193] Although the embodiment of the present invention has been described, the embodiment disclosed herein should be considered as illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0194] 1 manufacturing system, 10 database, 11 first time series data, 11a, 12a, 13a data piece, 12 second time series data, 13 third time series data, 14 video data, 15 time designation module, 16 data reproduction module, 17 time control module, 18 data request module, 18a first query generation unit, 18b second query generation unit, 18c third query generation unit, 20 simulator, 21 operation setting module, 21a correspondence information, 22 initial information setting module, 22a initial information, 23 operation reproduction module, 24 script execution module, 25 physical simulation execution module, 26 data management module, 30 data acquisition module, 31 coordinate acquisition module, 32 shape acquisition module, 40 first visualization module, 41 model generation module, 42 model display module, 43 waveform display module, 44 program display module, 45 video playback module, 50 shape analysis module, 51 video analysis module, 52 Mesh Generation Module, 60 Operation Window, 61 Cursor 62,66 input field, 63 play button, 64 frame rewind button, 65 frame forward button, 80,82,84,86 window, 80a icon group, 100 information processing system, 100A PC, 101 processor, 102,206 memory, 103,208 storage, 104 input interface, 105 display controller, 106 communication interface, 107,270 reader / writer, 110 input device, 120 display, 131 development program, 132 viewer program, 133 data reproduction program, 134 simulation program, 135 shape analysis program, 136 database construction program, 181 conversion table, 200 control device, 201 power supply circuit, 202 CPU, 204 chipset, 207 buffer area, 211 system program, 213 scheduler program, 214 program data set, 250 USB controller, 252 Field network controller, 252a, 254a connectors, 254 network controller, 260 timer, 300 controlled object, 400 work, 500 network camera, 700 storage medium.

Claims

1. An information processing system, one or more databases storing first time series data of values ​​of one or more variables representing a control state of the control device during a target period; a simulator that simulates the motion of a first virtual model of the object during the target period based on the first time-series data, a dependency relationship between a control target of the control device and an object not controlled by the control device during the target period, and physical properties of the object; a data acquisition module that acquires second time series data of a position and orientation of the first virtual model during the target period from a simulation result by the simulator, and stores the second time series data in the one or more databases; a first visualization module that generates a virtual image in which the first virtual model and the second virtual model of the control target are visualized in an overlapping manner, The first visualization module, in the virtual image, visualizing the first virtual model so as to assume a position and orientation at a specified time indicated by the second time-series data; An information processing system that visualizes the second virtual model so as to have a state of the controlled object corresponding to the values ​​of the one or more variables at the specified time indicated by the first time series data.

2. a shape analysis module that analyzes a change in shape of the object during the target period based on a moving image obtained by photographing the object; the data acquisition module acquires third time series data of the shape of the object during the target period from the analysis result by the shape analysis module, and stores the third time series data in the one or more databases; The information processing system according to claim 1 , wherein the first visualization module visualizes the first virtual model in the virtual image so as to have a shape at the specified time indicated by the third time-series data.

3. The simulator comprises: simulating an operation of the second virtual model based on values ​​of the one or more variables indicated by the first time-series data during a first section of the target period, and simulating a motion of the first virtual model so as to follow the motion of the second virtual model; The information processing system according to claim 1 , further comprising: simulating a motion of the first virtual model based on the physical characteristics in a second section other than the first section of the target period.

4. one or more second visualization modules that generate a first window of data in which values ​​of the one or more variables at the specified time are visualized; The information processing system according to claim 1 , further comprising a display controller that displays the first window and a second window including the virtual image side-by-side on a display.

5. 5. The information processing system according to claim 4, wherein the one or more second visualization modules include at least one of a waveform display module that visualizes a graph showing a change in the value of the one or more variables, and a program display module that visualizes an execution status of a control program for controlling the control target in the control device.

6. a video playback module that generates data of a first window including a frame at the specified time out of the video obtained by shooting the control target; The information processing system according to claim 1 , further comprising a display controller that displays the first window and a second window including the virtual image side-by-side on a display.

7. An information processing method using one or more databases that store first time series data of values ​​of one or more variables that represent a control state of a control device during a target period, comprising: One or more processors simulate a motion of a first virtual model of the object during the target period based on the first time series data, a dependency relationship between a control target of the control device and an object not controlled by the control device during the target period, and physical properties of the object; The one or more processors acquire second time series data of a position and orientation of the first virtual model during the target period from a simulation result, and store the second time series data in the one or more databases; The one or more processors generate a virtual image in which the first virtual model and the second virtual model of the control object are visualized in an overlapping manner, generating the virtual image The one or more processors visualize the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time-series data; An information processing method comprising: the one or more processors visualizing the second virtual model in the virtual image so as to have a state of the controlled object corresponding to the values ​​of the one or more variables at the specified time indicated by the first time series data.

8. A program for causing a computer to execute an information processing method using one or more databases that store first time series data of values ​​of one or more variables that represent a control state of a control device during a target period, The information processing method includes: simulating a motion of a first virtual model of the object during the target period based on the first time series data, a dependency relationship between a control target of the control device and an object not controlled by the control device during the target period, and physical properties of the object; obtaining second time series data of a position and orientation of the first virtual model during the target period from a simulation result, and storing the second time series data in the one or more databases; generating a virtual image in which the first virtual model and the second virtual model of the control target are visualized in an overlapping manner, generating the virtual image visualizing the first virtual model in the virtual image so as to assume a position and orientation at a specified time indicated by the second time-series data; and visualizing the second virtual model in the virtual image so as to have a state of the control object corresponding to values ​​of the one or more variables at the specified time indicated by the first time series data.

Images