Method for creating simulation data, method for displaying preview images, and simulation device.

Abstract

This technology provides the ability to understand the simulation content before constructing the virtual space. [Solution] The method for creating simulation data includes the steps of (a) operating a virtual robot placed in a virtual workspace, which is a virtual three-dimensional workspace, based on user operation instructions, and displaying a simulator image representing the operation of the virtual robot in the virtual workspace on a display device; (b) outputting a snapshot representing a two-dimensional image obtained by capturing the state in which virtual three-dimensional objects, including the virtual robot, are placed in the virtual workspace with a virtual camera; and (c) saving an operation program representing the target operation of the virtual robot based on operation instructions, layout data representing the position and orientation of the object in the virtual workspace, attribute data representing the attributes of the object, and the snapshot as simulation data in memory.

Description

Technical Field

[0001] The present disclosure relates to a method for creating simulation data, a method for displaying a preview image, and a simulation device.

Background Art

[0002] Patent Document 1 describes a technique related to a simulation device that calculates the behavior of a robot.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a desire to save data indicating information on the behavior of a robot calculated by simulation and, for example, to confirm the behavior of the robot in a virtual space using the above data on a computer different from the computer on which the simulation was executed. In this case, it is assumed that data indicating information on the behavior of the robot calculated by simulation is saved. In the technique described in Patent Document 1, saving data indicating information on the behavior of the robot calculated by simulation is not considered. Also, since the process of constructing a virtual space is highly loaded, a technique for saving simulation data in a manner that allows a user to grasp the content of the simulation before constructing the virtual space has been desired.

Means for Solving the Problems

[0005] The present disclosure can be realized in the following forms.

[0006] A first embodiment of this disclosure provides a method for creating simulation data. This method for creating simulation data includes: (a) operating a virtual robot placed in a virtual workspace, which is a virtual three-dimensional workspace, based on user operation instructions, and displaying a simulator image representing the operation of the virtual robot in the virtual workspace on a display device; (b) outputting a snapshot representing a two-dimensional image obtained by capturing a state in which a virtual three-dimensional object, including the virtual robot, is placed in the virtual workspace with a virtual camera; and (c) storing in memory as simulation data an operation program representing the target operation of the virtual robot based on the operation instructions, layout data representing the position and orientation of the object in the virtual workspace, attribute data representing the attributes of the object, and the snapshot.

[0007] A second embodiment of this disclosure provides a method for displaying a preview image using simulation data. The simulation data includes (i) an action program consisting of a plurality of commands for causing a robot to perform a target action; (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace; (iii) attribute data representing the attributes of the objects; and (iv) a snapshot, which is a two-dimensional image representing the state of the virtual workspace. The display method includes (a) a step of receiving a selection of the simulation data; (b) a step of reading the snapshot included in the selected simulation data from memory; (c) a step of displaying the snapshot as a preview image on the display device in a manner that allows the selected simulation data to be identified; (d) a step of reading the layout data and attribute data included in the selected simulation data from memory when an instruction to construct a simulator environment is received; and (f) a step of constructing the simulator environment in which the objects are placed in the virtual workspace using the read layout data and attribute data, and displaying a simulator image on the display device representing the action of the virtual robot placed in the virtual workspace.

[0008] A third embodiment of this disclosure provides a simulation device that performs processing using simulation data. The simulation data includes (i) an action program consisting of a plurality of commands that cause a robot to perform a target action; (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace; (iii) attribute data representing the attributes of the objects; and (iv) a snapshot which is a two-dimensional image representing the state of the virtual workspace. The simulation device includes a simulator unit which constructs a simulator environment in which the objects are placed in the virtual workspace using the layout data and attribute data, operates the virtual robot placed in the virtual workspace in the simulator environment based on user operation instructions, and displays a simulator image representing the operation of the virtual robot placed in the virtual workspace on a display device; a snapshot output unit which outputs a snapshot which is a two-dimensional image obtained by capturing the state in which the objects, including the virtual robot, are placed in the virtual workspace with a virtual camera; and a data storage unit which stores simulation data, including an operation program representing the target operation of the virtual robot based on the operation instructions, the layout data, the attribute data, and the snapshot, in memory. When the simulator unit receives a selection of desired simulation data from the user, it reads the snapshot included in the selected simulation data from memory and displays the snapshot as a preview image on the display device in a manner that allows the selected simulation data to be identified. When the simulator unit receives an instruction from the user to build a simulator environment, it reads the layout data and attribute data included in the selected simulation data from memory, builds the simulator environment using the read layout data and attribute data, and displays the simulator image on the display device. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram showing the overall configuration of the robot system according to this embodiment. [Figure 2] This is a block diagram showing the functional configuration of an information processing device. [Figure 3] This flowchart shows the process for selecting simulation files. [Figure 4] This is an explanatory diagram showing an example of a folder selection screen. [Figure 5] This is an explanatory diagram showing an example of a confirmation screen. [Figure 6] This is an explanatory diagram showing an example of a simulator screen. [Figure 7] This flowchart shows the process of saving the simulation files. [Figure 8] Figure 7 is a flowchart showing the process of generating a snapshot of step S250. [Figure 9] This is a flowchart showing the process for generating a snapshot according to the second embodiment. [Figure 10] This is an explanatory diagram showing an example of a screen that displays the Pareto solution. [Modes for carrying out the invention]

[0010] A. First Embodiment: Figure 1 is a schematic diagram showing the overall configuration of the robot system 10 according to this embodiment. As shown in Figure 1, the robot system 10 comprises a robot 100, an end effector EE, a controller 500 that controls the robot 100, and an information processing device 700. The information processing device 700 is, for example, a personal computer. The end effector EE is, for example, a gripper.

[0011] In Figure 1, the robot coordinate system RC is set. The robot coordinate system RC is a three-dimensional Cartesian coordinate system with the origin at a predetermined position on the robot 100. For example, the origin can be set at any position on the base 105. The X and Y axes are axes along the horizontal direction, and the Z axis is an axis along the vertical direction.

[0012] Robot 100 is a vertical articulated robot with six joints. Robot 100 is driven by controller 500. Robot 100 comprises a base 105 that supports the arm 120, the arm 120, a force sensor 140, and drive mechanisms that drive each joint. The arm 120 includes six arm elements and joints J1 to J6 that connect the arm elements. Note that the illustration of each drive mechanism that drives joints J1 to J6 is omitted. The force sensor 140 detects the magnitude of forces parallel to the X, Y, and Z axes acting on the end effector EE, and the magnitude of torque around each axis, in a sensor coordinate system different from the robot coordinate system RC. The sensor coordinate system is a three-dimensional Cartesian coordinate system with the origin at any position of the force sensor 140. The force sensor 140 outputs the detected values ​​to controller 500.

[0013] The control point of the robot 100 is set, for example, at a predetermined position at the tip of the arm 120. The control point is the reference point for controlling the robot 100 in the robot coordinate system RC. The control point is sometimes called the TCP (Tool Center Point). The position of the control point in the robot coordinate system RC can be expressed by the position in the X-axis direction, the Y-axis direction, and the Z-axis direction. The orientation of the control point in the robot coordinate system RC can be expressed by the rotation angle around the X-axis, the Y-axis direction, and the Z-axis direction. Each drive mechanism corresponding to joints J1 to J6 drives the corresponding joint according to the control of the controller 500, so that the end effector EE is positioned in the specified position and orientation in the robot coordinate system RC.

[0014] FIG. 2 is a block diagram showing the functional configuration of the information processing apparatus 700. The information processing apparatus 700 executes a simulation in a state where a virtual robot corresponding to the robot 100 is arranged in a virtual work space. The virtual robot may also be referred to as a "virtual robot". By executing the simulation, for example, it is possible to teach the operation of the robot 100 and confirm the arrangement of the robot 100 in the work space. The information processing apparatus 700 may also be referred to as a "simulation apparatus". Hereinafter, the virtual work space will be referred to as a "virtual work space". In the virtual work space, one or more virtual objects corresponding to one or more objects actually arranged around the robot 100 when the robot 100 actually operates are arranged. Objects actually arranged around the robot 100 in the actual work space are, for example, a belt conveyor and a workbench.

[0015] The information processing apparatus 700 includes a memory 710, an interface circuit 720, an input device 730 and a display device 740 connected to the interface circuit 720, and a CPU 750 as a processor. A controller 500 is further connected to the interface circuit 720. The information processing apparatus 700 and the controller 500 can communicate with each other. For example, the information processing apparatus 700 can supply an operation command based on an operation program to the controller 500, and the controller 500 can operate the robot 100 based on the given operation command. The input device 730 is, for example, a keyboard or a mouse. The display device 300 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0016] The memory 710 stores a simulation program SP, robot model data RD, object model data OD, user-defined model data UD, layout data LD, operation program RP, and snapshot SD.

[0017] When the CPU 750 executes the simulation program SP, the functions of the simulator 751, the snapshot output unit 752, and the data storage unit 753, which will be described later, are realized. The simulator 751 is also referred to as the "simulator unit".

[0018] The robot model data RD includes data on various robot characteristics including the configuration and movable range of the arm 120. Further, the robot model data RD includes 3D CAD (Computer Aided Design) data representing the three-dimensional shape of the robot. When the robot is also treated as one of the objects, the data representing the three-dimensional shape of the robot is included in the "attribute data" representing the attributes of the object. The three-dimensional shape of the robot may be represented by a polygon mesh which is a set of vertices, edges, and faces defining the shape of the robot.

[0019] The object model data OD includes 3D CAD data representing the three-dimensional shape of an object that can be placed in the virtual work space. In the object model data OD, an object model whose shape is defined in advance is defined. The object models whose shapes are defined in advance are, for example, a sphere, a rectangular parallelepiped, and a cylinder.

[0020] The user-defined model data UD includes the three-dimensional CAD data representing the three-dimensional shape of an object that can be placed in the virtual work space and that is uniquely defined by the user. The data representing the three-dimensional shape of the object is included in the "attribute data" representing the attributes of the object. The three-dimensional shape of the object may be represented by a polygon mesh which is a set of vertices, edges, and faces defining the shape of the object.

[0021] The layout data LD includes data on the position and orientation of the robot and one or more objects placed in the virtual workspace. For example, the position of the robot and one or more objects is represented by coordinate values ​​in the coordinate system set in the virtual workspace. For example, the orientation of the robot and one or more objects is represented by quaternions, Euler angles, etc. The layout data LD further includes data on the type of robot, and the type, size, and color of one or more objects placed in the robot's workspace. The size and color of the robot are uniquely determined according to the type of robot. The layout data LD is created or edited in the simulation described later. The type of robot and the type, size, and color of one or more objects are also called "object attributes". In this embodiment, the layout data LD includes some "attribute data" that represents the attributes of the objects.

[0022] The motion program RP consists of multiple motion commands to cause the robot 100 to perform a target action. The motion program RP is created or edited in the simulation described later.

[0023] A snapshot SD is a two-dimensional image representing the state of a robot and one or more objects in a virtual workspace. It represents the state of the robot and one or more objects at a specific moment in time. Snapshot SDs are created during the simulation described later.

[0024] In this embodiment, the layout data LD, the operation program RP, and the snapshot SD may be treated as a single unit. The layout data LD, the operation program RP, and the snapshot SD are collectively referred to as the simulation file group SF. In this embodiment, the mutually corresponding layout data LD, operation program RP, and snapshot SD are stored in memory 710 in a single folder. The simulation file group SF is also referred to as "simulation data".

[0025] The CPU 750 functions as a simulator 751, a snapshot output unit 752 that outputs a snapshot SD, and a data storage unit 753 that saves the simulation file group SF to memory 710 by executing the simulation program SP.

[0026] Simulator 751 constructs a simulator environment in which objects are placed in a virtual workspace using layout data LD, robot model data RD, object model data OD, and user-defined model data UD. Simulator 751 operates the robot placed in the virtual workspace within the simulator environment and displays a simulator image representing the robot's operation in the virtual workspace on display device 740.

[0027] Figure 3 is a flowchart showing the process for selecting a set of simulation files in this embodiment. When the user starts the simulation program SP, the process shown in Figure 3 begins. In this embodiment, an example using the set of simulation files SF created by the previous simulation will be described.

[0028] In step S101, the selection of the simulation file group SF is accepted. Figure 4 is an explanatory diagram showing an example of the folder selection screen SC1. As mentioned above, the simulator file group is stored in memory 710 in a single folder. When the simulation program SP is started, for example, the selection screen SC1 shown in Figure 4 is displayed on the display device 740. The selection screen SC1 displays a list of folders that already contain the created simulation file group SF. The user selects the desired folder on the selection screen SC1 using the input device 730. When the "OK" button BT1 is pressed, the selection of the simulation file group SF is accepted. When the "Cancel" button BT2 is pressed, the selection screen SC1 is closed. The processing in step S101 is executed by the CPU 750, which functions as the simulator 751.

[0029] As shown in Figure 3, in step S103, it is determined whether a valid set of simulation files SF has been selected. Selecting a valid set of simulation files SF means that the user has selected a folder containing the set of simulation files SF, which includes the layout data LD, the operation program RP, and the snapshot SD. If a valid set of simulation files SF is selected (step S103; YES), the process in step S105 is executed. If a valid set of simulation files SF is not selected (step S103; NO), for example, a notification that a valid set of simulation files SF has not been selected is displayed on a screen (not shown), and then the process in step S101 is executed again. The process in step S103 is executed by the CPU 750, which functions as the simulator 751.

[0030] In step S105, a confirmation screen SC2 containing a snapshot SD included in the selected simulation file group SF is displayed on the display device 740.

[0031] Figure 5 is an explanatory diagram showing an example of the confirmation screen SC2. The confirmation screen SC2 includes a display area F1 that shows the name of the folder where the selected simulator files are stored, a display area F2 that shows the snapshot SD, a display area F3 that shows the layout of the robot and one or more objects placed in the virtual workspace, and a display area F4 that shows the attributes of the objects. In the confirmation screen SC2, the snapshot SD is displayed as a preview image. In display area F3, two or more hierarchical objects are displayed in a manner that makes it visually apparent that they are hierarchical. Two or more objects being hierarchical means that two or more objects are grouped together and the rank of the objects is defined. For example, "SBox_2" and "SBox_3" are grouped together, and "SBox_2" is set to a higher rank than "SBox_3". In other words, for example, "SBox_2" is an object in the upper hierarchy, and "SBox_3" is an object in the lower hierarchy. For example, if the placement position of an object in the upper hierarchy of two or more hierarchical objects is changed in display area F3 by user operation, the object in the lower hierarchy will also be moved along with the object in the upper hierarchy.

[0032] To display the confirmation screen SC2, first, the snapshot SD included in the selected simulation file group SF is read from memory 710. Furthermore, the layout data LD included in the selected simulation file group is read from memory 710. Based on the read snapshot SD and layout data LD, the confirmation screen SC2 is generated. The process in step S105 is executed by the CPU 750, which functions as the simulator 751.

[0033] As shown in Figure 3, in step S107, it is determined whether or not a restore has been instructed. If the "Cancel" button BT5 is pressed on the confirmation screen SC2, it is determined that the restore has been canceled. If the restore is canceled (step S107; NO), the confirmation screen SC2 is closed. Then, the process in step S101 is executed again. If the "OK" button BT4 is pressed on the confirmation screen SC2, it is determined that a restore has been instructed. If the restore has been instructed (step S107; YES), the process in step S109 is executed. The instruction to restore is also called the "instruction to build the simulator environment". Note that if the "Select" button BT3 is pressed on the confirmation screen SC2, the confirmation screen SC2 is not closed, and the selection screen SC1 is displayed again. In this case, the user can re-select a folder on the selection screen SC1. The process in step S107 is executed by the CPU 750, which functions as the simulator 751.

[0034] In step S109, a restore is performed based on the selected simulation file set SF. First, the layout data LD, robot model data RD, and object model data OD included in the selected simulation file set SF are read from memory 710. If user-defined model data UD is included, the user-defined model data UD is also read from memory 710. Based on the layout data LD and robot model data RD, a 3D object of the robot corresponding to its appearance in the virtual workspace represented as a 3D space is generated. Based on the layout data LD and object model data OD, a 3D object of the object corresponding to its appearance in the virtual 3D workspace is generated. If the object to be placed is an object uniquely defined by the user, the user-defined model data UD is used instead of the object model data OD to generate the 3D object. Each generated 3D object is placed in the virtual workspace based on the layout data LD. In this way, the simulator environment is constructed.

[0035] Figure 6 is an explanatory diagram showing an example of the simulator screen SC3. The simulator screen SC3 is displayed on the display device 740 once the simulator environment is established. Here, the display device 740 displays an image IM_S representing the virtual workspace as viewed from a predetermined line of sight. In the virtual workspace, the robot and one or more objects are arranged in positions and orientations based on the layout data LD. In the simulator environment, the positions and orientations of the robot and objects are changed according to the user's operation instructions. Furthermore, in the simulator environment, the robot can be operated according to the motion program RP. In addition, in the simulator environment, the robot's operation can be changed according to the user's operation instructions, and the motion program RP can be updated to reflect the changed operation.

[0036] As explained above, in this embodiment, when the user selects the folder containing the simulation file group SF, the simulation environment is not built, and a confirmation screen SC2 including the snapshot SD is displayed (see Figure 5). If restore is instructed on the confirmation screen SC2, the robot model data RD, object model data OD, and user-defined model data UD are read from memory 710, and the process of generating 3D objects is executed in order to build the simulator environment. Generally, in a simulation, the state in which virtual 3D objects, including a virtual robot, are placed in a virtual workspace is determined, and attribute information such as shape is assigned to each object. For this reason, processing such as reading the shape data of each object and reproducing the shape of each object is necessary. Furthermore, the more objects placed in the virtual workspace, the higher the load of these processes becomes.

[0037] In this embodiment, when the user selects a folder containing the simulation file set SF, the robot model data RD, object model data OD, etc., are not read from memory 710, but the snapshot SD and layout data LD are read from memory 710. The snapshot SD is then displayed as a preview image. Therefore, before executing processes such as reading shape data for each object and reproducing the shapes of each object to build the simulator environment, the user can check the snapshot SD as a preview image representing the contents of the selected folder. Thus, the user can confirm that the data stored in the selected folder is data related to the desired simulation environment. Therefore, the occurrence of errors such as executing a process to build a simulation environment using incorrect simulation data can be suppressed.

[0038] Next, we will explain the process of saving the simulation file set SF. Figure 7 is a flowchart of the process for saving the simulation file set SF. The process shown in Figure 7 is executed when the simulation environment has been set up and the user instructs to save the simulation file set SF.

[0039] In step S210, layout data LD reflecting the current simulation environment is saved to memory 710. In step S230, operation program RP reflecting the current simulation environment is saved to memory 710. In step S250, a snapshot SD is generated. Details of the process for generating the snapshot SD will be described later. In step S270, the snapshot SD is saved to memory 710. The layout data LD, operation program RP, and snapshot SD are stored in memory 710 in a single folder. After that, the process shown in Figure 7 is completed. Steps S210, S230, and S270 are executed by the processor, which functions as a data storage unit 753. Step S250 is executed by the CPU 750, which functions as a snapshot output unit 752.

[0040] Figure 8 is a flowchart showing the process of generating a snapshot SD in step S250 as shown in Figure 7.

[0041] In step S251, a virtual camera is positioned in the virtual workspace with its lens facing the center of gravity. In this embodiment, the center of gravity is defined as the position representing the center between multiple objects placed in the virtual workspace. Here, the robot is also treated as one of the objects. For example, when multiple objects are viewed from above in the virtual workspace, the center of gravity of the polygon drawn by connecting the centers of each object with a straight line is determined as the center of gravity. If there are only two objects, a polygon cannot be drawn, so the midpoint of the line segment connecting the two objects is determined as the center of gravity. The position of the virtual camera is represented by coordinate values ​​in the coordinate system set in the virtual workspace. The orientation of the virtual camera is represented by quaternions, Euler angles, etc. For example, the virtual camera is pointed towards the center of gravity determined from a predetermined direction. By pointing the virtual camera at the center of gravity of a set of objects, it becomes more likely that as many objects as possible will be included in the virtual camera's shooting range. Here, it is not necessarily required to place the virtual camera within the virtual workspace; it is sufficient to determine the location where the virtual camera will be placed. The same applies to the following steps.

[0042] Alternatively, a different calculation method may be used to determine the center of gravity. For example, the center of gravity may be calculated by dividing the sum of the coordinate positions of all objects placed in the virtual workspace by the number of objects and averaging the result. Specifically, the center of gravity can be calculated using the following formula (M1). In the following formula (M1), n ​​represents the number of objects. i This represents the coordinate value of an object in virtual space. For example, if the position in the virtual workspace is represented in a Cartesian coordinate system, then P i represents the coordinates (x, y, z) of the i-th object. In equation (M1), P i The sums of the x, y, and z coordinates are calculated as the sums of the x, y, and z coordinates.

[0043]

number

[0044] In step S253, the position and orientation of the virtual camera are adjusted.

[0045] The position and orientation of the virtual camera are adjusted as follows: The virtual camera must be positioned so that at least the robot is included within its field of view. For example, the position of the virtual camera may be changed by adding a constant value to the X, Y, and Z coordinates of the virtual camera in the coordinate system set in the virtual workspace, and setting the modified coordinates as the new position of the virtual camera. Alternatively, the position and orientation of the virtual camera may be changed so that the distance between the virtual camera and any object is reduced by a predetermined distance. Any object may be the robot, the object closest to the virtual camera, or the object furthest from the virtual camera.

[0046] In step S255, it is determined whether or not shooting is possible. For example, it is determined that shooting is possible if all of the following conditions are met. (1) The virtual camera's field of view includes at least the robot. (2) The straight-line distance between the virtual camera and the selected object is within a predetermined range.

[0047] Regarding (1) above, for example, if the angle between the optical axis of the virtual camera and the line connecting the virtual camera and a predetermined part of the robot is less than or equal to half the field of view of the virtual camera, then it can be determined that the robot is included in the shooting range of the virtual camera. Half the field of view is half the field of view. In addition, the same method can be used to determine whether objects other than robots are included in the shooting range of the virtual camera.

[0048] Regarding (2) above, the straight-line distance between the virtual camera and the selected object can be calculated based on the coordinate values ​​of the virtual camera and the selected object in the coordinate system set in the virtual workspace. The selected object may be a robot. The selected object may also be the object closest to the virtual camera. If the straight-line distance exceeds the upper limit of a predetermined range, it indicates that the virtual camera is too far from the subject being photographed. If the straight-line distance falls below the lower limit of a predetermined range, it indicates that the virtual camera is too close to the subject being photographed.

[0049] If it is determined that shooting is possible (step S255; YES), step S257 is executed. If it is determined that shooting is not possible (step S255; NO), the process in step S253 is executed again.

[0050] In step S257, a snapshot SD is generated. Specifically, a 2D image obtained by a virtual camera capturing the state in which an object is placed in the virtual workspace, at the virtual camera's position and orientation when it is determined that shooting is possible, is generated as the snapshot SD. The above describes the process for generating the snapshot SD.

[0051] B. Second Embodiment: In the second embodiment, an optimization process is used as a method for determining the position and orientation of the virtual camera when generating a snapshot SD. The following description will focus on configurations that differ from the first embodiment, and will omit descriptions of configurations similar to the first embodiment. In this embodiment, it is assumed that multiple objects, including a robot, are placed in the virtual workspace. As the optimization process method, multi-objective optimization algorithms such as NSGA-II (Non-dominated Sorting Genetic Algorithms II) and SPEA2 (Strength Pareto Evolutionary Algorithm) are used.

[0052] Figure 9 is a flowchart showing the process for generating a snapshot according to this embodiment.

[0053] Steps S251 and S253 are the same as in the first embodiment. In step S255b, it is determined whether or not photography is possible. For example, if any object is included in the shooting range, it is determined that photography is possible.

[0054] If it is determined that shooting is not possible (step S255b; NO), the process in step S253 is executed again. If it is determined that shooting is possible (step S255b; YES), step S257b is executed.

[0055] In step S257b, a virtual camera captures the state in which the object is placed in the virtual workspace. Specifically, a 2D image is generated obtained by the virtual camera capturing the state in which the object is placed in the virtual workspace at the position and orientation of the virtual camera when it is determined that capturing is possible.

[0056] In step S259, the number of objects in the 2D image obtained in step S257b and the ratio of the area occupied by objects to the entire image are calculated. Objects in the image can be detected using known object detection algorithms. In this embodiment, the number of objects in the 2D image (hereinafter referred to as evaluation value A) and the ratio of the area occupied by objects to the entire image (hereinafter referred to as evaluation value B) are values ​​that should be optimized. The calculated evaluation values ​​A and B are stored in memory 710 in association with information that identifies the 2D image.

[0057] In step S261, the optimization process is performed using evaluation values ​​A and B as initial values. Here, NSGA-II is used as the optimization algorithm. Constraints are set, for example, that evaluation value A is not greater than or equal to a predetermined first value, and that evaluation value B is not greater than or equal to a predetermined second value. If evaluation value A is greater than or equal to the predetermined first value, it indicates that there are too many objects in the image. If there are too many objects in the image, it is assumed that the virtual camera is too far away from the objects. If evaluation value B is greater than or equal to the predetermined second value, it indicates that the area occupied by the objects in the image is too large. If the area occupied by the objects in the image is too large, it is assumed that the virtual camera is too close to the objects.

[0058] In step S263, it is determined whether the termination condition is met. For example, if the process in step S261 has been executed a predetermined number of times, it is determined that the termination condition is met. If it is determined that the termination condition is not met (step S263; NO), step S253 is executed again. If step S253 is executed again, the position and orientation of the virtual camera are adjusted, for example, by moving the virtual camera in a predetermined direction by a predetermined distance. In addition, if an optimization algorithm such as NSGA-II is used, the adjustment of the position and orientation of the virtual camera may be determined using the optimization algorithm.

[0059] If it is determined in step S263 that the termination condition has been met (step S263; YES), then step S265 is executed.

[0060] Figure 10 is an explanatory diagram showing an example of a results screen displayed to present a Pareto solution. As shown in Figure 9, in step S265, one solution is selected from the set of Pareto solutions obtained by the optimization process. For example, a results screen SC4 including an image representing the Pareto solution, as shown in Figure 10, may be displayed on the display device 740 so that the user can select one solution. The results screen SC4 shows a graph with either evaluation value A or evaluation value B on the horizontal axis and the other on the vertical axis, plotting all solutions obtained by the optimization process in addition to the Pareto solution. In the example shown in Figure 10, the plotted points of the solutions are hatched so that the user can recognize them as Pareto solutions. On the results screen SC4 shown in Figure 10, the user can select any Pareto solution using the input device 730.

[0061] In step S267, the two-dimensional image associated with evaluation values ​​A and B represented by the selected solution 1 is output as a snapshot SD. This completes the process for generating a snapshot. In this embodiment, the position and orientation of the virtual camera can be easily determined by using the optimization process.

[0062] C. Other embodiments: (C1) In the first embodiment, it was determined that imaging was possible when all of the following conditions were met (see step S255 in Figure 8). (1) The virtual camera's field of view includes at least the robot. (2) The straight-line distance between the virtual camera and the selected object is within a predetermined range.

[0063] The conditions for determining whether photography is possible are not limited to (1) and (2) above. For example, the following conditions can be adopted. (3) The number of objects included in the shooting range is L or more (where L is an integer) and M or less (where M is an integer greater than L).

[0064] When multiple objects are placed in the virtual workspace, a snapshot SD that reproduces the layout of the objects in the virtual workspace as accurately as possible can be obtained by including as many objects as possible within the shooting range of the virtual camera. Alternatively, it may be determined that shooting is possible if any one of the above conditions (1) to (3) is met.

[0065] (C2) The information processing device 700 may, after generating a two-dimensional image obtained by a virtual camera capturing the state in which objects are placed in the virtual workspace, determine whether the two-dimensional image can be adopted as a snapshot SD. For example, it is determined that shooting is possible if at least the robot is included in the shooting range of the virtual camera. A two-dimensional image is generated by the virtual camera capturing the state in which objects are placed in the virtual workspace at the position and orientation of the virtual camera when it is determined that shooting is possible. Subsequently, it is determined whether the generated two-dimensional image can be adopted as a snapshot SD. For example, it may be determined that the generated two-dimensional image can be adopted as a snapshot SD if the ratio of the area occupied by the object in the two-dimensional image to the entire image is within a predetermined range.

[0066] (C3) In the first embodiment, an example was described in which, when the virtual camera is initially positioned, the virtual camera is positioned and oriented in the virtual workspace so that its lens faces the center of gravity, which is the center position between multiple objects placed in the virtual workspace (see step S251 in Figure 8). Alternatively, when the virtual camera is initially positioned, the position of the virtual camera may be determined so that the selected object is within the shooting range of the virtual camera. Specifically, one object is selected from among multiple objects placed in the virtual workspace. The virtual camera is positioned so as to be able to photograph the selected object from a predetermined direction, and at a predetermined distance from the selected object. The selected object may be a robot or an object specified by the user.

[0067] (C4) In the first embodiment, an example was described in which the position of the virtual camera is changed by adding a constant value to the X, Y, and Z coordinates of the virtual camera. Alternatively, the information processing device 700 may change the position and orientation of the virtual camera in response to user instructions. For example, a virtual camera is displayed on a simulator screen as shown in Figure 6. In response to user instructions via the input device 730, the information processing device 700 may change the position and orientation of the virtual camera on the simulator screen.

[0068] (C5) In the second embodiment, an example was described in which a multi-objective optimization algorithm is used as the optimization method. Alternatively, a single-objective optimization algorithm may be used as the optimization method. For example, an evaluation value A, which is the number of objects contained in a 2D image that is a candidate for snapshot SD, and an evaluation value B, which is the ratio of the area occupied by objects in the image to the entire image, are optimized. When using a single-objective optimization algorithm, the acceptable range of one evaluation value can be set as a constraint, and the other evaluation value can be optimized. As a single-objective optimization algorithm, random search, grid search, CMA-ES (Covariance Matrix Adaptation Evolution Strategy), particle swarm optimization (PSO), etc. can be used.

[0069] (C6) In the second embodiment, an example was described in which the value to be optimized is the number of objects included in the two-dimensional image obtained in step S257b, and the ratio of the area occupied by the objects to the entire image. Alternatively, the value to be optimized may be the value obtained by multiplying the number of objects of a predetermined type included in the two-dimensional image obtained in step S257b by the priority set for the objects. In this case, a single-objective optimization algorithm can be used. Prior to executing the optimization process, the user needs to specify the type of object and set the priority of the object.

[0070] (C7) In the second embodiment, an example was described in which the information processing device 700 displays a result screen SC4 (see Figure 10) representing the Pareto solution and accepts the user's selection of a solution. Alternatively, the information processing device 700 may randomly select a solution from the Pareto solutions. Furthermore, according to the user's specifications, the setting may be switched between a setting in which the user selects a solution and a setting in which the information processing device 700 randomly selects a solution.

[0071] (C8) In the first embodiment, an example was described in which the robot system 10 includes a controller 500 (see Figure 1), but the robot system 10 does not have to include a controller 500. In this case, the information processing device 700 may function as the controller of the robot system 10. Furthermore, the present disclosure may be implemented with the information processing device 700 alone.

[0072] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

[0073] D. Other forms: (1) According to the first embodiment of the present disclosure, a method for creating simulation data is provided. This method for creating simulation data includes: (a) operating a virtual robot placed in a virtual workspace, which is a virtual three-dimensional workspace, based on user operation instructions, and displaying a simulator image representing the operation of the virtual robot in the virtual workspace on a display device; (b) outputting a snapshot representing a two-dimensional image obtained by capturing a state in which a virtual three-dimensional object, including the virtual robot, is placed in the virtual workspace with a virtual camera; and (c) storing in memory as simulation data an operation program representing the target operation of the virtual robot based on the operation instructions, layout data representing the position and orientation of the object in the virtual workspace, attribute data representing the attributes of the object, and the snapshot. For example, a simulation may be run on a different computer than the one on which the simulation was originally performed, using saved simulation data. Building a simulation environment requires processes such as reading shape data for each object and reproducing the shape of each object. The more objects there are in the virtual workspace, the higher the processing load becomes. In the above configuration, the simulation data includes snapshots, which are 2D images taken by a virtual camera of the virtual robot placed in the virtual workspace. Therefore, before performing processes such as reading shape data for each object and reproducing the shape of each object, the user can be presented with a snapshot as a preview image. This allows the user to confirm that the contents of the selected folder are the desired simulation data. Thus, the occurrence of errors such as executing the process of building a simulation environment using incorrect simulation data can be suppressed. (2) In the above-described method for creating simulation data, in step (b), the virtual camera may be positioned so that at least the virtual robot is included in the shooting range of the virtual camera. (3) In the method for creating simulation data in the above form, step (b) may include (b1) changing the position and orientation of the virtual camera so that, if a plurality of objects are arranged in the virtual workspace, the shooting range includes a number of objects equal to or greater than a predetermined threshold; (b2) repeating step (b1) until a condition for termination of the repetition is met; and (b3) outputting the two-dimensional image obtained by the virtual camera capturing the state in which the objects are arranged in the virtual workspace at the position and orientation of the virtual camera when the termination condition is met, as the snapshot. With the above configuration, if there are many objects placed in the virtual workspace, it becomes difficult to include all objects in the virtual camera's field of view. However, by including as many objects as possible in the virtual camera's field of view, it is possible to obtain a snapshot that reproduces the layout of the objects in the virtual workspace as accurately as possible. (4) In the method for creating simulation data in the above form, step (b) may include (b4) a step performed before step (b1) in which, when a plurality of the objects are arranged in the virtual workspace, the virtual camera is positioned at a position and orientation facing the center of gravity of the collection of objects determined using the layout data. According to the above configuration, by pointing a virtual camera at the centroid of the collection of objects, it becomes more likely that as many objects as possible will be included in the virtual camera's field of view. (5) In the method for creating simulation data in the above form, if in step (b1) a plurality of objects are arranged in the virtual workspace and the straight-line distance between the virtual camera and one of the plurality of objects is greater than or equal to a predetermined threshold, the position and orientation of the virtual camera may be changed so that the straight-line distance is reduced by a predetermined distance. (6) In the method for creating simulation data in the above form, if a plurality of objects are arranged in the virtual workspace in step (b1), the position and orientation of the virtual camera may be determined using an optimization process, and in the optimization process, the number of the plurality of objects included in the shooting range of the virtual camera and the sum of the areas occupied by the objects within the shooting range of the virtual camera may be optimized. According to the above configuration, the position and orientation of the virtual camera can be easily determined. (7) In the method for creating simulation data in the above form, the shape of the object is represented by a polygon mesh, and the attribute data representing the attributes of the object may include information defining the polygon mesh. (8) In the method for creating simulation data in the above form, when an instruction to save the simulation data is received, steps (b) and (c) may be executed. (9) A second embodiment of the present disclosure provides a method for displaying a preview image using simulation data. The simulation data includes (i) an action program consisting of a plurality of commands for causing a robot to perform a target action; (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace; (iii) attribute data representing the attributes of the objects; and (iv) a snapshot which is a two-dimensional image representing the state of the virtual workspace. The method for displaying this preview image includes: (a) accepting the selection of the simulation data; (b) reading the snapshot included in the selected simulation data from memory; (c) displaying the snapshot as a preview image on a display device in a manner that allows the selected simulation data to be identified; (d) when an instruction to construct a simulator environment is received, reading the layout data and attribute data included in the selected simulation data from memory; and (f) using the read layout data and attribute data to construct the simulator environment in which the objects are placed in the virtual workspace, and displaying a simulator image on the display device that represents the operation of the virtual robot placed in the virtual workspace. (10) According to a third embodiment of the present disclosure, a simulation device is provided that performs processing using simulation data. The simulation data includes (i) an action program consisting of a plurality of commands that cause a robot to perform a target action; (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace; (iii) attribute data representing the attributes of the objects; and (iv) snapshots which are two-dimensional images representing the state of the virtual workspace. The simulation device includes a simulator unit which constructs a simulator environment in which the objects are placed in the virtual workspace using the layout data and attribute data, operates the virtual robot placed in the virtual workspace in the simulator environment based on user operation instructions, and displays a simulator image representing the operation of the virtual robot placed in the virtual workspace on a display device; a snapshot output unit which outputs a snapshot which is a two-dimensional image obtained by capturing the state in which the objects, including the virtual robot, are placed in the virtual workspace with a virtual camera; and a data storage unit which stores simulation data, including an operation program representing the target operation of the virtual robot based on the operation instructions, the layout data, the attribute data, and the snapshot, in memory. When the simulator unit receives a selection of desired simulation data from the user, it reads the snapshot included in the selected simulation data from the memory and displays the snapshot as a preview image on the display device in a manner that allows the selected simulation data to be identified. When the simulator unit receives an instruction from the user to build the simulator environment, it reads the layout data and attribute data included in the selected simulation data from the memory, builds the simulator environment using the read layout data and attribute data, and displays the simulator image on the display device. [Explanation of symbols]

[0074] 10...Robot system, 100...Robot, 105...Base, 120...Arm, 140...Force sensor, 300...Display device, 500...Controller, 700...Information processing device, 710...Memory, 720...Interface circuit, 730...Input device, 740...Display device, 750...CPU, 751...Simulator, 752...Snapshot output unit, 753...Data storage unit, EE...End effector, F1, F2, F3, F4...Display area, J1...Joint, LD...Layout data, OD...Object model data, RC...Robot coordinate system, RD...Robot model data, RP...Motion program, SC1...Selection screen, SC2...Confirmation screen, SC3...Simulator screen, SC4...Result screen, SD...Snapshot, SF...Simulation file group, SP...Simulation program, UD...User-defined model data

Claims

1. A method for creating simulation data, (a) A step of operating a virtual robot placed in a virtual workspace, which is a virtual three-dimensional workspace, based on user operation instructions, and displaying a simulator image representing the operation of the virtual robot in the virtual workspace on a display device, (b) A step of outputting a snapshot representing a two-dimensional image obtained by capturing a virtual camera of a virtual three-dimensional object, including the virtual robot, placed in the virtual workspace, (c) A step of saving in memory as simulation data an operation program representing the target movement of the virtual robot based on the operation instructions, layout data representing the position and orientation of the object in the virtual workspace, attribute data representing the attributes of the object, and the snapshot. A method for creating simulation data, including [specific details].

2. A method for creating simulation data according to claim 1, In step (b) above, the virtual camera is positioned so that at least the virtual robot is included in the shooting range of the virtual camera. How to create simulation data.

3. A method for creating simulation data according to claim 2, The above step (b) is, (b1) If multiple objects are arranged in the virtual workspace, the position and orientation of the virtual camera are changed so that the shooting range includes a number of objects equal to or greater than a predetermined threshold, (b2) A step of repeating step (b1) until the termination condition for the repetition is met, (b3) The steps of outputting the two-dimensional image obtained by the virtual camera capturing the state in which the object is placed in the virtual workspace at the position and orientation of the virtual camera when the termination condition is met, as the snapshot, including, How to create simulation data.

4. A method for creating simulation data according to claim 3, The above step (b) is, (b4) A step performed before step (b1), wherein, when a plurality of the objects are arranged in the virtual workspace, the virtual camera is positioned and oriented so as to face the center of gravity of the collection of objects determined using the layout data. A method for creating simulation data, including [specific details].

5. A method for creating simulation data according to claim 4, In step (b1), if a plurality of objects are arranged in the virtual workspace and the straight-line distance between the virtual camera and one of the plurality of objects is greater than or equal to a predetermined threshold, the position and orientation of the virtual camera are changed so that the straight-line distance is reduced by a predetermined distance. How to create simulation data.

6. A method for creating simulation data according to claim 4, In step (b1) above, if multiple objects are placed in the virtual workspace, the position and orientation of the virtual camera are determined using an optimization process. In the optimization process described above, the number of objects included in the shooting range of the virtual camera and the sum of the areas occupied by the objects within the shooting range of the virtual camera are optimized. How to create simulation data.

7. A method for creating simulation data according to any one of claims 1 to 6, The shape of the aforementioned object is represented by a polygon mesh, The attribute data representing the attributes of the object includes information defining the polygon mesh. How to create simulation data.

8. A method for creating simulation data according to claim 7, When an instruction is received to save the simulation data, steps (b) and (c) are executed. How to create simulation data.

9. A method for displaying a preview image using simulation data, The simulation data includes (i) an action program consisting of multiple commands that cause the robot to perform a target action, (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace, (iii) attribute data representing the attributes of the objects, and (iv) a snapshot which is a two-dimensional image representing the state of the virtual workspace. The aforementioned display method is, (a) A step of accepting the selection of the simulation data, (b) A step of reading the snapshot included in the selected simulation data from memory, (c) A step of displaying the snapshot as a preview image on a display device in a manner that allows the selected simulation data to be identified, (d) When an instruction to build a simulator environment is received, the steps include reading the layout data and attribute data included in the selected simulation data from the memory, (f) Using the read layout data and attribute data, construct the simulator environment in which the objects are placed in the virtual workspace, and display a simulator image representing the operation of the virtual robot placed in the virtual workspace on the display device; How to display preview images, including [specific details].

10. A simulation device that performs processing using simulation data, The simulation data includes (i) an action program consisting of multiple commands that cause the robot to perform a target action, (ii) layout data representing the position and orientation of three-dimensional objects, including a virtual robot, placed in a virtual workspace which is a virtual three-dimensional workspace, (iii) attribute data representing the attributes of the objects, and (iv) a snapshot which is a two-dimensional image representing the state of the virtual workspace. The aforementioned simulation device is It is the simulator section, Using the layout data and attribute data, a simulator environment is constructed in which the objects are placed in the virtual workspace. In the simulator environment, the virtual robot placed in the virtual workspace is operated based on user instructions, and a simulator image representing the operation of the virtual robot placed in the virtual workspace is displayed on a display device. The simulator section, A snapshot output unit outputs a snapshot which is a two-dimensional image obtained by capturing the state in which the objects, including the virtual robot, are placed in the virtual workspace using a virtual camera. A data storage unit that stores simulation data in memory, including an operation program representing the target movement of the virtual robot based on the operation instructions, the layout data, the attribute data, and the snapshot. Equipped with, The aforementioned simulator unit is When the user selects the desired simulation data, The selected snapshot included in the simulation data is read from the memory, The snapshot is displayed on the display device as a preview image in a manner that allows the selected simulation data to be identified. When the user provides instructions for building the aforementioned simulator environment, The layout data and attribute data included in the selected simulation data are read from the memory. Using the read layout data and attribute data, the simulator environment is constructed, and the simulator image is displayed on the display device. Simulation device.

Images