Information processing device, information processing method, robot system, method of manufacturing an article using the robot system, program and recording medium

Abstract

Information processing device (500) that performs a simulation by using a virtual model (200V, 300V), the information processing device (500) comprising a processing unit (511), the processing unit (511) being configured displaying a virtual model (200V, 300V) and an operating section (700) on a display unit (514), the operating section (700) being used by a user to operate the virtual model (200V, 300V), and, when the virtual model (200V, 300V) is in a certain state as a result of a user input, changing a display of the operation section (700) to indicate that the virtual model (200V, 300V) is in the certain state.

Description

BACKGROUNDArea of Revelation

[0001] The present disclosure relates to information processing. Description of the related art

[0002] Until now, in adjustment work (also called teaching work) for robot operation, a robot simulator is used, which is capable of verifying the operation of a robot in advance without using a real machine of the robot. Examples of adjustment work include work to set a tool center point (hereinafter TCP), which is a representative point of an end effector of the robot, and generate a noise point, which represents the coordinates of a movement target of the TCP, and then generate a program to cause the robot to operate using the TCP and the teaching point. In such a robot simulator, models of a robot and environments are arranged in a virtual space, and simulations are performed by, for example, moving the model (TCP) of the robot and the models of the environments to teach the operation of the robot.There is a function that uses an operation adjustment lever (an operation section) as a device for a user to move the robot model and the models of the surroundings. Japanese Patent Laid-Open No. JP 2014-161921 A describes a function for displaying operation adjustment levers for each axes of the robot and a position of the TCP.

[0003] Document EP 3 363 604 A2 describes a robot simulator comprising a model storage module that stores model information related to a robot and an obstacle in the vicinity of the robot, an information acquisition module configured to receive first input information defining a start position and an end position of the operation of the robot, and an information processing module configured to generate a path for moving the distal end part of the robot from the start position to the end position while avoiding a collision between the robot and the obstacle on the basis of the first input information and the model information, and generate image data including an image of the obstacle and an index indicating a passing point of the path.

[0004] EP 3 136 350 A1 describes an extensible and manipulable modeling of equipment that provides a three-dimensional, manipulable base model image of an equipment item on a display system using optical base model data acquired with the aid of a computer system. The acquisition of the base model data can be controlled by the user, allowing spatial aspects of the base model image (size, perspective, orientation) to be modified. User inputs regarding functions, conditions, and the like can be transmitted from a model control unit to the computer system via a bidirectional communication link, thereby generating extension data that is combined with the base model data to create extended three-dimensional models.User inputs can also be received directly from the computer system, for example, via a touchscreen of the display system. The extension data received from the computer system and / or the display system can also be transmitted to the model control unit via the communication link.

[0005] EP 3 093 108 B1 describes a display device including a virtual environment screen displaying the state of an identified robot and a parameter setting screen numerically displaying the position and orientation data. When a change is made to a portion of the position and orientation data by the operation input unit, the portion of the position and orientation data is changed according to the content of the operation and input. Position and orientation are calculated to identify the position or orientation of each part of the robot based on the changed portion of the position and orientation data, and new position and orientation data are calculated based on the position and orientation calculation.The content of the virtual display on the virtual environment screen or the numeric value display on the parameter setting screen of the display device is updated based on the changed part of the position and orientation data and the new position and orientation data.

[0006] The document US 2021 / 0 154 845 A1 describes a teaching device comprising a display unit having a simulation area in which a viewpoint for a virtual robot as a simulation model of a robot is changeably displayed, and an operation area in which a plurality of operation symbols for moving a control point of the virtual robot by changing a posture of the virtual robot are displayed, and a display control unit that controls the operation of the display unit, wherein the display control unit changes the directions of the respective operation symbols in the operation area to interlock with a change in the viewpoint for the virtual robot in the simulation area.

[0007] Document US 2020 / 0 290 204 A1 describes a control device comprising: a storage unit that stores a work program of a robot; a display control unit that displays a virtual robot formed by virtualizing the robot and a teaching point on a simulator screen on a display unit based on the work program stored in the storage unit; and an acceptance unit that accepts a selection of the teaching point displayed on the simulator screen. The display control unit displays a first window containing a first command corresponding to the selected teaching point on the simulator screen when the acceptance unit accepts the selection of the teaching point.

[0008] US 11 027 428 B2 describes a simulation device that enables a virtual robot arm displayed on a display device to act, comprising a processor configured to execute computer-executable instructions to control a robot, the processor being configured to receive a pulling operation at a distal end of the virtual robot arm from an input device and to change a posture of the virtual robot arm based on the pulling operation. SUMMARY

[0009] It is an object of the invention to provide an improved information processing apparatus, an improved robot system, an improved manufacturing method, an improved information processing method, a corresponding program and a corresponding computer-readable recording medium.

[0010] This object is achieved by an information processing device according to claim 1 or 28, a robot system according to claim 25, a manufacturing method according to claim 26, an information processing method according to claim 27 or 30, a program according to claim 31, and a computer-readable recording medium according to claim 32. Advantageous further developments are specified in the dependent claims.

[0011] According to embodiments of the present disclosure, an information processing device performs a simulation using a virtual model. The information processing device includes a processing unit. The processing unit is configured to display the virtual model and an operation section on a display unit, wherein the operation section is used by a user to operate the virtual model and to change setting information in the operation section in response to user input.

[0012] Further features of the present disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWING Fig. 1 is a diagram illustrating a robot system according to a first embodiment. Fig. 2 is a diagram illustrating a robot arm body according to the first embodiment. Fig. 3 is a diagram illustrating an information processing apparatus according to the first embodiment. Fig. 4 shows a control block diagram of the information processing apparatus according to the first embodiment. Fig. 5 is a diagram showing an example of a simulation screen according to the first embodiment. Fig. 6A to 6C show diagrams regarding a problem in a display of an operation setting lever. Fig. 7 shows a control flowchart according to the first embodiment. Fig. 8A and Fig. 8B are diagrams showing an example of an operation setting lever setting screen according to the first embodiment. Fig. 9A to 9F are diagrams related to a display of the operation setting lever according to the first embodiment. Fig. 10 shows a control flowchart according to a second embodiment. Fig. 11 is a diagram showing an example of the operation setting lever setting screen according to the second embodiment. Fig. 12A to 12C are diagrams related to a display of the operation setting lever according to the second embodiment. Fig. 13 shows a control flowchart according to a third embodiment. Fig. 14 is a diagram showing an example of the operation setting lever setting screen according to the third embodiment. Fig. 15 is a diagram showing an example of the simulation screen according to the third embodiment. Fig. 16 shows a control flowchart according to a fourth embodiment. Fig. 17A to 17C are diagrams related to a display of the operation setting lever according to the fourth embodiment. Fig. 18 shows a control flowchart according to a fifth embodiment. Fig. 19A and Fig. 19B show diagrams related to a display of the operation setting lever according to the fifth embodiment. Fig. 20A to 20C show diagrams related to a singular point of a virtual robot arm body. Fig. 21 shows a control flowchart according to a sixth embodiment. Fig. 22A and Fig. 22B show diagrams related to a display of the operation setting lever according to the sixth embodiment. Fig. 23A and Fig. 23B show diagrams related to a display of the operation setting lever according to the sixth embodiment. Fig. 24A to 24D are diagrams related to a display of the operation setting lever according to the sixth embodiment. Fig. 25 shows a control flowchart according to a seventh embodiment. Fig. 26A to 26C are diagrams showing an example of the simulation screen according to the seventh embodiment. Fig. 27 is a diagram related to a display of the operation setting lever according to the seventh embodiment. Fig. 28 is a diagram illustrating the robot system according to an eighth embodiment. Fig. 29 is a diagram showing an example of the simulation screen according to the eighth embodiment. Fig. 30 is a diagram illustrating the robot system according to a ninth embodiment. Fig. 31 shows a control block diagram of the information processing apparatus according to the ninth embodiment. Fig. 32 is a diagram showing an example of the simulation screen according to the ninth embodiment. Fig. 33 is a diagram showing an example of the simulation screen according to the ninth embodiment. Fig. 34 shows a control flowchart according to the ninth embodiment. DESCRIPTION OF THE EMBODIMENTS

[0013] In Japanese Patent Laid-Open No. JP 2014 - 161921 A, there is a problem that depending on the display state of the operation setting lever, for example, the models of the environments overlap the display position of the operation section, the models of the environments make it difficult to visually check or access the operation setting lever, and as a result, make it difficult to operate the operation section.

[0014] Embodiments of the present disclosure will be described below with reference to the accompanying drawings. The embodiments described below are for illustrative purposes only, and for example, the configuration of detailed parts may be modified as needed by a person skilled in the art without departing from the scope of the present disclosure. Numerical values described in the present embodiments are numerical reference values and do not limit the present disclosure. In the drawings below, the arrows X, Y, Z in the figures represent the general coordinate system of a robot system. In general, a three-dimensional XYZ coordinate system represents a world coordinate system of a general installation environment.In addition, to simplify control, a local coordinate system may be used as needed for a robot hand, finger, joint, or the like. In the present embodiments, the world coordinate system, which is the general coordinate system, is represented by XYZ, and the local coordinate system is represented by xyz. First embodiment

[0015] Fig. 1 is a diagram illustrating the schematic configuration of a robot system 1000 according to the present embodiment.

[0016] Fig. Figure 1 schematically shows the robot system 1000 in a real space RS. The robot system 1000 includes a robot arm body 200, a robot hand body 300, a controller 400, and an information processing device 500. In the present embodiment, the controller 400 and the information processing device 500 are each formed from different computers. Alternatively, the controller 400 and the information processing device 500 may be formed from a single computer.

[0017] The robot arm body 200 is a vertically articulated robot arm that includes a base, a plurality of links, a drive source for driving the plurality of links, and a gear mechanism that, for example, reduces a speed when power is supplied from the drive source to actuate the links. The robot hand body 300 is supported by the robot arm body 200. The robot hand body 300 is attached to a predetermined region of the robot arm body 200, for example, to a distal end portion of the robot arm body 200.

[0018] The robot hand body 300 can be positioned at a selected position in an XYZ space by the robot arm body 200.

[0019] The robot hand body 300 includes fingers capable of holding a workpiece W, a drive source for operating the fingers, and a gear mechanism that, for example, reduces a speed when power is supplied from the drive source to operate the fingers. For example, the workpiece W, which is an object to be conveyed, is placed around the robot hand body 300, whereby it is possible to perform an operation for holding the workpiece W and, for example, assembling the workpiece W with another workpiece through the robot arm body 200 and the robot hand body 300. In the present embodiment, the robot arm body 200 may be referred to as a robot, or the robot arm body 200 and the robot hand body 300 may be collectively referred to as a robot.

[0020] The robot arm body 200 and the robot hand body 300 are communicatively connected to the controller 400 via wired lines. The controller 400 and the information processing device 500 are communicatively connected to each other via wired lines. In the present embodiment, the controller 400 and the information processing device 500 are connected via wired communication. Alternatively, the controller 400 and the information processing device 500 may be connected via wireless communication.

[0021] The information processing device 500 virtually executes and displays the operations of the robot arm body 200 and the robot hand body 300 at the time when the workpiece W is held through offline teaching, that is, computer simulation. The controller 400 acquires information about a holding position from the information processing device 500 and generates trajectory data of the robot arm body 200 from the holding position to a position that is a target of conveying the workpiece W. The controller 400 controls the robot arm body 200 and the robot hand body 300 according to the generated trajectory data to perform an operation for conveying the workpiece W. In the present embodiment, the robot arm body 200 and the robot hand body 300 perform an operation for conveying the workpiece W that is held and for assembling the workpiece W with another workpiece.It is thus possible to manufacture industrial products or articles. Calculation of trajectory data can be performed by the information processing device 500.

[0022] When the robot arm body 200 and the robot hand body 300 transport a workpiece W, the robot arm body 200 and the robot hand body 300 must be taught not to contact objects around them. Teaching the robot involves setting teaching points to obtain the trajectory data of the robot arm body 200 and / or the robot hand body 300.

[0023] Fig. Figure 2 shows a diagram illustrating the configurations of the robot arm body 200 and the robot hand body 300 according to the present embodiment. The robot arm body 200 includes a base 210 and a plurality of links 201, 202, 203, 204, 205 coupled by a plurality of rotationally driven joints J1, J2, J3, J4, J5, J6. The link 201 is coupled to the base 210.

[0024] Each of the joints of the robot arm body 200 includes a motor serving as a drive source that drives a corresponding one of the links, a reduction gear, and an encoder serving as a position detector that detects the rotation angle of the motor. The installation position and output method of the encoder are irrelevant. The robot hand body 300 is attached to the link 205, which is the distal end portion of the robot arm body 200. The robot hand body 300 is capable of rotating with the joint J6. By driving the joints J1 to J6 of the robot arm body 200, the robot arm body 200 can be adjusted to various postures.

[0025] The robot hand body 300 includes a palm 303 and a plurality of fingers, for example, two fingers 301, 302, held by the palm 303 to be openable and closable. The two fingers 301, 302 are arranged to oppose each other. The robot hand body 300 has a force control function that operates the fingers 301, 302 with a constant force. The palm 303 of the robot hand body 300 holds the fingers 301, 302 and includes a drive unit (not shown) that linearly operates the pair of fingers 301, 302. The drive unit includes a motor, a conversion mechanism that converts the rotational motion of the motor into a linear motion, and the like. When the drive unit is operated, the fingers 301, 302 can be moved in opening directions D11, D12 and closing directions D21, D22, which are indicated by arrows in Fig. 2. The drive unit is capable of causing the fingers 301, 302 to generate a holding force for holding a workpiece W by generating a driving force. The drive unit only needs to cause the fingers 301, 302 to generate a holding force so that the workpiece W is not displaced with respect to the robot arm body 200.

[0026] The number of fingers is two in the present embodiment; however, the number of fingers can be changed as needed by a person skilled in the art. In the present embodiment, the robot hand body 300 actuates the fingers by a motor drive. Alternatively, the robot hand body 300 may be a pneumatically driven air gripper or may be configured to hold by absorption or adsorption. The end effector may not be in the form of a robot hand but may be a tool that performs cutting, grinding, or the like on a workpiece. Likewise, the end effector may be a screw tool that performs screw fastening or the like.

[0027] Fig. 3 is a diagram illustrating the information processing device 500 according to the present embodiment. The information processing device 500 includes a device main body 501, a display 502, which is an example of a display device connected to the device main body 501, and a keyboard 503 and a mouse 504, which are examples of an input device connected to the device main body 501. The device main body 501 uses an operating system (OS) in a general personal computer (PC).

[0028] The display 502 displays a simulation screen 600 when the device main body 501 executes application software for implementing a simulation process, which is a teaching process. A virtual space VS constructed by the device main body 501 is displayed on the simulation screen 600. A virtual robot arm body 200V, a virtual robot hand body 300V, a virtual workpiece WV, and the like are arranged in the virtual space VS. These are displayed on the display 502 as 2D images or 3D images. An operation for inputting, editing, or changing various pieces of information of the simulator is configured to be performed through the input device, that is, the keyboard 503, the mouse 504, and the like.

[0029] The simulation screen 600 displayed on the display 502 is a screen for the user to edit teaching and program and check operations and faults of the robot system 100. The display 502 may be configured such that a so-called touch panel is layered on the surface of the display 502. In this case, an input operation equivalent to that of the input device, i.e., the keyboard 503, the mouse 504, and the like, can be performed through the touch panel. In some cases, the input device may be omitted.

[0030] The simulation screen 600 according to Fig. 2 is configured to include at least one virtual space screen. The virtual space screen may be configured as a graphical user interface (GUI). In this case, objects constituting the simulation screen 600 (such as menus, input fields for numeric values and text, and a virtual representation of the robot) may be configured to be operated by a pointing device, such as the mouse 504 (or the touch pad described above).

[0031] For example, a case where the information processing device 500 is a desktop PC, which is a general-purpose computer, will be described below; however, the configuration is not limited to this. For example, the information processing device 500 may be a general-purpose computer such as a laptop PC, a tablet PC, and a smartphone, or it may be a teaching aid, or it may be a simulator-related computer. The information processing device 500 may also be integrated with the controller 400. In other words, the controller 400 may have the functions of the simulator.

[0032] The virtual robot arm body 200V is a robot model corresponding to the robot arm body 200. The virtual robot hand body 300V is a robot model corresponding to the robot hand body 300. The virtual workpiece WV is a workpiece model corresponding to the workpiece W. Three-dimensional data of each model is input in advance into the device main body 501 as CAD data, for example.

[0033] When a worker inputs data into the device main body 501 by operating the keyboard 503 and the mouse 504, the worker is able to cause the device main body 501 to simulate the operations of the robot arm body 200 and the robot hand body 300 in the virtual space VS.

[0034] In the present embodiment, the operations of the robot arm body 200 and the robot hand body 300 are taught to the workpiece W through offline teaching. Determining the operations of the robot arm body 200 and the robot hand body 300 is determining the rotation amounts of the joints J1 to J6 and the movement amounts of the fingers 301, 302. However, if the robot hand body 300 itself has a joint and is capable of changing the positions of the fingers 301, 302 in a rotation direction, the rotation amount of the joint of the robot hand body 300 is also determined. When the fingers 301, 302 are moved in a closing direction in an open state of the robot hand body 300 to be brought into contact with the workpiece W, and a holding force is applied by the fingers 301, 302, the workpiece W can be held.

[0035] Here, a holding position is a relative position of the robot hand body 300 to the workpiece W at the time when the workpiece W is held by the robot arm body 200 and the robot hand body 300. A holding posture corresponds to the posture of the robot arm body 200 at the time when the workpiece W is held by the robot hand body 300 in a state where the workpiece W is positioned with respect to the robot arm body 200. Thus, in a state where the workpiece W is positioned with respect to the robot arm body 200, the robot hand body 300 can hold the workpiece W at a predetermined position by setting the robot arm body 200 to a predetermined posture.

[0036] Fig. 4 shows a control block diagram showing a control system of the information processing apparatus 500. As shown in Fig. As shown in Fig. 4, the device main body 501 of the information processing device 500 includes a central processing unit (CPU) 511 as hardware. The device main body 501 further includes a storage device 512 formed of a read-only memory (ROM) 512a, a random access memory (RAM) 512b, a hard disk drive (HDD) 512c, and the like.

[0037] The device main body 501 further includes an interface 513a for communicating and connecting with the input device, i.e., the keyboard 503, the mouse 504, and the like, and an interface 513b for communicating and connecting with the display 502. The device main body 501 includes an interface 514 for communication processing with the controller 400. The device main body 501 includes an interface 515 for transmitting and receiving data in the form of, for example, a file 530 to and from the controller 400 or an external device, such as another simulator device and the robot. These interfaces are each formed of, for example, a serial bus, a parallel bus, a network interface, or the like.

[0038] The ROM 512a is a non-transitory storage device. A basic program read by the CPU 511 when the computer starts up is stored in the ROM 512a. The RAM 512b is a temporary storage device used in arithmetic operation processing of the CPU 511. The HDD 512c is a non-transitory storage device that stores various data, such as arithmetic operation processing results of the CPU 511.

[0039] In the present embodiment, a program functioning as application software is stored in the HDD 512c. The CPU 511 functions as an information processing unit capable of simulating the behaviors of a virtual robot, a virtual hand, and a virtual workpiece in a virtual space (described below) by running the program.

[0040] In the present embodiment, a non-transitory computer-readable storage medium is the HDD 512c, and a program functioning as application software is recorded on the HDD 512c; however, the configuration is not limited thereto. The program can be recorded on any recording medium as long as the recording medium is a non-transitory computer-readable recording medium. Examples of the recording medium for supplying the program to a computer include a flexible floppy disk, an optical disk, a magneto-optical disk, a magnetic tape, and a non-volatile memory. A solid-state drive (SSD) can be used instead of the HDD.

[0041] The CPU 511 controls the entire system of the information processing device 500. Operation processing units of the CPU 511 include a display unit 516, an interpreting unit 517, and a calculating unit 518. The display unit 516 updates a display of the simulation screen 600 based on information saved in the storage device 512 and transmits a display command to a display device such as the display 502. The interpreting unit 517 controls an operation on the simulation screen 600 with the input device such as a mouse and a keyboard. The interpreting unit 517 interprets the details of an operation input, makes a request to the calculating unit 518 to perform a necessary calculation, and makes a request to the display unit 516 to update a display based on the calculated result.The calculation unit 518 executes calculation processing related to drawing models in the virtual space VS (described below) according to the details of an operation interpreted by the interpretation unit 517. The calculated result is saved in the storage device 512.

[0042] The storage device 512 stores display information of the models and the operation adjustment lever, which are components displayed on the simulation screen 600. The information stored in the storage device 512 is output in response to a request from the CPU 511 or updated in response to a request from the CPU 511. In response to a request from an external device or a specific operation on the keyboard 503 or the mouse 504, the CPU 511 is capable of transferring the information stored in the storage device 512 from the interface 515 in the form of the file 530. The CPU 511 is capable of reading the file 530 from an external source via the interface 515 as needed.

[0043] For example, during startup or restoration (recovery) processing of the information processing device 500, the CPU 511 reads the file 530 output in the past from an external device (an external storage device such as an SSD and a network attached storage (NAS)). Then, the CPU 511 can reproduce the previous storage state by updating the storage device 512. In the present embodiment, a storage area of the storage device 512 that stores the components is selectable. For example, a predetermined area in the RAM 512b or a storage area (corresponding to a predetermined file, for example) of the HDD 512c can be used. The configuration described above is an example of the overall configuration of the information processing device 500.

[0044] Fig. 5 shows an example of the simulation screen 600 according to the present embodiment. The virtual space VS and a menu bar 610 are displayed on the simulation screen 600. Virtual objects (virtual models) in the virtual space VS are defined by three-dimensional model data, such as CAD data, and are visualized and drawn as structures for convenience. An operation setting lever setting button 620 (a virtual button) for displaying a screen for setting a display of the operation setting lever (described below) is displayed on the menu bar 610.

[0045] Virtual objects defined in the virtual space VS, which is in Fig. 5 are described. Virtual objects are defined by three-dimensional model data including the robot arm body 200, the robot hand body 300 and the workpiece W shown in Fig. 1, in the virtual space VS. An absolute coordinate system world of the virtual space VS is shown. In the virtual space VS, the virtual workpiece WV, which consists of three-dimensional model data defining the workpiece W, is defined around the virtual robot arm body 200V. The CPU 511 performs teaching in an operation for holding the virtual workpiece WV with the virtual robot arm body 200V and the virtual robot hand body 300V. The virtual space VS, which is shown in Fig. 5 is shown, with a still image or a moving image on the simulation screen 600 of the display 502, which is shown in Fig. 2 is shown.

[0046] Fig. 6A to 6C show the operation setting lever 700 displayed in the virtual space VS according to the present embodiment.

[0047] Fig. 6A shows the operation setting lever 700 displayed in a case where the layout of the virtual robot arm body 200V, which is a target to be operated, is changed. Fig. 6B shows the operation setting lever 700 in a case where a tool center point (TCP) of the virtual robot arm body 200V, which is a target to be operated, is selected and teaching is performed. Fig. 6C shows the operation setting lever 700 displayed in a case where the layout of the virtual workpiece WV, which is a target to be operated, is changed.

[0048] Here, the operation setting lever is a function for setting the layout of a virtual model and for teaching the robot arm body 200V in the virtual space VS. The operation setting lever 700 in the Fig. 6A to 6C include arrows 700a extending in orthogonal three-axis directions and rings 700b, each used to rotate the model about a corresponding one of the arrows 700a. When any of the arrows 700a is dragged with the mouse 504, the target model can be translated following the operation. When any of the rings 700b is dragged with the mouse 504, the target model can be rotated about a corresponding one of the arrows 700a following the operation. With these operations, the layout of the virtual model is changed, the posture of the virtual robot arm body 200V is changed by a tool center point (TCP) operation (not shown), and teaching is performed.

[0049] In the condition according to Fig. 6A, the directions of the operation adjustment lever 700 coincide with those of the absolute coordinate system World, and they coincide with directions corresponding to the position of the origin of the virtual robot arm body 200V. When the operation adjustment lever 700 is operated in this state, the virtual robot arm body 200V and the virtual robot hand body 300V can be translated or rotated while the overall posture remains unchanged. In the state according to Fig. 6B, the directions of the operation adjustment lever 700 correspond to the directions of the TCP. When the operation adjustment lever 700 is operated in this state, the virtual robot hand body 300V can be displaced and rotated. Accordingly, the posture of the virtual robot arm body 200V changes to adjust the virtual robot hand body 300V to the position and posture corresponding to the operation. In the state according to Fig. 6C, the directions of the operation adjustment lever 700 are inclined by 60° in a clockwise direction on the sheet around the Z direction from the absolute coordinate system Welt, coinciding with the directions corresponding to the position of the origin of the model of the workpiece WV.

[0050] When the operation setting lever 700 is operated in this state, the workpiece WV can be moved and rotated.

[0051] As it is in the Fig. 6A, Fig. 6B and Fig. As shown in Fig. 6C, in the above-described display state of the operation setting lever 700, the operation setting lever 700 in the virtual model is buried in the display, and it may be difficult to operate the operation setting lever 700. In the present embodiment, this is addressed by performing an operation for changing setting information at the display position of the operation setting lever 700. The following describes in detail the operation for displaying the operation setting lever 700 in the present embodiment. In this way, in the specification, setting information in a display of the operation setting lever 700 may be referred to as a display condition.

[0052] Fig. Fig. 7 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 8A and Fig. 8B show an operation lever setting screen 800 according to the present embodiment. Fig. 9A to 9F are diagrams illustrating a display of the operation setting lever 700 whose display position is changed according to the present embodiment. It is assumed that the control flow in the present embodiment is executed by the CPU 511 of the information processing device 500. The control flowchart in Fig. 7 shows a control flow from the state where the operation setting lever 700 is displayed.

[0053] Initially, the setting of the operation setting lever 700 is changed in step S1. When the operation setting lever setting button 620 shown in Fig. 5, is pressed by the user, the operation setting lever setting screen 800 shown in the Fig. 8A and Fig. 8B is displayed. Only the operation setting lever setting screen 800 is displayed in the Fig. 8A and Fig. 8B. The operation lever setting screen 800 may be displayed in a superimposed manner in a dialog window (a second screen) on the simulation screen 600, or may be displayed in a display area different from the simulation screen 600 in the display 502. Fig. 8A shows the operation adjustment lever setting screen 800 in a state where a part coordinate system is selected. Fig. 8B shows the operation setting lever setting screen 800 in a state where a world coordinate system is selected.

[0054] In the Fig. 8A and Fig. 8B, a display position setting (a numerical value input) of the operation adjustment lever and a coordinate system selection (radio button or radio button), which is a position reference, are displayed on the operation adjustment lever setting screen 800. These are displayed such that the display position of the operation adjustment lever 700 displayed in the virtual space VS is aligned with a relative position from a reference position set in the Fig. 6A to 6C, is updated with respect to the directions of the selected coordinate system. An OK button 806 and a Clear button 807 are displayed. When the OK button 806 is pressed, a display of the operation setting lever 700 is updated according to the input details. When the Clear button 807 is pressed, the input details are not retained, and the operation setting lever setting screen 800 is closed.

[0055] Position input boxes 801 to 803 for inputting positions in X, Y, and Z directions, which are the directions of the coordinate system, are provided in the display position setting, and the position of the operation setting lever 700 is changed by inputting numerical values into the position input boxes 801 to 803. Input to the position input boxes 801 to 803 may be directly inputting numerical values displayed with the keyboard 503 and the mouse 504, or may be setting with up-down arrow keys of the position input boxes 801 to 803 with the mouse 504.

[0056] The coordinate system selection consists of radio buttons, where any of a part coordinate system button 804 and a world coordinate system button 805 can be selected. If the part coordinate system button 804 is selected, as shown in Fig. 8A, the operation setting lever 700 is moved according to the input numerical values with respect to the position of the operation setting lever 700 currently displayed with respect to the XYZ directions (arrows 708a) in the operation setting lever 700 shown in the Fig. 6A to 6C. Then, the display position of the operation setting lever 700 is updated. In Fig. 8A, 100 mm is entered into each of the position input boxes 801 to 803, with the part coordinate system key 804 selected in the coordinate system selection.

[0057] When the world coordinate system key 805 is pressed as in the case of Fig. 8B is selected, the operation setting lever 700 is moved based on the input numerical values with respect to the position of the operation setting lever 700 currently displayed with respect to the XYZ directions of the world coordinate system World, which is shown in the Fig. 6A to 6C. Then, the display position of the operation setting lever 700 is updated. In Fig. 8B, 100 mm is entered in each of the position input boxes 801 to 803, with the world coordinate system button 805 selected in the coordinate system selection.

[0058] Subsequently, in step S2, a display of the operation setting lever 700 is updated according to the values set in step S1. Fig. 9A to 9F are diagrams illustrating a display of the operation setting lever 700 whose display position is changed according to the present embodiment.

[0059] When the display position of the operation setting lever 700 is changed by selecting the part coordinate system key 804 as shown in Fig. 8A, the display position of the operation setting lever 700 is updated as shown in the Fig. 9A to 9C shown updated. In the Fig. 9A to 9C, the display position of the operation setting lever 700 with respect to the XYZ directions (arrows 708a) of the operation setting lever 700 is shown with respect to the display position of the operation setting lever 700 which is in a state according to each of the Fig. 6A to 6C is displayed, relatively moved. In Fig. 9A, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions (arrows 708a) of the operation setting lever 700 from the state shown in Fig. 6A moves. In Fig. 9B, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions (arrows 708a) of the operation setting lever 700 from the state shown in Fig. 6B moves. In Fig. 9C, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions (arrows 708a) of the operation setting lever 700 from the state shown in Fig. 6C moves.

[0060] When the display position of the operation setting lever 700 is changed by selecting the world coordinate system button 805 as shown in Fig. 8B, the display position of the operation setting lever 700 is updated as shown in the Fig. 9D to 9F shown updated. In the Fig. 9D to 9F, the display position of the operation setting lever 700 with respect to the XYZ directions of the world coordinate system World is compared with the display position of the operation setting lever 700 which is in a state according to each of the Fig. 6A to 6C is displayed, relatively moved. In Fig. 9D, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions of the world coordinate system from the state according to Fig. 6A moves. In Fig. 9E, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions of the world coordinate system from the state according to Fig. 6B moves. In Fig. 9F, the display position of the operation setting lever 700 is changed by 100 mm in the X direction, 100 mm in the Y direction and 100 mm in the Z direction with respect to the XYZ directions of the world coordinate system from the state according to Fig. 6C moves.

[0061] As described above, according to the present embodiment, as shown in the Fig. 6A to 6C, when the operation setting lever 700 overlaps the inside of the virtual model of the virtual space VS and consequently it is difficult to be operated, the display position of the operation setting lever 700 is moved (changed) as shown in the Fig. 9A to 9F. Thus, setting information related to a display of the operation setting lever 700 can be set by the user, making it possible to easily operate the operation setting lever 700. When the display position of the operation setting lever 700 is selectively changed by the user, the operation setting lever 700 can be displayed at an optimal location for the user, thus improving operability of the operation setting lever 700. Specifically, in Japanese Patent Laid-Open No. JP 2014-161921 A, setting information about the operation setting lever itself is already determined by the simulator, such as displaying a specific operation setting lever in one color with respect to another operation setting lever or displaying it in a different size.However, when setting information about the operation setting lever 700 itself is allowed to be changed by the user as in the case of the present disclosure, the operability of the operation setting lever 700 is improved.

[0062] In the present embodiment, the operation lever setting screen 800 is displayed when the operation lever setting key 620 is depressed; however, the configuration is not limited to this. The operation lever setting screen 800 may be displayed, for example, by a shortcut key such as a symbol and a function key of the keyboard 503 or a virtual keyboard. The operation lever setting screen 800 may be constantly displayed. In the operation lever setting screen 800, a coordinate system other than the part coordinate system or the world coordinate system may be configured to be selected. For example, a coordinate system selectively set by the user may be configured to be selected. Second embodiment

[0063] Next, a second embodiment of the present disclosure will be described in detail. When the size of the operation setting lever 700 is constant, operation of the operation setting lever 700 may be difficult depending on the situation. Therefore, in the present embodiment, a case where a display magnification is changed to change setting information in a display of the operation setting lever 700 will be described.

[0064] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those in the above-described embodiment, and a detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0065] Fig. 10 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 11 shows the operation lever setting screen 800 according to the present embodiment. Fig. 12A to 12C are diagrams illustrating a display of the operation setting lever 700 whose display magnification is changed according to the present embodiment.

[0066] Initially, the setting of the operation setting lever 700 is changed in step S11. When the operation setting lever setting button 620 is pressed by the user, the operation setting lever setting screen 800 shown in Fig. 11. Only the operation setting lever setting screen 800 is displayed in Fig. 11. The operation lever setting screen 800 may be displayed in a superimposed manner as a dialog window on the simulation screen 600 or may be displayed in a display area different from the simulation screen 600 in the display 502.

[0067] In the operation setting lever setting screen 800 according to the present embodiment, in addition to the elements described in the first embodiment, a slider 808, a bar 809, and a magnification input box 810 are displayed as a setting of the display magnification of the operation setting lever 700. A magnification indicated by the slider 808 in the bar 809 is synchronized with the magnification of the magnification input box 810, and when the magnification of one of them is changed, the magnification of the other is also changed. The display magnification of the operation setting lever 700 is changed by changing the numerical value of the slider 808 or the magnification input box 810.An input into the magnification input box 810 may be a direct input of a numerical value displayed with the keyboard 503 and the mouse 504 or may be an adjustment with an up-down arrow key of the magnification input box 810 with the mouse 504.

[0068] In the bar 809, only a magnification change from 1[%] to 100[%], that is, 1 / 100 times to 10 times, is performed; however, a numerical value less than or equal to 1[%] or a numerical value greater than or equal to 1000[%] is permitted by entering a selected numerical value into the magnification input box 810. When a magnification greater than or equal to the magnification indicated by the bar 809 is entered into the magnification input box 810, the slider 808 is positioned at one of the ends and does not move. For example, when a magnification less than or equal to 1[%] is entered into the magnification input box 810, the slider 808 is positioned at the left end of the bar 809 on the sheet and does not move.For example, if a magnification greater than or equal to 1000[%] is entered into the magnification input box 810, the slider 808 is positioned at the timely end of the bar 809 on the sheet and does not move. When inputting into the magnification input box 810, input may only be permitted for the range of magnification indicated by the bar 809. For example, if a magnification less than or equal to 1[%] is entered into the magnification input box 810, "1" is displayed in the magnification input box 810. For example, if a magnification greater than or equal to 1000[%] is entered, "1000" is displayed in the magnification input box 810. For example, with the up-down arrow key, a numeric value less than or equal to 1[%] or greater than or equal to 1000[%] is not displayed.

[0069] Subsequently, in step S12, a display of the operation setting lever 700 is updated based on the values set in step S11. Fig. 12A to 12C are diagrams illustrating a display of the operation setting lever 700 whose display magnification is changed according to the present embodiment.

[0070] Fig. 12A shows a state in which the display magnification of the operation setting lever 700 is 100% of the state shown in Fig. 9E, that is, the state according to Fig. 9E as it is. Fig. Fig. 12B shows a case where the display magnification is changed to 40[%] from the state shown in Fig. 9E and Fig. 12A is set and the display is updated. If Fig. 9E and Fig. 12A are compared with each other, the operation adjustment lever 700 does not overlap the virtual robot arm body 200V, so that the visibility of the virtual robot arm body 200V is improved. Fig. Fig. 12C shows a case where the display magnification is changed to 300[%] from the state shown in Fig. 9E and Fig. 12A is set and the display is updated. If Fig. 9E and Fig. 12A, the operation setting lever 700 overlaps the virtual robot arm body 200V, but the sufficiently large arrows 700a and rings 700b are displayed in the operation setting lever 700. Consequently, it is possible to enable the user to operate the operation setting lever 700 without difficulty.

[0071] As described above, according to the present embodiment, as shown in the Fig. 6A to 6C, when the operation setting lever 700 overlaps the inside of the virtual model of the virtual space VS and consequently it is difficult to operate it, the display magnification of the operation setting lever 700 can be changed as shown in Fig. 12C. Thus, setting information related to a display of the operation setting lever 700 can be set by the user, making it possible to make it easy to operate the operation setting lever 700. When the display magnification of the operation setting lever 700 is selectively changed by the user, the operation setting lever 700 can be displayed at an optimal size for the user, thus improving the operability of the operation setting lever 700. When the display position and the display magnification of the operation setting lever 700 are changed together, the operation setting lever 700 can be displayed to display the virtual model as shown in Fig. 12B, and improve the visibility of the virtual model. Allowing setting information about the operation setting lever 700 itself to be changed by the user improves the operability of the operation setting lever 700.

[0072] In the present embodiment, the operation lever setting screen 800 is displayed when the operation lever setting key 620 is depressed; however, the configuration is not limited thereto. The operation lever setting screen 800 may be displayed, for example, by a shortcut key such as a symbol and a function key of the keyboard 503. The operation lever setting screen 800 may be constantly displayed. In the operation lever setting screen 800, in the example according to Fig. 11, the display position and display magnification of the operation adjustment lever 700 are changed on the same screen. Alternatively, the display position and display magnification of the operation adjustment lever 700 may be changed on different screens. The various embodiments and modifications described above can be implemented in combination with the present embodiment and / or modification. Third embodiment

[0073] Next, a third embodiment of the present disclosure will be described in detail. Depending on the simulator operation, it is assumed that operating the operation setting lever 700 in the virtual space VS is difficult. Therefore, in the present embodiment, a case will be described where the operation setting lever 700 is displayed in a different area from the area where the virtual space VS is displayed in order to change setting information in a display of the operation setting lever 700.

[0074] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those in the various embodiments described above, and the detailed description is omitted. In the embodiment described below, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0075] Fig. 13 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 14 shows the operation adjustment lever setting screen 800 according to the present embodiment. Fig. 15 is a diagram illustrating a display of the operation setting lever 700 whose display range is changed according to the present embodiment.

[0076] Initially, the setting of the operation setting lever 700 is changed in step S21. When the operation setting lever setting button 620 is pressed by the user, the operation setting lever setting screen 800 shown in Fig. 14. Only the operation setting lever setting screen 800 is displayed in Fig. 14. The operation lever setting screen 800 may be displayed in a superimposed manner as a dialog window on the simulation screen 600 or may be displayed in a display area different from the simulation screen 600 in the display 502.

[0077] Fig. 14 is a diagram illustrating the operation lever setting screen 800 according to the present embodiment. In addition to elements described in the above-described various embodiments, radio buttons, i.e., a show button 811 and a hide button 812, are displayed as different area display settings. Fig. 14 the Show button 811 is selected.

[0078] Subsequently, in step S22, a display of the operation setting lever 700 is updated based on the values set in step S21.

[0079] Fig. 15 is a diagram illustrating a display of the operation setting lever 700 whose display range is changed according to the present embodiment.

[0080] As it is in Fig. 15, when display updating is performed in a state where the show button 811 is selected, the operation setting lever screen 690 is displayed in an area (screen) different from the area of the simulation screen 600 in which the virtual space VS is displayed. Fig. 15 shows a state in which, based on the setting values specified in Fig. 14, the display position and display magnification of the operation setting lever 700 are not changed, and an update is carried out such that an operation setting lever 700' in another area (another screen) is changed from the state shown in Fig. 6B. The operating adjustment lever 700', which is meshed with the operating adjustment lever 700, is displayed in the operating adjustment lever screen 690. A close button 691 is displayed in the upper right part of the operating adjustment lever screen 690 on the sheet. When the close button 691 is pressed, the operating adjustment lever screen 690 can be closed. Fig. 15, the operation adjustment lever screen 690 is displayed so as to be superimposed on the simulation screen 600; however, the configuration is not limited to this. For example, the operation adjustment lever screen 690 may be displayed in a display area different from the simulation screen 600 in the display 502.

[0081] Subsequently, in step S23, the operation setting lever 700' is operated on the operation setting lever screen 690. When the arrows 700a' or the rings 700b' are operated by dragging with the mouse 504, the arrows 700a or the rings 700b displayed in the virtual space VS are operated in synchronization with an operation of the arrows 700a' or the rings 700b', thereby editing the virtual robot arm body 200V.

[0082] In the present embodiment described above, an operation setting lever is displayed in a different area (screen) from the simulation screen 600. Thus, it is possible for the user to set setting information related to a display of the operation setting lever 700, and even if the operation setting lever overlaps the model in the virtual space, it is possible to perform an operation with the operation setting lever in a different area (another screen), thus improving operability. When setting information about the operation setting lever 700 itself is allowed to be changed by the user, the operability of the operation setting lever 700 is improved. The various embodiments and modifications described above can be implemented in combination with the present embodiment and / or modification. Fourth embodiment

[0083] Next, a fourth embodiment of the present disclosure will be described in detail. When the size of the operation setting lever 700 is updated in synchronization with a zoom operation of the virtual space VS, the operation setting lever is reduced or enlarged in synchronization with zooming out and zooming in, so that it may be difficult to operate the operation setting lever 700. Therefore, in the present embodiment, a mode in which, as setting information when displaying the operation setting lever 700, the size of the operation setting lever 700 is maintained even when a zoom operation of the virtual space VS is performed will be described.

[0084] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those of the various embodiments described above, and the detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0085] Fig. 16 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 17A to 17C are diagrams illustrating a display of the operation setting lever 700 whose display magnification is changed according to the present embodiment.

[0086] Initially, in step S31, a zoom operation of the screen of the virtual space VS is performed. Zoom operation can be performed using a general wheel operation of the mouse 504, a slider operation, input of a numeric value, or the like. If the display 502 displaying the simulation screen 600 is a touch panel, zoom operation can be performed by pinching out or pinching in by a user's touch.

[0087] Subsequently, in step S32, a zoomed display magnification in the operation setting lever 700 is acquired corresponding to a set display magnification and a zoom magnification by a zoom operation. For the set display magnification, the set display magnification described in the above-described second embodiment is acquired. A default set display magnification is set to 100[%]. A zoom magnification is an input magnification in a zoom operation. The display magnification set this time is 100[%]. Then, a description will be given on the assumption that a zoom magnification in the case of zooming out in a zoom operation is 30[%]. Likewise, a description will be given on the assumption that a zoom magnification in the case of zooming in is 300[%] in a zoom operation.

[0088] In step S32, the quotient of a set display magnification divided by a zoom magnification (set display magnification / zoom magnification) is calculated to one decimal place. By including not an integer but one decimal place, precise adjustment of the size by zooming can be performed. In the present embodiment, a case will be described where calculation is performed to two decimal places. In the case of the present embodiment, since a zoom-out operation is the quotient of 100[%] divided by 30[%], that is, 3.33..., the display magnification after a zoom operation is 333.33[%]. Since for a zoom-in operation, the quotient of 100[%] divided by 300[%] is 0.33..., the display magnification after a zoom operation is 33.33[%].

[0089] Subsequently, in step S33, a display of the operation setting lever 700 is updated according to the zoomed display magnification acquired in step S32. Fig. 17A shows the state according to Fig. 9E and shows the operating adjustment lever 700 before zooming. Fig. Fig. 17B shows the operation setting lever 700 displayed in the virtual space VS, which is changed at a zoom magnification of 30[%] from the state shown in Fig. 17A has been zoomed out. Fig. 17C shows the operation setting lever 700 displayed in the virtual space VS zoomed in at a zoom magnification of 300[%].

[0090] The entire virtual model, which is different from the operation setting lever 700, is smaller than that in Fig. 17B due to zooming out; however, the operation setting lever 700 is displayed at a magnification that is the acquired zoomed display magnification of 333.33[%] and is obviously larger than the virtual model. In other words, the size equivalent to the size of the operation setting lever 700 in the state shown in Fig. 17A. Consequently, even when a zoom operation is performed, the apparent size of the screen controlled by the operation setting lever 700 remains in the state shown in Fig. 17A unchanged.

[0091] The entire virtual model, which is different from the operation setting lever 700, is larger than that in Fig. 17C due to zooming in; however, the operation setting lever 700 is displayed at a magnification that is the acquired zoomed display magnification of 33.33[%] and is obviously smaller than the virtual model. In other words, the size equivalent to the size of the operation setting lever 700 in the state shown in Fig. 17A. Consequently, even when a zoom operation is performed, the apparent size on the screen remains unchanged from the operation setting lever 700 according to Fig. 17A.

[0092] According to the present embodiment, even when a zoom operation is performed in the virtual space VS, a display is updated such that the size of the operation setting lever on the screen is apparently unchanged. Thus, it is possible to reduce difficulty in accessing the operation setting lever due to excessive reduction in the size of the operation setting lever resulting from zooming out. It is also possible to reduce difficulty in accessing the operation setting lever due to excessive enlargement of the size of the operation setting lever resulting from zooming in to cause an occurrence of a part of the operation setting lever being displayed outside the screen. It is thus possible to easily operate the operation setting lever after a zoom operation, so that operability is improved.When setting information about the operation setting lever 700 itself is allowed to be changed by the user, the operability of the operation setting lever 700 is improved. The various embodiments and modifications described above can be implemented in combination with the present embodiment and / or modification. To allow the user to set whether the display magnification is maintained, a setting button or the like may be displayed on the simulation screen 600 or the operation setting lever setting screen 800. Fifth embodiment

[0093] Next, a fifth embodiment of the present disclosure will be described in detail. When the operation setting lever 700 is operated, a cursor is placed and dragged with the mouse 504 over the model corresponding to the arrows 700a and the rings 700b. However, if the arrows 700a or the rings 700b are buried within the virtual model of the robot arm or the workpiece, the operation setting lever 700 may not be able to be selected. For this reason, in the present embodiment, a mode will be described in which virtual models other than the operation setting lever 700 are displayed in a transparent mode while the operation setting lever 700 is displayed as setting information in a display of the operation setting lever 700, in order to make it easier to operate the operation setting lever 700.

[0094] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those according to the various embodiments described above, and the detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0095] Fig. 18 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 19A and Fig. 19B are diagrams illustrating virtual models in the transparent mode according to the present embodiment. Fig. 19A shows the virtual space VS in the simulation screen 600 in a state where the operation setting lever 700 is not displayed. Fig. 19B shows the virtual space VS in the simulation screen 600 in a state in which the operation setting lever 700 is displayed. As shown in the Fig. 19A and Fig. 19B, in the present embodiment, the operation setting lever display button 630 is displayed on the simulation screen 600.

[0096] As it is in Fig. 18, the operation setting lever 700 is initially displayed in step S41. When the operation setting lever display button 630 is pressed with the mouse 504 in a state according to Fig. 19A is depressed, the operation setting lever 700 is moved in the virtual space VS as shown in Fig. 19B shown. In the state according to Fig. 19B, the operation setting lever display button 630 is indicated by “HIDE OPERATION SETTING BUTTON (I)”, and when the operation setting lever display button 630 is depressed in this state, the operation setting lever 700 is hidden as shown in Fig. 19A. For example, a show / hide state can be toggled by an icon or a shortcut key, such as a function key of the keyboard 503. A show / hide state can be toggled by clicking on the virtual space VS.

[0097] Subsequently, in step S42, as shown in Fig. 19B, after the operation setting lever 700 is displayed, the virtual robot arm body 200V, the virtual robot hand body 300V, and the virtual workpiece WV, which are the virtual models other than the operation setting lever 700, are displayed in a semi-transparent mode.

[0098] Subsequently, in step S43, when a portion where the transparent virtual model overlaps the operation setting lever 700 is accessed by a user, priority is given to access to the operation setting lever 700. In the present embodiment, the order of priority is set; however, access only to the operation setting lever 700 can be accepted.

[0099] When a display is updated to a transparent mode and access to the operation setting lever 700 is given priority, the positions of the arrows 700a and the rings 700b are found even if, for example, the operation setting lever 700 overlaps the workpiece WV, and the arrows 700a or the rings 700b can be operated with the mouse 504. While the operation setting lever 700 is displayed, the transparent mode and the access priority state are maintained.

[0100] Subsequently, in step S44, when the operation setting lever display button 630 is depressed, the operation setting lever 700 is hidden. Subsequently, in step S45, the transparent mode of the virtual models other than the operation setting lever 700 is ended, and the display state is changed to the display state according to Fig. 19A is returned (updated). The access order of priority set in step S43 is cleared. When only access to the operation setting lever 700 is accepted, access to the virtual models is also accepted.

[0101] According to the present embodiment, even when the operation setting lever and the virtual models overlap each other, the virtual models different from the operation setting lever are set to be displayed in a transparent mode while the operation setting lever is displayed. Thus, it is possible to allow the user to accurately understand the position of the operation setting lever. It is possible to accurately operate the operation setting lever by giving higher priority to access to the operation setting lever over access to the virtual models. When setting information about the operation setting lever 700 itself is allowed to be changed by the user, the operability of the operation setting lever 700 is improved.

[0102] In the present embodiment, the description is given in a state where the operation setting lever 700 is in Fig. 19A; however, the configuration is not limited to this. For example, in Fig. 19A, the operation adjustment lever 700 may be configured to be displayed in a transparent mode. The various embodiments and modifications described above may be implemented in combination with the present embodiment and / or modification. Sixth embodiment

[0103] Next, a sixth embodiment of the present disclosure will be described in detail. When teaching the virtual robot arm body 200V, it is possible to make it easy for the user to understand whether the posture being taught is a posture that the robot arm body 200 can assume, so that it is possible to provide further highly operable simulation. In the present embodiment, an operation mode will be described to make it easy for the user to understand whether the posture of the virtual robot arm body 200V is a posture that can be assumed by the robot arm body 200 via the operation setting lever 700.

[0104] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those according to the various embodiments described above, and the detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0105] The Fig. 20A to 20C are diagrams showing examples of the posture of a singular point of the virtual robot arm body 200V according to the present embodiment. A singular point is a posture where the robot arm becomes uncontrollable from the standpoint of robot arm posture calculation equations. Examples of the singular point include the posture of a state where a hand (the virtual robot hand body 300V) is extended to the end. Fig. Figure 20A shows a state where the position of the virtual robot arm body 200V is not a singular point. Fig. Figure 20B shows a state where the posture of the robot arm body 200V is near a singular point. Fig. 20C shows a state where the virtual robot arm body 200V is a singular point. In the present embodiment, the singular point of the state where the hand is extended to the end is described; however, the singular point is not limited to this. For example, in a case of a six-axis articulated robot, a case where two or more axes are arranged in a straight line is also a singular point.

[0106] In the case according to Fig. 20A, there are degrees of freedom in the up, down, right, and left directions of the virtual robot hand body 300V, whereby control is possible. In the state according to Fig. 20C, there are no degrees of freedom in a planar direction with respect to a direction in which the hand is extended. Therefore, when this state is implemented in the robot arm body 200 of the real machine, control due to the singular point is impossible. It is important to check for such a singular point at the time of teaching. Heretofore, such a singular point has mostly been found as an error when a program using teaching points obtained through teaching is run on real machines. The present embodiment makes it possible to check the state of a singular point during operation of the virtual robot arm body 200V with the operation setting lever 700.

[0107] Fig. Fig. 21 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 22A and Fig. 22B and the Fig. 23A and Fig. 23B show diagrams related to a display of the operation setting lever 700 with an indication of a singular point according to the embodiment.

[0108] Initially, in step S51, the user operates the virtual robot arm body 200V and / or the virtual robot hand body 300V with the operation adjustment lever 700 to change the posture of the virtual robot arm body 200V.

[0109] Subsequently, in step S52, singular point calculation processing is executed. In singular point calculation processing, generally, Jacobian calculation is performed with the pose at a movement target as input, it is determined whether a Jacobian becomes 0 (singular point: abnormal) or a value close to 0 (near a singular point: warning), and the result is output. It is determined whether the pose is a singular point or approaching a singular point. A general technique is used for Jacobian calculation, so the description thereof is omitted. A method of singular point calculation processing may be a method other than Jacobian.For example, in the case of a singular point where two or more axes are arranged in a straight line, a specific joint angle may be monitored, or axes may be monitored by providing a threshold value in a certain range from an extension line of a specific axis.

[0110] Subsequently, in step S52, the result of the singular point calculation at the operation setting lever 700 is indicated. Fig. 22A shows a diagram at the time of showing the state of the singular point with color on a square model (object) 700c provided at the center of the operation setting lever 700. In the present embodiment, the arrows 700a may be referred to as a first model, the square model 700c may be referred to as a second model, and the rings 700b may be referred to as a third model. In a normal (neither a singular point nor near a singular point) case, the square model 700c is displayed in a first color (e.g., blue). In a warning (near a singular point) case, the square model 700c is displayed in a second color (e.g., yellow). In an abnormal (singular point) case, the square model 700c is displayed in a third color (e.g., red). The arrangement of colors is not limited to this.Any color can be used as long as different, distinguishable colors are used. The quadrangular model 700c can have a different shape. A selected shape, such as a triangular shape, a rectangular shape, a trapezoidal shape, a circular shape, and a star shape, can be used. The normal case can be referred to as a first phase, the warning case can be referred to as a second phase, and the abnormal case can be referred to as a third phase.

[0111] Fig. 22B shows a diagram at the time of displaying the state of a singular point with a pattern, such as a design pattern, on the square model (object) 700c provided at the center of the operation setting lever 700. In the normal (neither a singular point nor near a singular point) case, the square model 700c is displayed in a first pattern, which could also be no pattern (unpatterned). In the warning (near a singular point) case, the square model 700c is displayed in a second pattern (for example, a mesh pattern). In the abnormal (singular point) case, the square model 700c is displayed in a third pattern (in a dot pattern). The pattern is not limited to this. Any pattern can be used as long as the user is able to identify normal, warning, or abnormal.In the present embodiment, the color and design of the central square model have been described; however, the configuration is not limited to this. For example, the arrows 700a or the rings 700b can be changed.

[0112] The Fig. 23A and Fig. 23B are diagrams illustrating a display of the operation setting lever 700 at the time of indicating the state of a singular point with a dialog window 710 according to the present embodiment. Fig. 23A shows the dialog window 710 in the warning (near a singular point) case. Fig. 23B shows the dialog window 710 in the abnormal (singular point) case. The dialog window 710 is displayed near the operation setting lever 700 and is thus interlocked with a movement of the operation setting lever 700. Thus, it is possible to determine the singular point state. In the present embodiment, a case will be described in which the dialog window 710 is displayed when interlocked with a movement of the operation setting lever 700. The dialog window 710 may be displayed at a predetermined fixed position in the virtual space VS, the simulation screen 600, or the display 502.

[0113] The Fig. 24A to 24D show changes in the display state of the operation setting lever 700 at the time of changing the posture of the virtual robot arm body 200V and / or the virtual robot hand body 300V with the operation setting lever 700 in the present embodiment.

[0114] Fig. Fig. 24A shows the simulation screen 600 at the time when the virtual robot arm body 200V is close to a singular point. In this state, the square model 700c of the operation setting lever 700 is colored yellow, as shown in Fig. 22A, or displayed as a network as shown in Fig. 22B. Suppose that the pose transitions from this state to a pose determined as a singular point in the virtual robot arm body 200V, as shown in Fig. 24B. At this time, when the user changes the posture of the virtual robot arm body 200V, the square model 700c of the operation setting lever 700 is colored red as shown in Fig. 22A, or dotted as shown in Fig. 22B. In this way, a display of the square model 700c of the operation setting lever 700 is synchronized (interlocked) with the state of the virtual robot arm body 200V changed by the user. Thus, as the user operates the operation setting lever 700, the user is immediately able to understand the state of the posture of the virtual robot arm body 200V.

[0115] In a similar way, Fig. 24C shows the simulation screen 600 at the time when it is determined that the virtual robot arm body 200V is near a singular point. In this state, the dialog window 710 displayed as "near a singular point" as shown in Fig. 23A, is displayed in a meshing with a movement of the operation setting lever 700. Suppose that the position changes from this state to a position in which it is determined that the virtual robot arm body 200V is a singular point, as shown in Fig. 24D. At this time, when the user changes the position of the virtual robot arm body 200V, the dialog window 710 is displayed with an indication of a singular point, as shown in Fig. 23B. In this way, a display of the dialog window 710 is synchronized (interlocked) with the state of the virtual robot arm body 200V being changed by the user. Thus, the user is immediately able to understand the state of the posture of the virtual robot arm body 200V.

[0116] According to the present embodiment, during operation of the robot arm, the display mode of the operation setting lever is changed to indicate the state of the singular point in the robot arm. With this configuration, it is possible to immediately determine the position that should not be taught, making it possible to effectively perform teaching work. The state of the robot arm is indicated at the model of the operation setting lever or near the operation setting lever. Thus, it is possible to make the user who operates the operation setting lever visually recognize the state of the robot arm reliably. To enable the user to set whether to indicate a singular point with the operation setting lever 700, a setting button or the like may be displayed on the simulation screen 600 or the operation setting lever setting screen 800.If setting information about the operation setting lever 700 itself is allowed to be changed by the user, the operability of the operation setting lever 700 is improved.

[0117] In the present embodiment, the description has been given using an example in which the robot arm becomes the state of a singular point; however, the configuration is not limited to this. For example, an interference state in which the virtual robot arm body 200V and / or the virtual robot hand body 300V is interfered with by a surrounding object can be classified into phases and displayed as described above. The interference state can be classified into a normal (neither interference nor near interference) case, a warning (near interference with respect to a threshold set between models) case, and an abnormal (interference that models overlap each other) case and displayed.An out-of-range state, that is, whether the virtual robot arm body 200V and / or the virtual robot hand body 300V falls outside an operating range of a mechanical mechanism such as a motor, a speed reducer, and a link in the robot arm body 200, can be classified into phases and displayed as described above. The out-of-range state can be classified and displayed into a normal (neither out-of-range nor near-out-of-range) case, a warning (near-out-of-range with respect to a threshold value that is set), and an abnormal (out-of-range) case. The various embodiments and modifications described above can be implemented in combination with the present embodiment and / or modification.

[0118] In the present embodiment, the color or pattern of the square model 700c is changed according to several phases; however, the configuration is not limited to this. For example, the shape of the square model 700c may be changed, such that in a normal case, the shape of the square model 700c is rectangular; in a warning case, the shape of the square model 700c is triangular; and in an abnormal case, the shape of the square model 700c is a star. When indicating the state of deterioration, a display of the operation setting lever that operates the virtual workpiece WV may be changed. Seventh embodiment

[0119] Next, a seventh embodiment of the present disclosure will be described in detail. In the various embodiments described above, when setting the setting information related to a display of the operation setting lever 700, a display of the operation setting lever 700 is updated, for example, by opening a menu and inputting numerical values. In the present embodiment, a mode in which setting information is changed in a display by directly changing it with the cursor of the operation setting lever 700 displayed in the virtual space VS will be described.

[0120] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those according to the various embodiments described above, and the detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0121] Fig. Fig. 25 shows a control flowchart related to a display of the operation setting lever 700 according to the present embodiment. Fig. 26A to 26C are diagrams illustrating a display of the operation setting lever 700 whose display position and / or display magnification is changed according to the present embodiment.

[0122] As it is in Fig. 25, initially, in step S61, an operation setting lever display change mode is set by the user. Fig. 26A shows a diagram at the time of setting the operation setting lever display change mode from an off state to an on state by depressing an operation setting lever display change mode button 640 with a cursor 650. The user is able to switch between the on and off states of the operation setting lever display change mode by depressing the operation setting lever display change mode button 640.

[0123] Subsequently, in step S62, operation of the arrows 700a and rings 700b in the operation setting lever 700 with the cursor 650 is disabled. Thus, it is possible to reduce a situation in which the virtual robot arm body 200V and / or the virtual robot hand body 300V is moved by mistakenly operating the arrows 700a or the rings 700b when the operation setting lever 700 itself is intended to be moved, enlarged, or reduced. A case where the operation setting lever display change mode is off can be referred to as a first mode, and a case where the operation setting lever display change mode is on can be referred to as a second mode.

[0124] Subsequently, in step S63, a change in the display position and / or the display magnification of the operation setting lever 700 is enabled with the cursor 650. Then, in step S64, a display of the operation setting lever 700 is updated with the display position and / or the display magnification set with the cursor 650. Fig. 26B shows a diagram at the time when the display position of the operation setting lever 700 is changed with the cursor 650. When the model representing the operation setting lever 700 is dragged in a state where it is clicked with the cursor 650, the model is allowed to be moved with respect to the coordinate system set in the operation setting lever setting screen 800. In Fig. 26B the display position is changed in the absolute coordinate system world.

[0125] Fig. 26C shows a diagram at the time when the display magnification of the operation setting lever 700 is increased with the cursor 650. When the operation setting lever 700 is clicked with the cursor 650, faces 641 and squares 642 are displayed. The squares 642 can have any shape, such as a triangular shape, a rectangular shape, a trapezoidal shape, a circular shape, and a star shape. When any of the squares 642 is dragged in a state where it is clicked, the size of a display of the operation setting lever 700 is changed. Fig. 26C is when the lower right quadrangle 642 is moved to the bottom right of the sheet in a state where it is clicked, from the state according to Fig. 26B, the operation setting lever 700 is in an enlarged state. On the other hand, when the lower right square 42 is moved to the upper left of the sheet in a state where it is clicked, from the state shown in Fig. 26C is pulled, the operating adjustment lever 700 can be reduced.

[0126] When the operation setting lever display change mode button 640 is depressed again by the user and the operation setting lever display change mode is canceled, changing the display position and / or display magnification of the operation setting lever 700 with the cursor 650 is disabled. Operation of the arrows 700a and rings 700b in the operation setting lever 700 with the cursor 650 is enabled.

[0127] According to the present embodiment, it is possible to allow the user to directly change setting information on the operation setting lever in the virtual space. Thus, it is possible to make the user intuitively understand the display position and display magnification of the operation setting lever, thus further improving the operability of the operation setting lever. Allowing the user to change setting information on the operation setting lever 700 itself improves the operability of the operation setting lever 700.

[0128] As it is in Fig. 27, the display position and / or display magnification of the operating adjustment lever 700 can be changed. In Fig. 27, the cursor 650 is operated while a predetermined key on the keyboard 503 is held down. Thus, the display position and / or the display magnification of the operation setting lever 700 can be changed. In Fig. 27, the cursor 650 is operated with the mouse 504 in a state where the Ctrl key and the D key remain depressed. Thus, the display position and / or the display magnification of the operation setting lever 700 is changed. Keys to be input and a combination of keys are not limited to the configuration described above.

[0129] As it is in Fig. As shown in Fig. 27, when the Ctrl key and the D key are input, it is determined that the operation setting lever display change mode is on. Then, operation of the arrows 700a and the rings 700b in the operation setting lever 700 with the cursor 650 is disabled, and change in the display position and / or display magnification of the operation setting lever 700 with the cursor 650 is enabled. When the model representing the operation setting lever 700 is dragged in a state where it is clicked with the cursor 650 in this state (the keys remain pressed down), the model is allowed to be moved with respect to the coordinate system set in the operation setting lever setting screen 800. When the operation setting lever 700 is clicked with the cursor 650 in this state (the buttons remain pressed down), pages 641 and squares 642 are displayed.When one of the squares 642 is dragged in a state where it is clicked, the size of a display of the operation setting lever 700 can be changed.

[0130] When the Ctrl key and the D key are released (fingers are released from the keys), changing the display position and / or display magnification of the operation setting lever 700 with the cursor 650 is disabled. Operating the arrows 700a and the rings 700b in the operation setting lever 700 with the cursor 650 is enabled.

[0131] Also, with the above-described embodiment, it is possible to allow the user to directly change a display of the operation setting lever in the virtual space. Thus, it is possible to make the user intuitively understand the display position and display magnification of the operation setting lever, so that the operability of the operation setting lever is further improved. Since a display of the operation setting lever can be directly changed immediately by a key input, further improvement in operability is possible. When setting information about the operation setting lever 700 itself is allowed to be changed by the user, the operability of the operation setting lever 700 is improved. The above-described various embodiments and modifications can be implemented in combination with the present embodiment and / or modification.The various embodiments and modifications described above may be implemented in combination with the present embodiment and / or modification. Eighth embodiment

[0132] Next, an eighth embodiment of the present disclosure will be described in detail. In the various embodiments described above, the description is given in an example where a device that executes a simulation is the information processing device 500, such as a personal computer. However, the device that executes a simulation may be a tablet handheld programming device.

[0133] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those according to the various embodiments described above, and the detailed description is omitted. In the embodiment mentioned below, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0134] Fig. Figure 28 is a diagram illustrating the robot system 1000 using a tablet teaching device 900 according to the present embodiment. The tablet teaching device 900 is connected to the controller 400. Fig. 29 is a diagram showing the simulation screen 600 according to the present embodiment. In Fig. 29, a hand operation section 901 and a joint operation section 902, which are used to operate the robot arm body 200 and the robot hand body 300 of the actual machine, are displayed. The hand operation section 901 enables the robot hand body 300 to be operated and allows the robot hand body 300 to be moved in three translational directions, respectively along three axes that are orthogonal to each other, and three rotational directions, respectively around the three axes. The joint operation section 902 allows the joints of the robot arm body 200 to be moved in predetermined rotational directions.

[0135] According to the present embodiment described above, the simulation screen 600 is displayed on the teaching pendant 900, and the operation setting lever 700 is displayed. Thus, with the teaching pendant 900, it is possible to teach the real machine while executing a simulation in the virtual space. This improves workability. The teaching pendant 900 may be an operator control panel for teaching. The teaching pendant 900 may have the same function as the information processing device 500, or the controller 400 may include a control unit and a calculation unit of the information processing device 500, and a result may be transmitted to the teaching pendant 900 through communication.The various embodiments and modifications described above may be implemented in combination with the present embodiment and / or modification. Ninth embodiment

[0136] A ninth embodiment of the present disclosure will be described in detail. In the various embodiments described above, the description is given in an example where the device that performs a simulation is a PC, a tablet computer, or the like. However, the device that performs a simulation may be an augmented reality (AR) system.

[0137] Hereinafter, basic parts of the hardware configuration, the configuration of the display screen, and the like are the same as those according to the various embodiments described above, and the detailed description is omitted. In the following embodiment, the same reference numerals are assigned to the same or substantially the same elements, and the detailed description is omitted.

[0138] Fig. 30 is a diagram illustrating the robot system 1000 using an AR system according to the present embodiment. A head-mounted display 910 and an operation part 920 are connected to the controller 400. In the present embodiment, the head-mounted display 910, the operation part 920, and the controller 400 can be collectively referred to as an information processing device. An augmented space AR in which augmented reality is superimposed on the robot arm body 200 of the real machine is displayed on the head-mounted display 910. The simulation screen 600 according to the present embodiment displays the robot arm body 200 of the real machine such that the operation setting lever 700 and the cursor 650 on the robot arm body 200 are superimposed by the augmented reality.An operation of the operation setting lever 700 on the simulation screen 600 can be performed by an operation of the cursor 650 with the operation part 920. The AR system is constructed of the head-mounted display 910 and the operation part 920. The details of a screen displayed on the head-mounted display 910 will be described below.

[0139] Fig. 31 is a control block diagram showing a control system according to the present embodiment. As shown in Fig. 31, the controller 400 includes a CPU 911 as hardware. The controller 400 further includes a storage device 915 formed of a ROM 915a, a RAM 915b, an HDD 915c, and the like. The controller 400 further includes an interface 916a for communicating and connecting with an input device such as the operation part 920, and an interface 916b for communicating and connecting with the head-mounted display 910.

[0140] The control device 400 further includes an interface 917 for transmitting and receiving data in the form of, for example, the file 530 to or from an external device, such as another simulator device and a robot device. These interfaces are each formed, for example, from a serial bus, a parallel bus, a network interface, or the like.

[0141] The ROM 915a is a non-transitory storage device. A basic program read by the CPU 911 upon startup of the computer is stored in the ROM 915a. The RAM 915b is a temporary storage device used in arithmetic operation processing of the CPU 911. The HDD 915c is a non-transitory storage device that stores various data, such as arithmetic operation processing results of the CPU 911.

[0142] In the present embodiment, a program functioning as application software is stored in the HDD 915c. The CPU 911 is capable of executing control over the robot arm body 200 and the robot hand body 300 in a real machine environment by running the program.

[0143] In the present embodiment, a non-transitory computer-readable storage medium is the HDD 915c, and a program functioning as application software is recorded on the HDD 915c; however, the configuration is not limited to this. The program can be recorded on any recording medium as long as the recording medium is a non-transitory computer-readable recording medium. Examples of the recording medium for supplying the program to a computer include a flexible floppy disk, an optical disk, a magento-optical disk, a magnetic tape, and a non-volatile memory. An SSD can be used.

[0144] The CPU 911 controls the entire system of the control device 400. Operation processing units of the CPU 911 include a display unit 912, an interpretation unit 913, and a calculation unit 914. The display unit 912 updates a display of an augmented reality screen based on an image from a camera of the head-mounted display 910 and information stored in the storage device 915, and transmits a display command to the head-mounted display 910.

[0145] The interpretation unit 913 controls operation of the augmented reality screen with the input device such as the operation part 912. The interpretation unit 913 interprets operation details input by the operation part 920, requests a necessary calculation from the calculation unit 914, and displays a display on the display unit 912 to update a display according to the calculated result. The interpretation unit 913 controls the robot arm body 200 and the robot hand body 300 according to the calculated result. The calculation unit 914 performs calculation processing responsive to dragging (described below) and the operation of the real machine according to operation information input by the interpretation unit 913. The calculated result is stored in the storage device 915.

[0146] The storage device 915 stores display information about the operation adjustment lever 700 displayed on the simulation screen 600, which is a screen displayed in augmented reality. The information stored in the storage device 915 is output in response to a request from the CPU 911 or updated in response to a request from the CPU 911. In response to the request from an external device or a specific operation on the operation part 920, the CPU 911 is capable of transferring the information stored in the storage device 915 from the interface 917 in the form of the file 530. The CPU 911 is capable of reading the file 530 from an external source via the interface 917 as needed.

[0147] For example, when starting the control device 400 or in restoration (recovery) processing, the CPU 911 reads the file 530 output in the past from an external device (an external storage device such as an SSD and a NAS). Then, the CPU 911 can reproduce a previous storage state by updating the storage device 915. In the present embodiment, a storage area of the storage device 915 that stores the components is selectable. For example, a predetermined area in the RAM 915b or a storage area (corresponding to a predetermined file, for example) of the HDD 915c can be used. An example of the control system of the control device 400 to which the head-mounted display 910 and the operation part 920 are connected is as described above.

[0148] Fig. 32 is a diagram illustrating the simulation screen 600 displayed on the head-mounted display 910 according to the present embodiment. An image acquired (filmed) by a camera installed in the head-mounted display 910 is displayed on the head-mounted display 910 as the augmented space AR, displaying the robot arm body 200, the robot hand body 300, and the workpiece W. Furthermore, the menu bar 610 is displayed as augmented reality, displaying the operation setting lever setting button 620, the operation setting lever display change mode button 640, a hand operation button 660, and an arm operation button 670. The operation adjustment lever 700 and the cursor 650 are displayed in a superimposed manner as augmented reality in the augmented space AR in which the robot arm body 200 and the robot hand body 300 are displayed.

[0149] The cursor 650, displayed as augmented reality, can be operated using the operation part 920. The hand operation button 660 or the arm operation button 670 can be pressed with the cursor 650. When the hand operation button 660 is pressed, the orientation of the robot hand body 300 can be changed by operating the operation setting lever 700 with the cursor 650 in a state where the posture of the robot arm body 200 remains unchanged and the position of the robot hand body 300 is fixed. When the posture that cannot be assumed by the robot hand body 300 is input through the operation setting lever 700 according to the movable range of the robot arm body 200, a dialog window 680 is displayed, which is Fig. 33. In the example shown in Fig. 33, "OUT OF MOVABLE RANGE!" is displayed in the dialog window 680 to warn the user. Then, the value input by the user is not output to the robot hand body 300 (the posture of the robot hand body 300 is maintained). Then, after the dialog window 680 is displayed for a predetermined time, the dialog window 680 is hidden, and an operation with the operation setting lever 700 is accepted again.

[0150] When the arm operation button 670 is depressed, the posture of the robot arm body 200 can be changed by operating the operation setting lever 700 with the cursor 650 so that the posture changed by the robot hand body 300 can be assumed. In other words, it is possible to change the position and posture of the robot hand body 300, with the posture of the robot arm body 200 being changed following the changed position and posture of the robot hand body 300. When the posture that cannot be assumed by the robot hand body 300 and the robot arm body 200 is inputted by the operation setting lever 700 according to the movable range of the robot arm body 200, the dialog window 680 is displayed as shown in Fig. 33. In the example shown in Fig. 33, "OUT OF MOVABLE RANGE!" is displayed in the dialog window 680 to warn the user. Then, the value input by the user is not output to the robot arm body 200 and the robot hand body 300 (the posture of the robot arm body 200 and the robot hand body 300 is maintained). Then, after the dialog window 680 is displayed for a predetermined time, the dialog window 680 is hidden, and an operation with the operation setting lever 700 is accepted again.

[0151] The dialog window 680 in the present embodiment indicates a state related to the movable range of the robot arm body 200 and the robot hand body 300; however, the configuration is not limited to this. The dialog window 680 may indicate, for example, a singular point state or an interference state. When the robot arm body 200 and / or the robot hand body 300 of the actual machine is operated with the operation setting lever 700, the operation setting lever 700 may indicate outside the movable range, a singular point, or interference in multiple phases, as described in the sixth embodiment.

[0152] When the operation setting lever setting button 620 is pressed, the operation setting lever setting screen 800 is displayed on the head-mounted display 910, as described in the first to third embodiments, and the display position, display magnification, or display in a different range of the operation setting lever can be performed. When the operation setting lever display button 630 is pressed, it is possible to change the show / hide state of the operation setting lever 700, as described in the fifth embodiment. When the operation setting lever display change mode button 640 is pressed, it is possible to directly change the display position and display magnification of the operation setting lever 700, as described in the seventh embodiment.

[0153] Next, a zoom operation of the operation setting lever 700 in a case where it is displayed by augmented reality will be described in detail. When augmented reality is used, a procedure differs from that according to the fourth embodiment. Fig. 34 shows a control flowchart according to the present embodiment.

[0154] As it is in Fig. 34, initially, an operation of the robot arm body 200 is started in step S71. The operation is started when the arm operation button 670 is depressed. When the arm operation button 670 is depressed, the operation setting lever 700 is displayed in the expanded space AR in step S72.

[0155] The operation setting lever 700 may be displayed at a position of the TCP or may be displayed at a relative position that is a predetermined distance away from the TCP, as described in the first embodiment.

[0156] Subsequently, in step S73, the user wearing the head-mounted display 910 approaches or moves away from the robot arm body 200. In step S74, a zoom magnification is calculated (acquired) so that the display magnification of the operation adjustment lever 700 remains constant even when the user approaches or moves away from the robot arm body 200. The zoom magnification acquired in step S74 is obtained by pattern matching the virtual robot arm body 200V with the robot arm body 200 of the real machine and acquiring the display magnification of the virtual robot arm body 200V on the display.

[0157] For example, when the user moves away from the robot arm body 200 and the robot arm body 200 decreases by 30[%] with respect to the virtual robot arm body 200V (zooms out), the display magnification is set to 333.33[%] according to the fourth embodiment. On the other hand, when the user approaches the robot arm body 200 and the robot arm body 200 increases by 300[%] with respect to the virtual robot arm body 200V (zooms in), the display magnification is set to 33.33[%] according to the fourth embodiment. For pattern matching, a general technique is used, so the description thereof is omitted. In step S75, a display of the operation setting lever 700 is updated according to the display magnification set in step S74.To allow the user to set whether the display magnification is maintained, a setting button or the like may be displayed on the simulation screen 600 or the operation setting lever setting screen 800.

[0158] The virtual robot arm body 200V and the virtual robot hand body 300V (not shown) can be displayed in a superimposed manner in the augmented AR space. In this case, the virtual robot arm body 200V and the virtual robot hand body 300V can be operated as a simulation before the robot arm body 200V and the robot hand body 300V of the real machine are operated.

[0159] According to the present embodiment, the simulation screen 600 is displayed on the head-mounted display 910 using augmented reality (AR), and the operation setting lever 700 is displayed. Thus, it is possible to intuitively operate the operation setting lever 700 while viewing the robot arm body 200 of the actual machine, thus improving operability. The various embodiments and modifications described above can be implemented in combination with the present embodiment and / or modification. Other embodiments

[0160] Procedures of the above-described embodiments are specifically executed by a CPU. Thus, the CPU may be configured to read a recording medium on which a program of software capable of performing the above-described functions is recorded, and run the program. In this case, the program itself read from the recording medium implements the functions of the above-described embodiments, and the program itself and the recording medium on which the program is recorded are components of the present disclosure.

[0161] In each of the embodiments, a case where a computer-readable recording medium is a ROM, a RAM, or a Flash ROM, and a program is stored in the ROM, the RAM, or the Flash ROM has been described. However, the present disclosure is not limited to such modes. The program for executing the present disclosure may be recorded on any recording medium, as long as the recording medium is a computer-readable recording medium.

[0162] In the various embodiments described above, a case in which an articulated robot arm in which the robot arm body 200 includes a plurality of joints is used has been described; however, the number of joints is not limited to this. A vertical multi-axis configuration is described as one form of the robot arm. A configuration equivalent to that described above can also be implemented with joints of a different form, such as a horizontal joint type, a parallel link type, and an orthogonal robot.

[0163] The various embodiments described above are applicable to machines capable of automatically performing expansion and contraction, bending and stretching, up and down movements, right and left movements, or a swing operation or a combined operation thereof in accordance with information of a storage device provided in a control device.

[0164] The present disclosure is not limited to the embodiments described above, and many modifications are applicable within the technical concept of the present disclosure. Advantageous effects described in the embodiments of the present disclosure are merely the most preferable advantageous effects obtained from the present disclosure, and advantageous effects of the present disclosure are not limited to those described in the embodiments of the present disclosure. The various embodiments and modifications described above can be implemented in combination.

[0165] An embodiment(s) of the present disclosure may also be implemented by a computer of a system or device that reads and executes computer-readable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") for performing the functions of one or more of the embodiments described above, and / or that includes one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing the functions of one or more of the embodiments described above, as well as by a method performed by the computer of the system or device, for example, by reading and executing the computer-executable instructions from the storage medium,to perform the functions of one or more of the embodiments described above, and / or which controls one or more circuits to perform the functions of one or more of the embodiments described above. The computer may comprise one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)) and may comprise a network of separate computers or separate processors to retrieve and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may comprise, for example, one or more of a hard disk, random access memory (RAM), read-only memory (ROM), a memory of a distributed computing system,an optical disc (such as a Compact Disc (CD), a Digital Versatile Disc (DVD) or a Blu-Ray Disc (BD)™), a flash memory device, a memory card and the like.

[0166] An information processing device that performs a simulation using a virtual model. The information processing device includes a processing unit. The processing unit is configured to display the virtual model and an operation section on a display unit, the operation section being used by a user to operate the virtual model and to change setting information via the operation section in response to user input.

Claims

[1] An information processing device (500) that performs a simulation using a virtual model (200V, 300V), the information processing device (500) comprising a processing unit (511), the processing unit (511) being configured displaying a virtual model (200V, 300V) and an operating section (700) on a display unit (514), the operating section (700) being used by a user to operate the virtual model (200V, 300V), and, when the virtual model (200V, 300V) is in a certain state as a result of a user input, changing a display of the operation section (700) to indicate that the virtual model (200V, 300V) is in the certain state. [2] The information processing apparatus (500) according to claim 1, wherein a value which is a display condition on the operation section (700) when the operation section (700) is displayed together with the virtual model (200V, 300V) is changeable by an input of the user. [3] The information processing apparatus (500) according to claim 1 or 2, wherein the processing unit (511) is configured to change a display position of the operating section (700) in response to a user input. [4] Information processing apparatus (500) according to claim 3, wherein the processing unit (511) is configured to adopt a setting of a coordinate system at a time of changing the display position. [5] Information processing device (500) according to one of claims 1 to 4, wherein the processing unit (511) is configured to change a display magnification of the operation section (700) in response to a user input. [6] Information processing device (500) according to one of claims 1 to 5, wherein the processing unit (511) is configured in response to a user input, display a new operating section (700') in an area different from an area in which the operating section (700) is displayed, the new operating section (700') being interlocked with the operating section (700). [7] Information processing device (500) according to one of claims 1 to 6, wherein the processing unit (511) is configured when the virtual model (200V, 300V) is enlarged or reduced by a user input, to maintain a display magnification of the operation section (700) which is determined before the virtual model (200V, 300V) is enlarged or reduced. [8] Information processing device (500) according to one of claims 1 to 7, wherein the processing unit (511) is configured displaying the virtual model (200V, 300V) in a semi-transparent mode in response to a user input, wherein the virtual model (200V, 300V) is an object to be operated with the operation section (700). [9] Information processing apparatus (500) according to one of claims 1 to 8, wherein the virtual model (200V, 300V) includes a virtual robot and the processing unit (511) is configured to change a position or posture of the virtual robot by the user operating the operation section (700). [10] Information processing apparatus (500) according to one of claims 1 to 9, wherein the operating section (700) is a first model (700a) indicated by arrows in three-axis directions, a second model (700c) to which the first model (700a) is connected, and a third model (700b) indicated by rings for rotations around the arrows, respectively, and the processing unit (511) is configured to change a display mode of the second model (700c) to indicate that the virtual model (200V, 300V) is in the specific state. [11] The information processing apparatus (500) according to claim 10, wherein the processing unit (511) is configured to identify the specific condition by classifying the specific condition into several phases, and to change the display mode among multiple colors or multiple patterns according to the phases to indicate that the virtual model (200V, 300V) is in the certain state. [12] Information processing apparatus (500) according to claim 11, wherein the phases comprise a first phase in which it is determined that the virtual model (200V, 300V) is not in the specific state, a second phase in which it is determined on the basis of a threshold value that the virtual model (200V, 300V) is approaching the specific state, and a third phase in which it is determined that the virtual model (200V, 300V) is in the specific state, and the processing unit (511) is configured to display the second model (200V, 300V) in a first color or pattern for the first phase, to display the second model in a second color or pattern for the second phase, and to display the second model in a third color or pattern for the third phase. [13] Information processing device (500) according to one of claims 1 to 12, wherein the operating section (700) is formed from a first model (700a) represented by arrows in three-axis directions, a second model (700c) to which the first model (700a) is connected, and a third model (700b) represented by rings for rotations around the arrows, respectively, and the processing unit (511) is configured to change a shape of the second model (700c) to indicate that the virtual model (200V, 300V) is in the determined state. [14] Information processing apparatus (500) according to one of claims 1 to 13, wherein the processing unit (511) is configured when the virtual model (200V, 300V) is in a certain state as a result of a user input, displaying a second screen together with the operation section (700) and indicating in the second screen that the virtual model (200V, 300V) is in the certain state. [15] Information processing apparatus (500) according to one of claims 1 to 14, wherein the virtual model (200V, 300V) is a virtual robot arm and the determined state comprises at least one of a state in which the virtual robot arm is a singular point, a state in which the virtual robot arm is outside a movable range, and a state in which the virtual robot arm comes into interference with another virtual model (200V, 300V). [16] The information processing apparatus (500) according to any one of claims 1 to 15, wherein, when the virtual model (200V, 300V) is placed in the predetermined state as a result of the user operating the virtual model (200V, 300V) using the operating section (700), the display is performed on the operating section (700) to indicate that the virtual model (200V, 300V) is in the predetermined state. [17] Information processing device (500) according to one of claims 1 to 16, wherein the processing unit (511) is configured a first operation mode in which the virtual model (200V, 300V) is operated with the operation section (700), and a second operation mode in which a display position or display magnification of the operation section (700) is changed. [18] Information processing device (500) according to claim 17, wherein the processing unit (511) is configured when, in the first mode, a virtual key (640) for switching between the first mode and the second mode is operated or a predetermined input is input from an input device, to execute the second mode. [19] Information processing device (500) according to claim 17 or 18, wherein the processing unit (511) is configured allowing the user to directly change a display position or display magnification of the operation section (700) in the second mode. [20] The information processing apparatus (500) according to any one of claims 1 to 19, wherein at least one of a PC, a hand-held programming device (900) and a head-mounted display (910) is provided as the display unit. [21] The information processing apparatus (500) according to any one of claims 1 to 20, wherein the display is performed to indicate that the virtual model (200V, 300V) is in the specified state by changing a display of a virtual operation section model corresponding to the operation section (700). [22] The information processing apparatus (500) according to any one of claims 1 to 21, wherein the determined state is a state in which the virtual model (200V, 300V) cannot be reached. [23] The information processing apparatus (500) according to claim 2, wherein a setting screen for changing the value by the user is displayed on the display unit (514). [24] The information processing apparatus (500) according to claim 14, wherein a dialog window (680) also moves together in synchronism with the operation section (700) when the operation section (700) moves. [25] A robot system (1000) comprising a robot whose operation is adjusted by the information processing apparatus (500) according to any one of claims 1 to 24. [26] A method of manufacturing an article using the robot system according to claim 25. [27] Information processing method that performs a simulation using a virtual model (200V, 300V), the information processing method comprising: displaying the virtual model (200V, 300V) and an operating section (700) on a display unit, wherein the operating section (700) is used by a user to operate the virtual model (200V, 300V); and, when the virtual model (200V, 300V) is in a certain state as a result of a user input, changing a display of the operation section (700) to indicate that the virtual model (200V, 300V) is in the certain state. [28] Information processing device (500) for operating a device and displaying a virtual model (200V, 300V) on a screen that accommodates the augmented reality device, the information processing device (500) comprising a processing unit (511), the processing unit (511) being configured display the screen on a display unit, displaying an operating section (700) on the screen as the virtual model (200V, 300V), the operating section (700) being used by a user to operate the device, and, when the virtual model (200V, 300V) is in a certain state as a result of a user input, changing a display of the operation section (700) to indicate that the virtual model (200V, 300V) is in the certain state. [29] Information processing device (500) according to claim 28, wherein the processing unit (511) is configured when the virtual model (200V, 300V) is enlarged or reduced as the user moves away from or approaches the device, maintaining a display magnification of the operation section (700) before the device is enlarged or reduced. [30] An information processing method operating a device and displaying a virtual model (200V, 300V) with augmented reality on a screen housing the device, the information processing method comprising: displaying the screen on a display unit; displaying an operating section (700) on the screen as the virtual model (200V, 300V), the operating section (700) being used by a user to operate the device; and, when the virtual model (200V, 300V) is in a certain state as a result of a user input, changing a display of the operation section (700) to indicate that the virtual model (200V, 300V) is in the certain state. [31] A program capable of executing the information processing method according to claim 27 or 30. [32] A computer-readable recording medium storing the program according to claim 31.

Images