Image generation device

Abstract

To provide an image generating device for studying arrangements of a robot.SOLUTION: An image generating device 3 is provided with: an image generating part 36 that generates a display image of a three-dimensional model of a virtual robot using a relative positional relation between a reference marker 6 and a display device 4; an output part 37 that outputs the display image to the display device 4; a registering part 34 that registers a position and an attitude of an end effector of the virtual robot in response to reception of a registration instruction; and a robot positional relation obtaining part 35 that obtains a new positional relation between the reference marker 6 and the virtual robot in response to reception of a movement instruction. The image generating part 36 generates a display image of a three-dimensional model of the virtual robot in which the end effector is in a registered position and in a registered attitude, which is arranged to have the new positional relation with the reference marker 6, in response to the movement instruction. Such configuration enables a position at which the virtual robot is arranged to be studied, in a state in which the position and the attitude of the end effector are fixed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image generation device that generates a display image for displaying a three-dimensional model of a virtual robot whose end effector is at a registered position or position and orientation.

Background Art

[0002] Conventionally, when considering the arrangement at the time of introducing a robot, catalog specifications have been referred to. Also, a simulator that can output the optimal arrangement of the robot has been used (see Patent Document 1). In this simulator, workpieces and peripheral devices other than the robot can be installed, and an optimal arrangement study can be performed considering the error between the actual environment and the virtual environment in the simulator.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when using a simulator, it is necessary to prepare three-dimensional models of workpieces and peripheral devices other than the robot, resulting in an increase in cost. Also, in the arrangement study considering the error between the actual environment and the virtual environment, if there is an error greater than expected, there is a possibility of interfering with peripheral devices during the operation of the robot in the actual environment. Therefore, there has been a demand for performing the arrangement study of the robot at low cost and with high accuracy.

[0005] The present invention has been made in response to the above situation, and an object thereof is to provide an image generation device for performing the arrangement study of a robot at low cost and with high accuracy.

Means for Solving the Problems

[0006] To achieve the above objective, an image generation apparatus according to one aspect of the present invention includes: a storage unit that stores a 3D model of a virtual robot corresponding to a real robot; a marker position relationship acquisition unit that acquires the relative positional relationship between a reference marker present in the real environment and a display device that overlays an image onto an image of the real environment or the real environment itself; a reception unit that receives instructions for operations on the 3D model of the virtual robot, instructions for registering the position or position and orientation of the end effector of the virtual robot, and instructions for moving the virtual robot; and a unit that displays the 3D model of the virtual robot, which is arranged to have a predetermined positional relationship with the reference marker based on the relative positional relationship with the 3D model of the virtual robot, in accordance with the received operations. The system comprises an image generation unit that generates a display image, an output unit that outputs the display image to a display device, a registration unit that registers the position or position and orientation of the end effector in the 3D model of the virtual robot when a registration instruction is received, and a robot position relationship acquisition unit that acquires a new positional relationship between a reference marker and the virtual robot in response to the receipt of a movement instruction. The image generation unit generates a display image for showing the 3D model of the virtual robot, which is arranged to have a new positional relationship with the reference marker, and in which the end effector is at the registered position or position and orientation, when a new positional relationship is acquired by the robot position relationship acquisition unit. [Effects of the Invention]

[0007] According to one aspect of the present invention, an image generation device allows users to view a display image of a virtual robot with a changed placement position alongside the actual environment, while keeping the position or position and orientation of the end effector in the 3D model of the virtual robot fixed. Therefore, robot placement studies can be realized at low cost and with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] A schematic diagram showing the configuration of an information processing system according to an embodiment of the present invention. [Figure 2]Flowchart showing the operation of the image generation device according to this embodiment. [Figure 3] Flowchart showing the operation of the image generation device according to this embodiment. [Figure 4] This figure shows the virtual robot before movement in the same embodiment. [Figure 5] This figure shows the virtual robot after movement in the same embodiment. [Figure 6] This figure shows multiple candidate destinations for the virtual robot in the same embodiment. [Modes for carrying out the invention]

[0009] The image generation apparatus according to the present invention will be described below using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and their repeated explanation may be omitted. The image generation apparatus according to this embodiment registers the position and orientation of the end effector of a virtual robot, and generates a display image that displays the 3D model of the virtual robot, which is the position and orientation of the end effector registered, at a new placement position.

[0010] Figure 1 is a schematic diagram showing the configuration of the information processing system 100 according to this embodiment. The information processing system 100 according to this embodiment comprises an image generation device 3 and a display device 4. The image generation device 3 and the display device 4 may be connected, for example, by wire or wireless.

[0011] The virtual robot to be operated is composed of a 3D model existing in the virtual environment and corresponds to the actual robot. That is, the virtual robot is the same as the actual robot except that it is composed of a 3D model. For example, it may have the same size and configuration as the actual robot, and it may be possible to change the angles of the joints of multiple arms, just like the actual robot. An end effector is attached to the tip of the virtual robot. The end effector is the same as the end effector in the actual environment except that it is composed of a 3D model. For example, it may have the same size and configuration as the end effector in the actual environment, and if the end effector in the actual environment has movable parts, the virtual robot may also have movable parts, just like the end effector in the actual environment. The end effector may be, for example, a welding torch, a hand with a gripping part for grasping an object to be transported, a hand on which an object to be transported is placed, or a tool with functions such as assembly or painting.

[0012] A real robot is typically an industrial robot, and may be a manipulator having multiple arms (links) connected by motor-driven joints. A real robot may be, for example, a vertical articulated robot or a horizontal articulated robot. Furthermore, a real robot may be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. A real robot, as an example, is a robot existing in a real environment. A real environment refers to the environment of real space.

[0013] The image generation device 3 registers the position and orientation of the virtual robot's end effector, and upon receiving a movement instruction for the virtual robot, acquires a new positional relationship between the reference marker 6 and the virtual robot. It then generates a display image for showing a 3D model of the virtual robot positioned to achieve this new positional relationship with the reference marker 6, with the end effector at the registered position and orientation. In this way, the placement of the virtual robot can be considered while fixing the position and orientation of the end effector. For example, if interference occurs with peripheral equipment when the virtual robot's end effector is in a desired position and orientation, the placement of the virtual robot can be considered without interference while fixing the position and orientation of the end effector. Details of the image generation device 3 will be described later.

[0014] The reference marker 6 is a predetermined two-dimensional image. The reference marker 6 may be, for example, an AR marker, a QR code (registered trademark), or any other two-dimensional image with a predetermined shape. The size of the reference marker 6 may be predetermined, for example. This embodiment mainly describes the case where the reference marker 6 is displayed on a sheet 6a. The reference marker 6 may be printed on a sheet 6a made of paper or resin, for example.

[0015] The predetermined positional relationship between the reference marker 6 and the 3D model of the virtual robot may be, for example, a predetermined positional relationship, or a positional relationship that can be changed by the operator operating the virtual robot. The predetermined positional relationship may be, for example, a positional relationship in which the 3D model of the virtual robot is displayed at the position of the reference marker 6, or a positional relationship in which the 3D model of the virtual robot is displayed at a position different from the reference marker 6. In the former case, the image generation device 3 may place the 3D model of the virtual robot at the position of the reference marker 6. Placing the 3D model of the virtual robot at the position of the reference marker 6 means that the 3D model of the virtual robot is placed in the virtual space in such a way that the situation in which a real robot is placed at the position of the reference marker 6 is virtually reproduced. For example, the 3D model of the virtual robot may be placed so that the end face on the base end side of the 3D model of the virtual robot (for example, the mounting surface to the floor, etc.) coincides with the surface of the reference marker 6. Placing the 3D model of the virtual robot at a position different from the reference marker 6 means, for example, that the 3D model of the virtual robot is placed next to the reference marker 6.

[0016] The display device 4 displays an image superimposed on an image of the real environment or the real environment itself. That is, the operator operating the virtual robot can see both the image of the real environment and the image of the virtual environment through the display device 4. The display device 4 may be a wearable display device worn on the head of the operator operating the virtual robot, or it may be a portable information processing terminal such as a tablet device. The wearable display device may be, for example, a head-mounted display. The display device 4 may also have, for example, a transparent display. In this case, the display device 4 will display the image superimposed on the real environment itself. Examples of wearable display devices 4 with transparent displays include HoloLens®. Such a display device 4 with a transparent display can also be considered a display device for realizing mixed reality (MR). The display device 4 may also have, for example, an opaque display. In this case, the display device 4 will display the image superimposed on an image of the real environment. Therefore, it is preferable that the display device 4 with an opaque display has a camera for capturing the real environment, or is connected to a camera for capturing the real environment. Images of the real environment captured by the camera are displayed in real time on an opaque display. Examples of wearable display devices 4 having an opaque display include the Oculus Quest. Such a display device 4 having an opaque display can also be considered a display device for realizing augmented reality (AR). A portable information processing terminal such as a tablet device may have, for example, a camera and a display, and may display images of the real environment captured by the camera on the display in real time. In this embodiment, the case in which the display device 4 is a head-mounted display having a transparent display will be mainly described.

[0017] As shown in FIG. 1, the image generation device 3 includes a storage unit 31, a marker position relationship acquisition unit 32, a reception unit 33, a registration unit 34, a robot position relationship acquisition unit 35, an image generation unit 36, and an output unit 37.

[0018] In the storage unit 31, a three-dimensional model of a virtual robot is stored. The three-dimensional model of the virtual robot may correspond to, for example, a real robot being considered for introduction. When there are multiple candidates for introduction, for example, three-dimensional models of multiple virtual robots may be stored in the storage unit 31. In this case, the display image may be generated while appropriately switching the three-dimensional models of the multiple virtual robots. It is assumed that an end effector is attached to the end of the three-dimensional model of the virtual robot. Also, information other than the three-dimensional model may be stored in the storage unit 31. For example, teaching data or the like may be stored in the storage unit 31.

[0019] The process by which information is stored in the storage unit 31 is not limited. For example, information may be stored in the storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage unit 31. The storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0020] The marker position relationship acquisition unit 32 acquires the relative position relationship between the reference marker 6 existing in the real environment and the display device 4. To acquire the relative position relationship between the reference marker 6 and the display device 4 may, for example, be to acquire the relative position relationship between the marker coordinate system which is the local coordinate system of the reference marker 6 and the display coordinate system which is the local coordinate system of the display device 4. This relative position relationship may, for example, be represented by a homogeneous transformation matrix indicating the transformation between the two coordinate systems. It does not matter how the marker position relationship acquisition unit 32 acquires this relative position relationship. The marker position relationship acquisition unit 32 may, for example, receive an image captured by the camera of the display device 4 and use three or more feature points of the reference marker 6 included in the image to acquire a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system. The acquisition of the homogeneous transformation matrix may be performed by the display device 4. In this case, the marker position relationship acquisition unit 32 may receive from the display device 4 a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system. That is, the acquisition of the relative position relationship by the marker position relationship acquisition unit 32 may be the reception of the relative position relationship.

[0021] The reception unit 33 receives operations on the three-dimensional model of the virtual robot. The reception unit 33 also receives a registration instruction for the position and orientation of the end effector of the virtual robot. The registration instruction may, for example, be received when the end effector is in a desired position or orientation in the display image of the three-dimensional model of the virtual robot. The reception unit 33 also receives a movement instruction for the virtual robot. The movement instruction may, for example, be input to eliminate interference with peripheral devices and the like that occurs when the end effector is in the registered position and orientation at the current arranged position of the virtual robot, or may be input according to other reasons.

[0022] Operations on the 3D model may include, for example, operations to change the position or orientation of at least a part of the 3D model. Instructions for operation may be received, for example, via an input device such as a teaching pendant present in the real environment, or via a virtual input interface such as a virtual button or virtual teaching pendant displayed on the display of the display device 4. The virtual input interface may be displayed on the display of the display device 4 in response to an action such as an air tap, and when a button or the like is selected by the operator's finger or pointing device, input corresponding to the operation of that button or the like may be passed from the display device 4 to the reception unit 33. In addition, the position or orientation of at least a part of the 3D model of the virtual robot (for example, an end effector or other end-effector) may be changed by a gesture of the operator's hand. In this case, for example, by the operator performing a pinching motion (holding motion) of the 3D model displayed on the display, the part pinched by the hand is identified as the target of operation, and by changing the position or orientation of the hand, an operation is performed to change the position or orientation of the identified target, and by ending the pinching motion, the operation on the identified target may be terminated. In this case, for example, information indicating the object of the operation (e.g., information indicating the position or part in the 3D model) and information indicating the content of the operation (e.g., information indicating a change in position or posture) may be passed from the display device 4 to the reception unit 33, or the results of hand tracking acquired by the display device 4 may be passed to the reception unit 33, and the object of the operation and the content of the operation may be identified in the image generation device 3. When information indicating the object of the operation and information indicating the content of the operation are acquired by the display device 4, the display device 4 may be able to access the information of the current 3D model of the virtual robot in the virtual space, which is held in the image generation device 3. Furthermore, when the operation is performed using the operator's hands, the display device 4 may have a camera, and the position of the operator's hands on the display may be identified by performing hand tracking of the operator's hands captured by the camera.Furthermore, methods for converting operations on a 3D model's display image in response to a worker's hand gestures into changes in the position and orientation of the 3D model in a 3D virtual space are already publicly known, and a detailed explanation of these methods will be omitted.

[0023] The registration instructions for the position and orientation of the virtual robot's end effector may be received, for example, via an input device in the real environment, or via a virtual input interface displayed on the display of the display device 4. Similarly, the movement instructions for the virtual robot may be received, for example, via an input device in the real environment, a virtual input interface displayed on the display of the display device 4, or by moving the 3D model of the virtual robot through gesture operation. The movement instructions for the virtual robot received via an input device or virtual input interface may, for example, indicate the direction and degree of movement numerically. In this case, for example, numerical values ​​corresponding to a vector indicating the direction and degree of movement may be received. The movement instructions for the virtual robot received through gesture operation may include, for example, an action of pinching the base end portion of the 3D model of the virtual robot with a hand, an action of changing the position of that hand, and an action of ending the pinching action. The received movement instruction is not particularly limited as long as it provides information indicating how the virtual robot's position will change as a result. For example, it may be a homogeneous transformation matrix showing the transformation between the robot's coordinate system before movement and the robot's coordinate system after movement, or it may be other information indicating the details of the movement.

[0024] The reception unit 33 may accept information or instructions other than those described above. For example, the reception unit 33 may accept information input from an input device or display device 4, or it may receive information transmitted via a wired or wireless communication line. The reception unit 33 may or may not include a device for receiving information (for example, an input device or a communication device). Furthermore, the reception unit 33 may be implemented by hardware, or by software such as a driver that drives a predetermined device.

[0025] The registration unit 34 registers the position and orientation of the end effector in the 3D model of the virtual robot when the registration instruction is received by the reception unit 33. The position and orientation of the end effector may be, for example, the position and orientation of a part of the end effector. For example, if the end effector is a welding torch, the position and orientation of the end effector may be the position of the tip of the welding torch and the orientation of the tip portion of the welding torch. Alternatively, the position of the end effector may be, for example, the position of the TCP (Tool Center Point) of the virtual robot. The position and orientation of the end effector may also be, for example, the position and orientation in the world coordinate system. The world coordinate system may, for example, be the marker coordinate system. The registration unit 34 can, for example, obtain the position and orientation of the end effector in the robot coordinate system from the image generation unit 36. The positional relationship between the robot coordinate system and a world coordinate system such as the marker coordinate system may be, for example, a predetermined positional relationship or a positional relationship obtained by the robot positional relationship acquisition unit 35. Therefore, the registration unit 34 can use this information to identify the position and orientation of the end effector in a world coordinate system such as a marker coordinate system. The position of the end effector may be indicated, for example, by coordinate values ​​in a predetermined coordinate system. The orientation of the end effector may be indicated, for example, by roll, pitch, yaw angle, or Euler angle. Furthermore, the position and orientation information registered by the registration unit 34 may be any information that can identify the position and orientation of the end effector. Registering the position and orientation of the end effector may, for example, mean storing information indicating the position and orientation in a storage unit. This storage unit may be, for example, the storage unit 31, or it may be another recording medium. In this embodiment, the latter case will be mainly described.

[0026] The robot position relationship acquisition unit 35 acquires a new position relationship between the reference marker 6 and the virtual robot in response to the receipt of a movement instruction by the reception unit 33. This new position relationship between the reference marker 6 and the virtual robot may, for example, be information indicating the virtual robot's placement position after movement in the world coordinate system in response to the movement instruction. The position relationship of the virtual robot before and after movement (for example, a homogeneous transformation matrix relating to the robot coordinate systems of the virtual robot before and after movement) can be determined based on the movement instruction. Furthermore, the position of the virtual robot's robot coordinate system before movement in the world coordinate system, such as the marker coordinate system, is known. Therefore, using this information, the robot position relationship acquisition unit 35 can determine the position of the virtual robot's robot coordinate system after movement in the world coordinate system, such as the marker coordinate system. In other words, the robot position relationship acquisition unit 35 can acquire a new position relationship between the reference marker 6 and the virtual robot. This new position relationship may, as an example, be a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the virtual robot's robot coordinate system at the placement position after movement.

[0027] The image generation unit 36 ​​generates a display image for displaying the 3D model of the virtual robot, which is positioned to have a predetermined positional relationship with the reference marker 6, based on the 3D model of the virtual robot stored in the memory unit 31 and the relative positional relationship acquired by the marker positional relationship acquisition unit 32, in response to an operation received by the reception unit 33. Furthermore, when a new positional relationship is acquired by the robot positional relationship acquisition unit 35, the image generation unit 36 ​​generates a display image for displaying the 3D model of the virtual robot, which is positioned to have a new positional relationship with the reference marker 6, and whose end effector is registered in the position and orientation of the virtual robot.

[0028] The relative positional relationship between the reference marker 6 and the display device 4 is acquired by the marker positional relationship acquisition unit 32. Furthermore, the positional relationship between the reference marker 6 and the 3D model of the virtual robot before movement, i.e., the initial positional relationship, is predetermined, and the positional relationship after movement in response to a movement instruction, i.e., the new positional relationship, is acquired by the robot positional relationship acquisition unit 35. Therefore, using this information, the image generation unit 36 ​​can determine the relative positional relationship between the 3D model of the virtual robot and the display device 4. Thus, the image generation unit 36 ​​can position the 3D model of the virtual robot in the virtual space and determine the position and orientation of the display device 4 relative to that 3D model, either the initial positional relationship or the new positional relationship. Then, the image generation unit 36 ​​can generate a 2D display image for displaying the 3D model by rendering the 3D model of the virtual robot in the virtual space based on the position and orientation of the display device 4. Note that the angles of each joint of the virtual robot are at their initial values ​​if no operation is performed, and at their post-operation values ​​if an operation is performed. Furthermore, when generating a display image of the virtual robot where the end effector is located at a registered position and orientation, the angles of each joint will correspond to the registered position and orientation of the end effector. The position and orientation of the end effector registered by the registration unit 34 are, for example, the position and orientation in a world coordinate system such as the marker coordinate system. The new placement position of the virtual robot in a world coordinate system such as the marker coordinate system can be determined by the new positional relationship acquired by the robot positional relationship acquisition unit 35. Therefore, the image generation unit 36 ​​can determine the position and orientation of the end effector in the local coordinate system (e.g., the robot coordinate system) of the new placement position of the virtual robot, and acquire the corresponding angles of each joint. The angles of each joint after operation, and the angles of each joint corresponding to the registered position and orientation of the end effector, may be calculated, for example, by inverse kinematics using the position and orientation of the end effector in the 3D model of the virtual robot, similar to a real robot. When the virtual robot is operated, the shape of the 3D model in the virtual space will change accordingly.Furthermore, if the position or orientation of the display device 4 changes in the real environment, the position and orientation of the viewpoint in the virtual space will change accordingly. Then, rendering is performed after this change, generating display images of the 3D model after the operation and display images of the 3D model after the change in the position and orientation of the display device 4. The image generation unit 36 ​​generates the display image so that when the display image of the 3D model of the virtual robot is displayed on the display device 4, the size of the display image matches that of the real environment. In other words, the display image is generated so that the 3D model of the virtual robot displayed on the display device 4 and the real robot placed in the real environment to have the same relative positional relationship as the 3D model appear to be the same size when viewed through the display device 4.

[0029] Furthermore, when a new positional relationship is acquired by the robot positional relationship acquisition unit 35, the image generation unit 36 ​​generates a display image corresponding to the new positional relationship if it can calculate the angles of each joint by inverse kinematics while the end effector of the virtual robot, which is positioned to have the new positional relationship with the reference marker 6, is in the registered position and orientation. Otherwise, it does not need to generate a display image corresponding to the new positional relationship. The inability to calculate the angles of each joint of the virtual robot by inverse kinematics may, for example, mean that there is no solution in the inverse kinematics calculation, or that there is a solution in the inverse kinematics calculation, but at least one of the solutions exceeds the pre-set operating range of each joint. The operating range of each joint may be set, for example, to prevent interference with the housing of the virtual robot, or to prevent cable breakage or avoid singularities.

[0030] Furthermore, it may not be possible to operate the virtual robot in response to the accepted operation. For example, an operation to move the virtual robot's end effector beyond its movable range or rotate it beyond its rotatable range may be accepted. In such cases, the image generation unit 36 ​​does not need to generate a display image corresponding to the operation, or it may generate a display image of the 3D model of the virtual robot, etc., that has been moved or rotated within the possible range.

[0031] The virtual robot will be displayed in a predetermined positional relationship (for example, the initial positional relationship or the acquired positional relationship) with the reference marker 6 placed in the real environment. Therefore, even if the orientation of the display device 4 changes, the display position of the 3D model of the virtual robot in the real environment will not change. In addition, the display image generated by the image generation unit 36 ​​may be, for example, an opaque image that does not allow the background image to be seen, or a semi-transparent image that allows the background image to be seen.

[0032] The output unit 37 outputs the display image generated by the image generation unit 36 ​​to the display device 4. When outputting the display image, the output unit 37 may output only the display image. In this case, the display image will be superimposed on the image of the real environment or the real environment itself on the display device 4. On the other hand, if the display device 4 has an opaque display and the image of the real environment captured by the display device 4 is received by the image generation device 3, the result of combining the image of the real environment and the display image may be output to the display device 4. This combination may be performed, for example, by a combination unit (not shown) of the image generation device 3.

[0033] If the 3D model of the virtual robot is displayed at a position different from the reference marker 6, for example, the 3D model 10 of the virtual robot may be displayed next to the reference marker 6, as shown in Figure 1. The reference marker 6 and the 3D model 10 of the virtual robot in Figure 1 schematically represent the situation as seen by the operator of the virtual robot via the display device 4.

[0034] Furthermore, the display of a 3D model of a virtual robot, and the manipulation of that 3D model using a virtual input interface on a display device or operator gestures, are already publicly known, as shown in references 1-3 below, for example, and therefore a detailed explanation will be omitted. Reference 1: Japanese Patent Publication No. 2017-100234 Reference 2: Japanese Patent Publication No. 2020-055075 Reference 3: Japanese Patent Publication No. 2020-069538

[0035] Furthermore, the output unit 37 may output a message indicating that the virtual robot cannot move to the new positional relationship if the angles of each joint cannot be calculated by inverse kinematics while the end effector of the virtual robot, which is positioned to have a new positional relationship with the reference marker 6, is in the registered position and orientation. Note that the output indicating that the virtual robot cannot move to the new positional relationship may also be, for example, an output indicating that the virtual robot cannot move to the new placement position corresponding to the new positional relationship. This output may be, for example, an output of an image or sound to the display device 4, a display to another display device, a transmission via a communication line to a predetermined device, a printout by a printer, an audio output by a speaker, storage on a recording medium, or a transfer to another component. Note that the output unit 37 may or may not include a device that performs the output (for example, a display device or a speaker). Furthermore, the output unit 37 may be implemented by hardware, or by software such as a driver that drives those devices.

[0036] Next, the operation of the image generation device 3 will be explained using the flowchart in Figure 2. (Step S101) The image generation unit 36 ​​generates a display image of the 3D model of the virtual robot, and the output unit 37 outputs the generated display image to the display device 4. The reception unit 33 may also receive operations on the end effector of the displayed virtual robot. When an operation is received, a display image of the 3D model of the virtual robot with the position and orientation of the end effector changed according to the operation may be generated and output. Details of this process will be described later using the flowchart in Figure 3.

[0037] (Step S102) The reception unit 33 determines whether it has received the registration instruction. If it has received the registration instruction, it proceeds to step S103; otherwise, it returns to step S101.

[0038] (Step S103) The registration unit 34 registers the position and orientation of the end effector in the 3D model of the virtual robot at that time.

[0039] (Step S104) The reception unit 33 determines whether it has received a movement instruction for the virtual robot. If it has received a movement instruction, it proceeds to step S105; otherwise, it repeats the process in step S104 until it receives a movement instruction.

[0040] (Step S105) The robot position relationship acquisition unit 35 acquires the new position relationship between the reference marker 6 and the virtual robot after movement in response to the receipt of the movement instruction.

[0041] (Step S106) The image generation unit 36 ​​determines whether to generate a display image corresponding to the new positional relationship. For example, if the end effector of the virtual robot, which is positioned to have a new positional relationship with the reference marker 6, is in the registered position and orientation, and the angles of each joint can be calculated by inverse kinematics, the image generation unit 36 ​​may decide to generate a display image corresponding to the new positional relationship and proceed to step S107. Otherwise, it may decide not to generate a display image corresponding to the new positional relationship and return to step S104. When returning to step S104, the output unit 37 may output, for example, that the virtual robot cannot be moved to the new positional relationship.

[0042] (Step S107) The image generation unit 36 ​​generates a display image for showing a 3D model of a virtual robot that is positioned to have the new positional relationship with the reference marker 6 acquired in step S105, and whose end effector is at the registered position and orientation. Details of this process will be described later using the flowchart in Figure 3.

[0043] (Step S108) The reception unit 33 determines whether it has received an instruction to terminate the series of processes. If it has received an instruction to terminate, it returns to step S101; otherwise, it proceeds to step S109.

[0044] (Step S109) The reception unit 33 determines whether it has received a movement instruction for the virtual robot. If it has received a movement instruction, it returns to step S105; otherwise, it returns to step S107.

[0045] Note that while the flowchart in Figure 2 shows a case where the position and orientation of the end effector can only be registered at the initial position of the virtual robot, it may also be possible to register the position and orientation of the end effector at the placement position after the virtual robot has moved. Also, the order of processing in the flowchart in Figure 2 is just one example, and the order of each step may be changed if the same result can be obtained. Furthermore, in the flowchart in Figure 2, processing is terminated by power off or processing termination interrupt.

[0046] Next, using the flowchart in Figure 3, we will explain the process of generating the display image of the virtual robot in the flowchart in Figure 2 (steps S101, S107). In step S101, the object of operation and display is the 3D model of the virtual robot positioned to have the initial positional relationship with the reference marker 6, and in step S107, the object of operation and display is the 3D model of the virtual robot positioned to have a new positional relationship with the reference marker 6. However, in Figure 3, we will not particularly distinguish between the two.

[0047] (Step S201) The reception unit 33 determines whether it has received an operation request for the 3D model. If it has received the operation request, it proceeds to step S203; otherwise, it proceeds to step S202.

[0048] (Step S202) The image generation unit 36 ​​determines whether to generate a display image. If it decides to generate a display image, it proceeds to step S203; otherwise, it returns to the flowchart in Figure 2. The image generation unit 36 ​​may, for example, periodically decide whether to generate a display image. This decision ensures that, for example, even if no operation is performed, if the position or orientation of the display device 4 is changed, the display image corresponding to the changed position or orientation will be displayed on the display device 4.

[0049] (Step S203) The marker position relationship acquisition unit 32 acquires the relative position relationship between the reference marker 6 and the display device 4.

[0050] (Step S204) The image generation unit 36 ​​generates a display image for displaying the 3D model of the virtual robot, using the 3D model of the virtual robot stored in the memory unit 31 and the relative positional relationship acquired in step S203. If an operation is received, a display image for displaying the 3D model after it has been modified according to that operation will be generated. When generating a display image in step S107, a display image for displaying the 3D model of the virtual robot at the registered position and orientation of the end effector will be generated. If the position or orientation of the display device 4 is changed, a display image of the 3D model corresponding to the changed position and orientation of the display device 4 will be generated. The display image for displaying the 3D model of the virtual robot will be generated so as to have the initial positional relationship or a new positional relationship with the reference marker 6.

[0051] (Step S205) The output unit 37 outputs the generated display image to the display device 4. Then, the process returns to the flowchart in Figure 2. Depending on this output, for example, a 3D model of the virtual robot will be displayed.

[0052] Next, the operation of the image generation device 3 according to this embodiment will be explained using a specific example. First, the operator places the sheet 6a with the reference marker 6 displayed on it at the position where the actual robot to be introduced will be placed. Next, the operator puts the head-mounted display device 4 on their head and starts the processing of the image generation device 3. The image generation device 3 then generates a display image of the virtual robot and outputs it to the display device 4 (step S101). Specifically, the image generation unit 36 ​​determines that it is time to generate a display image and gives an instruction to the marker position relationship acquisition unit 32 to acquire the relative position relationship between the reference marker 6 and the display device 4 (step S202). In response to this instruction, the marker position relationship acquisition unit 32 acquires the relative position relationship and passes it to the image generation unit 36 ​​(step S203). Upon receiving the relative positional relationship, the image generation unit 36 ​​uses that relative positional relationship and the 3D model of the virtual robot stored in the memory unit 31 to generate a display image for displaying the 3D model at a predetermined initial position and initial orientation, and passes it to the output unit 37 (step S204). The output unit 37 outputs the received display image to the display device 4 (step S205). As a result, the operator can see the virtual robot superimposed on the real space.

[0053] Subsequently, the operator moves the tip of the welding torch, which is the displayed end effector, to the desired position and orientation using input devices such as a teaching pendant or gesture operations, as shown in Figure 4 (steps S201, S203-S205). Note that in Figure 4, the display of workpieces and peripheral equipment placed in the actual environment other than the reference marker 6 and the 3D model 10 of the virtual robot is omitted. The same applies to Figure 5, which will be described later. At this point, when the operator inputs a registration instruction via an input device or virtual input interface, the input is received by the reception unit 33 and passed to the registration unit 34 (step S102). The registration unit 34 then acquires the position and orientation of the end effector at that time and stores it in a recording medium (not shown) (step S103).

[0054] Next, the operator inputs movement instructions for the displayed virtual robot using an input device, a virtual input interface, or gesture control. These movement instructions may be input, for example, to resolve interference when the 3D model 10 of the virtual robot, whose end effector is in a desired position and orientation, is interfering with peripheral equipment. The movement instructions are received by the reception unit 33 and passed to the robot position relationship acquisition unit 35 (step S104). Upon receiving the movement instructions, the robot position relationship acquisition unit 35 acquires the new positional relationship between the reference marker 6 and the virtual robot and passes it to the image generation unit 36 ​​(step S105).

[0055] Upon receiving the new positional relationship between the reference marker 6 and the virtual robot, the image generation unit 36 ​​obtains the registered position and orientation of the end effector from the registration unit 34 and determines whether the angles of each joint of the virtual robot can be calculated by inverse kinematics when the end effector of the virtual robot, positioned to match the new positional relationship between the reference marker 6 and the virtual robot, is in the registered position and orientation (step S106). In this case, it is assumed that the angles of each joint can be calculated by inverse kinematics. Then, the image generation unit 36 ​​generates a display image for displaying the 3D model of the virtual robot, which is positioned to match the new positional relationship between the reference marker 6 and the virtual robot, and the position and orientation of the end effector are in the registered position and orientation. The output unit 37 then outputs the display image to the display device 4 (steps S107, S202~S205). As a result, as shown in Figure 5, the operator can see a display image of the 3D model 10 of the virtual robot, which is positioned at the registered location and orientation of the 3D model 10a of the end effector, and at the location after the move. For example, they can confirm in the real environment what the situation would be like when the end effector of the moved virtual robot is at the desired position and orientation. They can also check, for example, whether the virtual robot interferes with peripheral equipment, etc., placed in the real environment.

[0056] Furthermore, when the operator inputs a movement instruction, if, for example, the virtual robot is moved too far, and the end effector of the virtual robot is placed in the new position, and the angles of each joint of the virtual robot cannot be calculated by inverse kinematics (step S106), the system will accept the input of a movement instruction again. In this case, the system may also output a message indicating that the robot cannot be moved to the new position, thereby informing the operator that a movement instruction exceeding the possible range has been entered.

[0057] Furthermore, the display of the 3D model 10 of the virtual robot is changed by the operator manipulating the displayed 3D model 10 of the virtual robot (steps S107, S201, S203~S205). Therefore, the operator can also confirm the operation of the virtual robot at its new placement location. In addition, when the operator inputs a new movement instruction, the instruction is received by the reception unit 33 (step S109), and the new positional relationship is acquired, and the virtual robot is displayed according to the new positional relationship (steps S105~S107). In this case as well, a display image is generated to show the 3D model of the virtual robot placed at the new position, where the position and orientation of the end effector are registered, and this image is displayed on the display device 4.

[0058] As described above, the image generation device 3 according to this embodiment can generate and output a display image of a 3D model of a virtual robot whose placement position has been changed while the position and orientation of the end effector are fixed. Therefore, for example, by changing the placement position of a virtual robot while fixing the position and orientation of the welding torch of a virtual robot, which is a welding robot, to the welding start position, it becomes possible to consider a robot placement position that does not interfere with peripheral equipment, etc., placed in the actual environment. Furthermore, since the display image of the 3D model 10 of the virtual robot can be overlaid on an image of the actual environment or the actual environment itself, highly accurate placement studies can be performed. In addition, since there is no need to prepare 3D models of workpieces and peripheral equipment in the actual environment, placement studies can be performed at a low cost. Moreover, it is possible to determine whether to generate a display image corresponding to the new positional relationship by determining whether the angles of each joint can be calculated by inverse kinematics while the end effector of the virtual robot, which is placed in a new positional relationship with the reference marker 6, is in the registered position and orientation. For example, if the virtual robot's display image is not shown at a new placement location, the operator can understand that they cannot move the virtual robot to that location, meaning that the virtual robot placed at that location cannot assume the position and orientation registered for the end effector. Furthermore, if the angles of each joint cannot be calculated using inverse kinematics, outputting a message indicating that the robot cannot move to the new location can notify the operator that they have entered a movement instruction exceeding the possible range, prompting them to enter a new movement instruction.

[0059] In this embodiment, the case where the position and orientation of one end effector are registered has been mainly described, but this is not required. Two or more positions and orientations of the end effector may be registered by the registration unit 34. For example, if the end effector is a welding torch, the position and orientation of the welding torch at the start position of welding and the position and orientation of the welding torch at the end position of welding may be registered. In this case, when two or more positions and orientations of the end effector are registered by the registration unit 34, the image generation unit 36 ​​may generate display images for each of the three-dimensional models of the virtual robot, which are arranged to have a new positional relationship with the reference marker 6 when a new positional relationship is acquired by the robot positional relationship acquisition unit 35, and which represent the two or more positions and orientations of the registered end effector.

[0060] More specifically, if a first position and orientation and a second position and orientation of the end effector are registered, the image generation unit 36 ​​may generate a display image to show a 3D model of the first virtual robot where the end effector is in the first position and orientation to which it is registered, and which is positioned to have a new positional relationship with the reference marker 6, and a 3D model of the second virtual robot where the end effector is in the second position and orientation to which it is registered, and which is positioned to have a new positional relationship with the reference marker 6. Furthermore, the image generation unit 36 ​​may generate a display image corresponding to the new positional relationship if it can calculate the angles of each joint by inverse kinematics when the end effector of the first virtual robot, positioned to have a new positional relationship with the reference marker 6, is in the first position and orientation to which it is registered, and if it can calculate the angles of each joint by inverse kinematics when the end effector of the second virtual robot, positioned to have a new positional relationship with the reference marker 6, is in the second position and orientation to which it is registered, but may not generate a display image corresponding to the new positional relationship otherwise.

[0061] In this manner, if N positions and orientations of end effectors are registered, the image generation unit 36 ​​may generate display images for displaying 3D models of N virtual robots. N is an integer of 2 or more. The 3D models of the N virtual robots are 3D models of virtual robots that have the same base end position, i.e., placement position, but different end end position and orientation. Furthermore, the image generation unit 36 ​​may generate display images of N virtual robots corresponding to the new positional relationship if it can calculate the angles of each joint by inverse kinematics when the end effectors of the virtual robots, which are positioned to have a new positional relationship with the reference marker 6, are in the registered first to N positions and orientations. Otherwise, it may not generate display images corresponding to the new positional relationship.

[0062] The process of registering the positions and orientations of the two end effectors may be carried out as follows, for example. First, the operator registers the position and orientation of the first end effector. Once this registration is complete, the 3D model of the virtual robot with the position and orientation of the first end effector registered is displayed, and a new 3D model of the virtual robot is displayed at the same placement location. Then, the operator manipulates the new 3D model of the virtual robot to register the position and orientation of the second end effector. After that, when the operator moves the placement locations of the two 3D models of the virtual robots using a movement command, the two 3D models of the virtual robots are displayed at the new placement locations accordingly. The position and orientation of the end effectors of each 3D model of the virtual robot will be the registered positions and orientations. Here, it is assumed that the angles of each joint can be calculated by inverse kinematics at the new placement locations.

[0063] Furthermore, if two or more positions and orientations of the end effector are registered, it may not be easy to find the placement position of the virtual robot that can achieve the two or more registered positions and orientations of the end effector. For example, if two or more positions and orientations of the end effector are registered by the registration unit 34, the robot position relationship acquisition unit 35 may acquire a plurality of new position relationships set at predetermined intervals. The image generation unit 36 ​​may then identify a new position relationship from among the plurality of new position relationships that allows the angle of each joint to be calculated by inverse kinematics for each state in which the virtual robot's end effector is one of the two or more registered positions and orientations. This new position relationship is the position relationship between the reference marker 6 and the virtual robot. Subsequently, the image generation unit 36 ​​may generate display images for each of the three-dimensional models of the virtual robot, which are arranged to achieve the position relationship identified with the reference marker 6, and which represent the three-dimensional models of two or more virtual robots that are in the two or more registered positions and orientations of the end effector. In this way, the end effector may automatically find the placement location of the virtual robot, which can be one of two or more registered positions and orientations, and generate display images of multiple virtual robots at that placement location. In this case, the operator can easily find the placement location of the virtual robot, which can be one of two or more registered positions and orientations.

[0064] More specifically, the robot position relationship acquisition unit 35 may, for example, set multiple new placement positions at predetermined intervals around the current placement position of the virtual robot, and acquire a new positional relationship with the reference marker 6 for each of these multiple new placement positions. For example, as shown in Figure 6, if the current placement position 30 is the position of the reference marker 6, the robot position relationship acquisition unit 35 may set multiple grid points arranged at predetermined intervals around the current placement position 30 as new placement positions 31. The multiple grid points may, for example, be arranged on the plane where the reference marker 6 is located. In Figure 6, all black circles other than the current placement position 30 are new placement positions 31. For example, the new placement positions 31 may be set in the robot coordinate system at that time. Since the relationship between the robot coordinate system and the marker coordinate system is known, the robot position relationship acquisition unit 35 can acquire multiple new positional relationships with the reference marker 6 for each of the multiple new placement positions 31. The placement positions 30 and 31 may, for example, be the locations where the center point of the end face on the base end of the 3D model of the virtual robot is located when the virtual robot is placed. Furthermore, the orientation of the virtual robot when its 3D model is placed at each placement position 30 and 31 may be determined, for example, according to the orientation of the reference marker 6, or according to other criteria.

[0065] The image generation unit 36 ​​may select one new positional relationship from among several newly acquired positions and, if the angle of each joint can be calculated by inverse kinematics for each of the two or more registered positions and orientations of the virtual robot's end effector positioned to match the reference marker 6 and the selected new positional relationship, then the selected new positional relationship may be identified as the positional relationship for generating a display image. The image generation unit 36 ​​may then generate a display image for displaying a 3D model of the virtual robot positioned to match the identified positional relationship with the reference marker 6, and for displaying 3D models of two or more virtual robots at two or more registered positions and orientations. On the other hand, if the angle of each joint cannot be calculated by inverse kinematics for each of the two or more registered positions and orientations of the virtual robot's end effector positioned to match the selected new positional relationship with the reference marker 6, the image generation unit 36 ​​may select a new positional relationship that has not been selected so far and repeat the same process. The selection of new positional relationships may be performed, for example, in a predetermined order or randomly. The image generation unit 36 ​​may, for example, generate a display image of a virtual robot positioned according to a new positional relationship when it finds one new positional relationship from which the angles of each joint can be calculated by inverse kinematics for two or more registered positions and orientations of the end effector. Alternatively, after finding multiple new positional relationships from which the angles of each joint can be calculated by inverse kinematics for two or more registered positions and orientations of the end effector, it may identify one of the multiple new positional relationships and generate a display image of a virtual robot positioned according to the identified new positional relationship. In the latter case, for example, one new positional relationship may be randomly identified from multiple new positional relationships, or a new positional relationship corresponding to the position closest to the center of gravity of the multiple placement positions corresponding to the multiple new positional relationships may be identified, or a new positional relationship corresponding to the position furthest from the tip of the end effector may be identified, or other new positional relationships may be identified.

[0066] As mentioned above, the range of motion of each joint of the virtual robot may be set, which may prevent the first axis of the virtual robot, i.e., the proximal joint, from rotating 360 degrees. For example, the range of motion of the first axis may be limited to a range from -160 degrees to 160 degrees, i.e., a total range of 320 degrees. In such cases, the image generation unit 36 ​​may calculate the inverse kinematics for K angles obtained by rotating the virtual robot by (360 / K) degrees at each placement position of the virtual robot, and if it is possible to calculate the angle of each joint using inverse kinematics at any of these angles, then the angle of each joint at that placement position can be calculated using inverse kinematics. K is an integer of 1 or more calculated by the following equation. In the following equation, ceil is the ceiling function, and M is the angle (deg) of the range of motion of the first axis of the virtual robot. K = ceil(360 / M)

[0067] For example, if M=320 degrees as described above, then K=2, and the image generation unit 36 ​​may place the virtual robot at an arbitrary angle in a certain position and perform inverse kinematics calculations. If it is not possible to calculate the angles of each joint in these calculations, it may rotate the virtual robot 180 degrees (=360 / K) at that position and perform inverse kinematics calculations again. If it is possible to calculate the angles of each joint in these calculations, then it may be possible to calculate the angles of each joint at that position using inverse kinematics. Note that the rotation of the virtual robot is performed around an axis in the direction of the normal to the plane on which the virtual robot is placed. Also, if K=1, it is not necessary to rotate the virtual robot at each position.

[0068] Furthermore, when the image generation unit 36 ​​performs inverse kinematics calculations for two or more registered positions and orientations of the end effector, it may also determine that it is not possible to calculate the angle of each joint by inverse kinematics for each of the two or more registered positions and orientations of the end effector if the end effector cannot move continuously between the two or more registered positions and orientations. The inability of the end effector to move continuously between two or more registered positions and orientations means, for example, that when the end effector moves from a registered first position and orientation to a registered second position and orientation, the joint angle, which is within the operating range of -180 degrees to 180 degrees, may change to -179 degrees, -180 degrees, and -181 degrees. A joint angle of -180 degrees to -181 degrees would mean a 359-degree rotation of the joint in a real robot, and such movement is not possible.

[0069] Furthermore, although this embodiment mainly describes the case where the virtual robot after movement is displayed in response to the receipt of a movement instruction, the virtual robot may also be displayed in real time during movement. In this case, the image generation unit 36 ​​may, for example, generate a display image of the moving virtual robot in real time even during movement when a movement instruction for the virtual robot is input by gesture operation. That is, the processing in steps S104 to S107 may be repeated from the start to the end of the movement of the virtual robot. In this case, until the movement is completed, only the processing in steps S203 to S205 may be performed in step S107. In this way, the operator can check the posture of the virtual robot even during movement.

[0070] Furthermore, although this embodiment mainly describes the case where the position and orientation of the virtual robot's end effector are registered, only the position of the virtual robot's end effector may be registered. In this case, the position and orientation of the end effector in the above description may refer to the position of the end effector. More specifically, the receiving unit 33 may receive an instruction to register the position of the virtual robot's end effector, the registration unit 34 may register the position of the end effector in the 3D model of the virtual robot, and the image generation unit 36 ​​may generate a display image for displaying the 3D model of the virtual robot, which is arranged to have a new positional relationship with the reference marker 6, and in which the end effector is at the registered position. The orientation of the end effector in the display image may be any orientation. Thus, when only the position of the end effector is registered, the orientation of the end effector is arbitrary when the end effector is at the registered position. Furthermore, in this case, if the angles of each joint can be calculated by inverse kinematics while the end effector of the virtual robot, which is positioned to have a new positional relationship with the reference marker 6, is in the registered position, a display image corresponding to that new positional relationship will be generated. Otherwise, a display image corresponding to that new positional relationship may not be generated. The state in which the end effector of the virtual robot is in the registered position means the position in which the end effector of the virtual robot is registered, and may be in any pose.

[0071] Furthermore, the registration of the position and orientation of the end effector may also be the registration of the position and orientation of a location that has a predetermined positional relationship with the end effector. However, it is assumed that there is a one-to-one relationship between the position and orientation of that location and the position and orientation of the end effector. That is, once the position and orientation of that location are determined, the position and orientation of the end effector are also uniquely determined. In this case, registering the position and orientation of that location is substantially the same as registering the position and orientation of the end effector. Therefore, it may be considered that the registration of the position and orientation of the end effector also includes the registration of such a location. In addition, the position and orientation of that location may be used to generate display images, etc. That location may be, for example, a location a predetermined distance away from the end effector in a predetermined direction. That location may be, as an example, a location on a virtual robot. The same applies when only the position of the end effector is registered.

[0072] Furthermore, although this embodiment mainly describes the case where the virtual robot is placed on the floor, i.e., on the floor, the placement position of the virtual robot is not limited. For example, the virtual robot may be a wall-mounted virtual robot placed on a wall, or a ceiling-mounted virtual robot. In these cases, the placement surface of the virtual robot may be changed to a wall or ceiling other than the floor by movement instructions.

[0073] Furthermore, in the above embodiment, each process or function may be implemented by centralized processing by a single device or a single system, or by distributed processing by multiple devices or multiple systems. For example, at least a part of the configuration of the image generation device 3 may be physically included in a device having a display, etc. Therefore, the device divisions shown in Figure 1 may be considered to be for convenience based on function rather than corresponding to physical devices.

[0074] Furthermore, in the above embodiment, each process or function may be implemented by centralized processing by a single device or a single system, or by distributed processing by multiple devices or multiple systems.

[0075] Furthermore, in the above embodiment, if two or more components included in the image generation device 3 have a communication device, an input device, etc., the two or more components may have a single physical device, or they may have separate devices.

[0076] Furthermore, in the above embodiment, each component may be configured with dedicated hardware, or, if it is a component that can be implemented by software, it may be implemented by executing a program. For example, each component can be implemented by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. During execution, the program execution unit may execute the program while accessing the storage unit or recording medium. The program may also be executed by being downloaded from a server or the like, or by being executed by reading a program recorded on a predetermined recording medium (e.g., an optical disk, magnetic disk, semiconductor memory, etc.). Furthermore, this program may be used as a program that constitutes a program product. Furthermore, the computer executing the program may be one or multiple computers. That is, centralized processing may be performed, or distributed processing may be performed.

[0077] Furthermore, the embodiments described above are illustrative examples for specifically carrying out the present invention and do not limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims rather than by the description of the embodiments, and modifications within the literal scope and equivalent meaning of the claims are intended. [Explanation of symbols]

[0078] 3 Image generation device, 4 Display device, 6 Reference marker, 31 Storage unit, 32 Marker position relationship acquisition unit, 33 Reception unit, 34 Registration unit, 35 Robot position relationship acquisition unit, 36 Image generation unit, 37 Output unit

Claims

1. A memory unit that stores a 3D model of a virtual robot corresponding to a real robot, A marker position relationship acquisition unit acquires the relative positional relationship between a reference marker present in the real environment and a display device that overlays an image onto the image of the real environment or the real environment itself. A receiving unit that receives instructions for manipulating the 3D model of the virtual robot, instructions for registering the position or position and orientation of the end effector of the virtual robot, and instructions for moving the virtual robot. An image generation unit generates a display image for displaying the 3D model of the virtual robot, which is positioned to have a predetermined positional relationship with the reference marker, based on the 3D model of the virtual robot and the relative positional relationship, in response to an received operation. An output unit that outputs the display image to the display device, A registration unit that registers the position or position and orientation of the end effector in the 3D model of the virtual robot when a registration instruction is received, The system includes a robot position relationship acquisition unit that acquires a new position relationship between the reference marker and the virtual robot in response to the receipt of a movement instruction, The image generation unit is an image generation device that, when a new positional relationship is acquired by the robot positional relationship acquisition unit, generates a display image for displaying a 3D model of a virtual robot that is arranged to have a new positional relationship with the reference marker, and whose end effector is at the registered position or position and orientation.

2. The image generation device according to claim 1, wherein when a new positional relationship is acquired by the robot positional relationship acquisition unit, the image generation unit generates a display image corresponding to the new positional relationship if the angle of each joint can be calculated by inverse kinematics while the end effector of the virtual robot, which is positioned to have a new positional relationship with the reference marker, is at the registered position or position and orientation, and does not generate a display image corresponding to the new positional relationship otherwise.

3. The image generation device according to claim 2, wherein the output unit outputs a statement indicating that it cannot move to the new positional relationship when the end effector of the virtual robot, which is positioned to have a new positional relationship with the reference marker, is at the registered position or position and orientation and the angles of each joint cannot be calculated by inverse kinematics.

4. The image generation device according to claim 1, wherein the image generation unit generates a display image for displaying a 3D model of a virtual robot arranged to have a new positional relationship with the reference marker, where the 3D model of the virtual robot is at the two or more registered positions or positions and orientations of the end effector when the registration unit registers two or more positions or positions and orientations of the end effector and the robot positional relationship acquisition unit acquires a new positional relationship.

5. The robot position relationship acquisition unit acquires a plurality of new position relationships set at predetermined intervals when two or more positions or positions and orientations of the end effector are registered by the registration unit. The image generation device according to claim 2 or 3, wherein the image generation unit identifies a new positional relationship from among the plurality of new positional relationships, in which the angle of each joint can be calculated by inverse kinematics for each state in which the end effector of the virtual robot is registered at two or more positions or positions and postures, and generates a display image for displaying a 3D model of the virtual robot arranged to have the identified positional relationship with the reference marker, and the 3D models of two or more virtual robots being at two or more positions or positions and postures in which the end effector is registered.

Images