Image generation device

Abstract

To provide an image generating device that generates a display image for displaying a reachable range of a tool in a predetermined attitude.SOLUTION: An image generating device 3 is provided with: a positional relation obtaining part 32 that obtains a relative positional relation between a reference marker 6 existing in a real environment and a display device 4; an attitude obtaining part 33 that obtains a target attitude concerning a virtual tool; an identifying part 34 that identifies a reachable range of a virtual tool in the target attitude attached to a virtual robot arranged to have a predetermined positional relation with the reference marker, on the basis of a three-dimensional model of the virtual robot attached with the virtual tool; an image generating part 36 that generates a display image for displaying the reachable range identified by the identifying part 34, on the basis of the relative positional relation; and an output part 37 that outputs the display image to the display device 4. Such configuration enables the reachable range of the virtual tool in the target attitude to be displayed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image generation device that generates an image for displaying a reachable range of a virtual tool in a target posture.

Background Art

[0002] Conventionally, when introducing a robot, whether a tool attached to the robot can reach a desired position has been determined by referring to catalog specifications. Also, a virtual point has been created in a simulator, and the range of postures of the robot that can approach the virtual point has been calculated (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when the tool attached to the robot is in a predetermined posture, there is a problem that it is difficult to determine the reachable range of the tool based on the catalog. Also, usually, what is described in the catalog is the reachable position of the robot alone without the tool, and there is also a problem that it is difficult to determine from the information regarding the robot alone the range where the tool attached to the robot can reach in a desired posture.

[0005] In addition, in Patent Document 1 above, considering the conditions for picking an object to be worked on, the range of postures of the robot that can approach the virtual point has been calculated, and regarding the range where a tool in a predetermined posture can reach, it was not possible to know at the time of considering the arrangement of the robot.

[0006] The present invention has been made in response to the above circumstances and aims to provide an image generation device that generates a display image for indicating the range that a tool in a predetermined posture can reach. [Means for solving the problem]

[0007] To achieve the above objective, an image generation device according to one aspect of the present invention includes: a storage unit that stores three-dimensional models of a virtual robot and a virtual tool corresponding to a real robot and a real tool; a position relationship acquisition unit that acquires the relative positional relationship between a reference marker present in the real environment and a display device that overlays an image onto an image of the real environment or the real environment itself; a posture acquisition unit that acquires a target posture for a virtual tool; a specification unit that identifies the reachable range of a virtual tool in a target posture, which is mounted on a virtual robot positioned to have a predetermined positional relationship with a reference marker, based on a three-dimensional model of a virtual robot to which the virtual tool is mounted; an image generation unit that generates a display image for displaying the reachable range identified by the specification unit based on the relative positional relationship; and an output unit that outputs the display image to a display device. [Effects of the Invention]

[0008] According to one aspect of the present invention, an image generation device can generate a display image that shows the range that a tool in a predetermined posture can reach. Therefore, by looking at the display image, it becomes easy to know, for example, what range a tool in a predetermined posture can reach when considering the placement of a robot. [Brief explanation of the drawing]

[0009] [Figure 1] A schematic diagram showing the configuration of an information processing system according to an embodiment of the present invention. [Figure 2] Flowchart showing the operation of the image generation device according to this embodiment. [Figure 3] Flowchart showing the operation of the image generation device according to this embodiment. [Figure 4] Figure showing an example of multiple grid points in the same embodiment. [Figure 5] This figure shows an example of how the reachable range is displayed in the same embodiment. [Figure 6] Figure showing an example of multiple grid points in the same embodiment. [Modes for carrying out the invention]

[0010] The image generation apparatus according to the present invention will be described below using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and their repeated descriptions may be omitted. The image generation apparatus according to this embodiment generates a display image for showing the range that a virtual tool of a target posture can reach.

[0011] Figure 1 is a schematic diagram showing the configuration of the information processing system 100 according to this embodiment. The information processing system 100 according to this embodiment comprises an image generation device 3 and a display device 4. The image generation device 3 and the display device 4 may be connected, for example, by wire or wireless.

[0012] The virtual robot to be operated is composed of a 3D model existing in a virtual environment and corresponds to a real robot. That is, the virtual robot is the same as the real robot except that it is composed of a 3D model. For example, it may have the same size and configuration as the real robot, and it may be possible to change the angles of the joints of multiple arms, just like the real robot. A virtual tool may be attached to the tip of the virtual robot. The virtual tool is the same as the real tool except that it is composed of a 3D model. For example, it may have the same size and configuration as the real tool, and if the real tool has movable parts, the virtual tool may also have movable parts, just like the real tool. The real tool and virtual tool may be, for example, a welding torch, a hand with a gripping part for grasping an object to be transported, a hand on which an object to be transported is placed, or a tool that has functions such as assembly or painting.

[0013] The actual robot is typically an industrial robot and may be a manipulator having multiple arms (links) connected by motor-driven joints. The actual robot may be, for example, a vertical articulated robot or a horizontal articulated robot. The actual robot with the actual tool attached may be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. The actual robot and actual tool are, as an example, robots and tools that exist in a real environment. The real environment refers to the environment of real space.

[0014] The image generation device 3 identifies the reachable range of a virtual tool in a desired orientation, which is mounted on a virtual robot positioned in a predetermined positional relationship with the reference marker 6. It then generates a display image to show the identified reachable range and outputs it to the display device 4. Alternatively, the image generation device 3 may generate a display image to show a 3D model of the virtual robot with the virtual tool mounted, which is positioned in a predetermined positional relationship with the reference marker 6, and output it to the display device 4. Details of the image generation device 3 will be described later.

[0015] The reference marker 6 is a predetermined two-dimensional image. The reference marker 6 may be, for example, an AR marker, a QR code (registered trademark), or any other two-dimensional image with a predetermined shape. The size of the reference marker 6 may be predetermined, for example. This embodiment mainly describes the case where the reference marker 6 is displayed on a sheet 6a. The reference marker 6 may be printed on a sheet 6a made of paper or resin, for example.

[0016] The predetermined positional relationship between the reference marker 6 and the 3D model of the virtual robot may be, for example, a predetermined positional relationship, or a positional relationship that can be changed by the operator operating the virtual robot. The predetermined positional relationship may be, for example, a positional relationship in which the 3D model of the virtual robot is displayed at the position of the reference marker 6, or a positional relationship in which the 3D model of the virtual robot is displayed at a position different from the reference marker 6. In the former case, the image generation device 3 may place the 3D model of the virtual robot at the position of the reference marker 6. Placing the 3D model of the virtual robot at the position of the reference marker 6 means that the 3D model of the virtual robot is placed in the virtual space in such a way that the situation in which a real robot is placed at the position of the reference marker 6 is virtually reproduced. For example, the 3D model of the virtual robot may be placed so that the end face on the base end side of the 3D model of the virtual robot (for example, the mounting surface to the floor, etc.) coincides with the surface of the reference marker 6. Placing the 3D model of the virtual robot at a position different from the reference marker 6 means, for example, that the 3D model of the virtual robot is placed next to the reference marker 6.

[0017] The display device 4 displays an image superimposed on an image of the real environment or the real environment itself. That is, the operator operating the virtual robot can see both the image of the real environment and the image of the virtual environment through the display device 4. The display device 4 may be a wearable display device worn on the head of the operator operating the virtual robot, or it may be a portable information processing terminal such as a tablet device. The wearable display device may be, for example, a head-mounted display. The display device 4 may also have, for example, a transparent display. In this case, the display device 4 will display the image superimposed on the real environment itself. Examples of wearable display devices 4 with transparent displays include HoloLens®. Such a display device 4 with a transparent display can also be considered a display device for realizing mixed reality (MR). The display device 4 may also have, for example, an opaque display. In this case, the display device 4 will display the image superimposed on an image of the real environment. Therefore, it is preferable that the display device 4 with an opaque display has a camera for capturing the real environment, or is connected to a camera for capturing the real environment. Images of the real environment captured by the camera are displayed in real time on an opaque display. Examples of wearable display devices 4 having an opaque display include the Oculus Quest. Such a display device 4 having an opaque display can also be considered a display device for realizing augmented reality (AR). A portable information processing terminal such as a tablet device may have, for example, a camera and a display, and may display images of the real environment captured by the camera on the display in real time. In this embodiment, the case in which the display device 4 is a head-mounted display having a transparent display will be mainly described.

[0018] As shown in FIG. 1, the image generation device 3 includes a storage unit 31, a positional relationship acquisition unit 32, an attitude acquisition unit 33, a specifying unit 34, a reception unit 35, an image generation unit 36, and an output unit 37.

[0019] In the storage unit 31, three-dimensional models of virtual robots and virtual tools are stored. The three-dimensional models of the virtual robots and virtual tools may, for example, correspond to actual robots and actual tools under consideration for introduction. When there are multiple candidates for introduction, for example, three-dimensional models of multiple actual robots and multiple actual tools may be stored in the storage unit 31. Note that the three-dimensional model of the virtual robot and the three-dimensional model of the virtual tool may be stored separately in the storage unit 31, or a three-dimensional model of a virtual robot with a virtual tool attached may be stored. In the former case, for example, it may be possible to generate a three-dimensional model of a virtual robot with a virtual tool attached from the three-dimensional models of the virtual robot and the virtual tool. In this case, for example, the position and attitude of the virtual robot to which the virtual tool is attached may be determined in advance. Also, information other than the three-dimensional model may be stored in the storage unit 31. For example, teaching data or the like may be stored in the storage unit 31.

[0020] The process by which information is stored in the storage unit 31 is not limited. For example, information may be stored in the storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage unit 31. The storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0021] The position relationship acquisition unit 32 acquires the relative position relationship between the reference marker 6 existing in the actual environment and the display device 4. Acquiring the relative position relationship between the reference marker 6 and the display device 4 may, for example, involve acquiring the relative position relationship between the marker coordinate system, which is the local coordinate system of the reference marker 6, and the display coordinate system, which is the local coordinate system of the display device 4. This relative position relationship may be represented, for example, by a homogeneous transformation matrix indicating the transformation between the two coordinate systems. The method by which the position relationship acquisition unit 32 acquires this relative position relationship is not limited. The position relationship acquisition unit 32 may, for example, receive an image captured by the camera of the display device 4 and use three or more feature points of the reference marker 6 included in the image to obtain a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system. The acquisition of the homogeneous transformation matrix may be performed by the display device 4. In this case, the position relationship acquisition unit 32 may receive the homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system from the display device 4. That is, the acquisition of the relative position relationship by the position relationship acquisition unit 32 may be the reception of the relative position relationship.

[0022] The posture acquisition unit 33 acquires a target posture for the virtual tool. The target posture for the virtual tool may be, for example, the target posture of a part of the virtual tool. For example, if the virtual tool is a welding torch, the target posture for the virtual tool may be the target posture of the tip of the welding torch. This target posture may be, for example, the posture in the world coordinate system or the posture in the local coordinate system. The world coordinate system may be, for example, the marker coordinate system. The local coordinate system may be, for example, the display coordinate system which is the local coordinate system of the display device 4. The posture acquisition unit 33 may acquire, for example, the target posture corresponding to the 3D model of the manipulated virtual tool, or it may acquire the target posture for the virtual tool by other means. When the 3D model of the virtual tool is displayed, the posture of the 3D model of the virtual tool in the virtual space is known. In addition, the positional relationship between that virtual space and the world coordinate system or local coordinate system can also be acquired. Therefore, the posture acquisition unit 33 can acquire, for example, the target posture for the virtual tool in the world coordinate system or the local coordinate system of the display device 4. When the posture acquisition unit 33 acquires the posture of the displayed virtual tool, it may acquire, for example, the longitudinal angle of a predetermined part of the virtual tool (for example, the tip of a welding torch). This posture may be represented by, for example, roll, pitch, yaw angle, or Euler angle. Furthermore, the posture information acquired by the posture acquisition unit 33 may be any information that can identify the posture of the virtual tool as a result. The posture acquisition unit 33 may, for example, acquire the posture of the virtual tool at the time the reception unit 35 receives an instruction to acquire the target posture as the target posture. Also, when the target posture corresponding to the 3D model of the operated virtual tool is acquired, for example, the target posture of the virtual tool may be acquired when only the display image of the virtual tool is shown, or the target posture of the virtual tool may be acquired when the display image of the virtual robot to which the virtual tool is attached is shown. In this embodiment, the latter case will be mainly described.

[0023] The identification unit 34 identifies the reachable range of the virtual tool in the desired orientation mounted on the virtual robot, which is positioned to have a predetermined positional relationship with the reference marker 6, based on a three-dimensional model of the virtual robot to which the virtual tool is mounted. The reachable range of the virtual tool in the desired orientation may be identified in the robot coordinate system of the virtual robot. For example, the identification unit 34 may convert the orientation of the virtual tool acquired by the orientation acquisition unit 33 into an orientation in the robot coordinate system using the positional relationship between the coordinate system in which the orientation of the virtual tool was acquired and the robot coordinate system, and then identify the reachable range of the virtual tool in the desired orientation mounted on the virtual robot using the three-dimensional model of the virtual robot to which the virtual tool is mounted and its orientation. The reachable range of the virtual tool in the desired orientation may also be the reachable range of a predetermined position on the virtual tool in the desired orientation. If the virtual tool is a welding torch, the predetermined position may be, for example, the position of the tip of the welding torch. The specific unit 34 may, for example, identify a range corresponding to the set of grid points from which the angles of each joint can be calculated by inverse kinematics when a virtual tool with the desired posture is positioned on a virtual robot that is placed in a predetermined positional relationship with a reference marker 6, among a plurality of grid points arranged at predetermined intervals in three-dimensional space. The reachable range of the virtual tool with the desired posture may be defined as the set of grid points from which the angles of each joint can be calculated by inverse kinematics. The grid points from which the angles of each joint can be calculated by inverse kinematics may hereafter be referred to as "grid points reachable by the virtual tool." The inability to calculate the angles of each joint of the virtual robot by inverse kinematics may, for example, mean that there is no solution in the inverse kinematics calculation, or that there is a solution in the inverse kinematics calculation, but at least one of those solutions exceeds the pre-set operating range of each joint. The operating range of each joint may be set, for example, to prevent interference with the housing of the virtual robot, or to prevent cable breakage or avoid singularities. The three-dimensional space in which the plurality of grid points are arranged is a virtual space. Furthermore, placing a virtual tool in the desired orientation at a grid point may mean, for example, positioning the virtual tool such that a predetermined position of the virtual tool in the desired orientation (e.g., the position of the tip of the welding torch) is at the location of a grid point.

[0024] Multiple grid points may be set, for example, in the local coordinate system of the virtual robot. Alternatively, multiple grid points may be arranged, for example, so that eight adjacent grid points are located at the vertices of a cube. As an example, multiple grid points may be arranged at equal intervals in each axis direction of a three-dimensional Cartesian coordinate system. Furthermore, multiple grid points may be set in the coordinate system in advance, or they may be generated when the reachable range is determined. Since the virtual robot is positioned to have a predetermined positional relationship with the reference marker 6, for example, multiple grid points 50 may be arranged to have a predetermined positional relationship with the reference marker 6, as shown in Figure 4. In Figure 4, the reference marker 6 is assumed to be located at the center of the bottom surface of a cube on which the grid points 50 are displayed. The identification unit 34 may then determine for each of the multiple grid points 50 whether the angles of each joint of the virtual robot can be calculated by inverse kinematics, with a virtual tool attached to the virtual robot, which is positioned to have a predetermined positional relationship with the reference marker 6, placed at the grid points 50. To place a virtual tool at a grid point means, for example, placing a predetermined location on the 3D model 20 of the virtual tool (e.g., the tip of the virtual tool) at a grid point 50, as shown in Figure 4. In this way, multiple grid points are classified into grid points where the angles of each joint can be calculated by inverse kinematics, i.e., grid points that the virtual tool can reach, and grid points where the angles of each joint cannot be calculated by inverse kinematics, i.e., grid points that the virtual tool cannot reach. In Figure 4, for the sake of explanation, the display of the virtual robot is omitted, but in reality, the virtual robot is positioned in a predetermined positional relationship with the reference marker 6, and the virtual tool is attached to that virtual robot. Also, although only the grid points 50 on the surface are shown in Figure 4, it goes without saying that grid points 50 also exist inside. Furthermore, for example, there may be even more grid points outside the cube where the grid points 50 on the surface are displayed in Figure 4.

[0025] The reception unit 35 may, for example, accept operations on a 3D model of a virtual tool or a virtual robot equipped with a virtual tool, in order to acquire a desired posture related to the virtual tool. The reception unit 35 may also accept instructions to acquire a desired posture. The reception unit 35 may also accept operations on a 3D model of a virtual robot.

[0026] Operations on the 3D model may include, for example, operations to change the position or orientation of at least a part of the 3D model. Instructions for operation may be received, for example, via an input device such as a teaching pendant present in the real environment, or via a virtual input interface such as a virtual button or virtual teaching pendant displayed on the display of the display device 4. The virtual input interface may be displayed on the display of the display device 4 in response to an action such as an air tap, and when a button or the like is selected by the operator's finger or pointing device, input corresponding to the operation of that button or the like may be passed from the display device 4 to the reception unit 35. In addition, the position or orientation of at least a part of the 3D model of a virtual robot or virtual tool (for example, the end effector of a tool) may be changed by a gesture of the operator's hand. In this case, for example, by the operator performing a pinching motion (holding motion) of the 3D model displayed on the display, the part pinched by the hand is identified as the target of operation, and by changing the position or orientation of the hand, an operation is performed to change the position or orientation of the identified target, and by ending the pinching motion, the operation on the identified target may be terminated. In this case, for example, information indicating the object of the operation (e.g., information indicating the position or part in the 3D model) and information indicating the content of the operation (e.g., information indicating a change in position or posture) may be passed from the display device 4 to the reception unit 35, or the results of hand tracking acquired by the display device 4 may be passed to the reception unit 35, and the object of the operation and the content of the operation may be identified in the image generation device 3. When information indicating the object of the operation and information indicating the content of the operation are acquired by the display device 4, the display device 4 may be able to access the current 3D model information of the virtual robot or virtual tool in the virtual space, which is held in the image generation device 3. Furthermore, when the operation is performed using the operator's hands, the display device 4 may have a camera, and the position of the operator's hands on the display may be identified by performing hand tracking of the operator's hands captured by the camera.Furthermore, methods for converting operations on a 3D model's display image in response to a worker's hand gestures into changes in the position and orientation of the 3D model in a 3D virtual space are already publicly known, and a detailed explanation of these methods will be omitted.

[0027] The reception unit 35 may accept information or instructions other than those described above. For example, the reception unit 35 may accept information input from an input device or display device 4, or it may receive information transmitted via a wired or wireless communication line. The reception unit 35 may or may not include a device for receiving information (for example, an input device or a communication device). Furthermore, the reception unit 35 may be implemented by hardware, or by software such as a driver that drives a predetermined device.

[0028] The image generation unit 36 ​​generates a display image for showing the reachable range specified by the specification unit 34, based on the relative positional relationships acquired by the positional relationship acquisition unit 32. The relative positional relationship between the reference marker 6 and the display device 4 is acquired by the positional relationship acquisition unit 32. The positional relationship between the reference marker 6 and the 3D model of the virtual robot is also determined. Furthermore, the reachable range in the robot coordinate system is specified by the specification unit 34. Therefore, using this information, the image generation unit 36 ​​can specify the relative positional relationship between the reachable range and the display device 4. Thus, the image generation unit 36 ​​can place a 3D model indicating the reachable range in the virtual space and specify the position and orientation of the display device 4 relative to that 3D model, which has the specified positional relationship. Then, the image generation unit 36 ​​can generate a 2D display image for displaying the 3D model by rendering the 3D model indicating the reachable range in the virtual space based on the position and orientation of the display device 4. The 3D model indicating the reachable range is not particularly limited as long as it can indicate the reachable range. For example, it may be a 3D model showing the outermost surface of the reachable range, a 3D model showing multiple grid points that the virtual tool can reach, or any other 3D model indicating the reachable range. The image generation unit 36 ​​generates the display image so that when the display image of the 3D model indicating the reachable range is displayed on the display of the display device 4, the size of the display image matches that of the actual environment. That is, the display image is generated so that it shows the reachable range of the actual tool in the target posture attached to the actual robot, which is positioned in a predetermined positional relationship with the reference marker 6 in the actual environment. Furthermore, the display image of the 3D model of the reachable range may be, for example, an opaque image that does not allow the background image to be seen, or a semi-transparent image that allows the background image to be seen.

[0029] Furthermore, the image generation unit 36 ​​may generate a display image for showing a 3D model of the virtual robot equipped with the virtual tool, or a 3D model of the virtual tool, based on the relative positional relationship. When an operation on the virtual robot or virtual tool is received, the image generation unit 36 ​​may generate a display image for showing the operated 3D model. The display image for showing the 3D model of the virtual tool may be generated, for example, to acquire a desired posture for the virtual tool. In this case, a display image of only the virtual tool, i.e., a display image that does not include the virtual robot, may be generated.

[0030] The relative positional relationship between the reference marker 6 and the display device 4 is acquired by the positional relationship acquisition unit 32. Furthermore, the positional relationship between the reference marker 6 and the 3D model of the virtual robot is predetermined. Therefore, using this information, the image generation unit 36 ​​can determine the relative positional relationship between the 3D model of the virtual robot and the display device 4. Thus, the image generation unit 36 ​​can place the 3D model of the virtual robot, equipped with virtual tools, in the virtual space and determine the position and orientation of the display device 4 relative to that 3D model, given the determined positional relationship. The image generation unit 36 ​​then generates a 2D display image for displaying the 3D model by rendering the 3D model of the virtual robot equipped with virtual tools in the virtual space, using the position and orientation of the display device 4 as a reference. The angles of each joint of the virtual robot are at their initial values ​​if no operation is performed, and at their post-operation values ​​if an operation is performed. The post-operation angles of each joint may be calculated, for example, using inverse kinematics based on the position and orientation of the end-effector in the 3D model of the virtual robot after operation, similar to a real robot. When a virtual robot equipped with a virtual tool is operated, the shape of the 3D model in the virtual space changes accordingly, as described above. Also, if the position or orientation of the display device 4 changes in the real environment, the position and orientation of the viewpoint in the virtual space will change accordingly. After these changes, rendering is performed to generate display images of the 3D model after operation and display images of the 3D model after the change in the position and orientation of the display device 4. The image generation unit 36 ​​generates the display image so that when the display image of the 3D model of the virtual robot equipped with the virtual tool is displayed on the display device 4, the size of the display image matches that of the real environment. In other words, the display image is generated so that the 3D model of the virtual robot equipped with the virtual tool, displayed on the display device 4, and the real robot equipped with the real tool, placed in the real environment to have the same relative positional relationship as the 3D model, appear to be the same size when viewed through the display device 4.Furthermore, when generating a display image to show only the 3D model of a virtual tool, the display image can be generated in the same way as when displaying the 3D model of a virtual robot, for example, by fixing the positional relationship between the reference marker 6 and the virtual space. In this case as well, it is preferable that the display image is generated such that when the display image of the 3D model of the virtual tool is displayed on the display device 4, the size of the display image matches that of the real environment.

[0031] Furthermore, it may not be possible to operate the virtual robot in response to the accepted operation. For example, an operation may be accepted that moves the end effector of a virtual robot equipped with a virtual tool beyond its movable range or rotates it beyond its rotatable range. In such cases, the image generation unit 36 ​​does not need to generate a display image corresponding to the operation, or it may generate a display image of the 3D model of the virtual robot, etc., that has been moved or rotated within the possible range.

[0032] The virtual robot will be displayed in a predetermined positional relationship with the reference marker 6 placed in the real environment. The reachable range will be located at a specific position in the robot coordinate system. Therefore, even if the orientation of the display device 4 changes, the display position of the 3D models of the virtual robot and reachable range in the real environment will not change. Furthermore, even when only a display image of the virtual tool is generated, the orientation of the virtual tool in the world coordinate system may be changed, for example, only by accepting an operation on the 3D model of the virtual tool. In this case, even if the orientation of the display device 4 changes, the display position of the 3D model of the virtual tool in the real environment will not change.

[0033] The output unit 37 outputs the display image generated by the image generation unit 36 ​​to the display device 4. When outputting the display image, the output unit 37 may output only the display image. In this case, the display image will be superimposed on the image of the real environment or the real environment itself on the display device 4. On the other hand, if the display device 4 has an opaque display and the image of the real environment captured by the display device 4 is received by the image generation device 3, the result of combining the image of the real environment and the display image may be output to the display device 4. This combination may be performed, for example, by a combination unit (not shown) of the image generation device 3.

[0034] If the 3D model of the virtual robot is displayed at a different position from the reference marker 6, for example, the 3D model 10 of the virtual robot with the virtual tool attached may be displayed next to the reference marker 6, as shown in Figure 1. The reference marker 6 and the 3D model 10 of the virtual robot in Figure 1 schematically represent the situation as seen by the operator operating the virtual robot via the display device 4.

[0035] The display of 3D models of virtual robots and virtual tools, and the manipulation of these 3D models of virtual robots using virtual input interfaces on display devices or operator gestures, are already publicly known, as shown in references 1-3 below, and therefore a detailed explanation will be omitted. Reference 1: Japanese Patent Publication No. 2017-100234 Reference 2: Japanese Patent Publication No. 2020-055075 Reference 3: Japanese Patent Publication No. 2020-069538

[0036] Next, the operation of the image generation device 3 will be explained using the flowchart in Figure 2. (Step S101) The image generation unit 36 ​​generates a display image of the 3D model of the virtual robot equipped with the virtual tool, and the output unit 37 outputs the generated display image to the display device 4. The reception unit 35 may also accept operations on the displayed virtual tool. When an operation is accepted, a display image of the 3D model of the virtual tool with its posture changed according to the operation may be generated and output. Details of this process will be described later using the flowchart in Figure 3.

[0037] (Step S102) The reception unit 35 determines whether it has received input indicating that it will determine the desired stance regarding the virtual tool. If it has received the decision, it proceeds to step S103; otherwise, it returns to step S101.

[0038] (Step S103) The posture acquisition unit 33 acquires the posture of the virtual tool at that time as the target posture.

[0039] (Step S104) The identification unit 34 identifies the reachable range of the virtual tool in the target orientation, which is mounted on a virtual robot positioned to have a predetermined positional relationship with the reference marker 6.

[0040] (Step S105) The image generation unit 36 ​​generates a display image of a 3D model showing the reachable range identified in step S104, and the output unit 37 outputs the generated display image to the display device 4. Details of this process will be described later using the flowchart in Figure 3.

[0041] (Step S106) The reception unit 35 determines whether it has received an instruction to terminate the series of processes. If it has received an instruction to terminate, the series of processes is terminated; otherwise, it returns to step S105.

[0042] Note that the order of processing in the flowchart of Figure 2 is just one example, and the order of each step may be changed if similar results can be obtained. Also, the flowchart of Figure 2 may be executed, for example, when a mode to display the reachable range of the tool in the desired posture is activated while a display image of a 3D model of a virtual robot equipped with a virtual tool, corresponding to the received operation, is being generated and output to the display device 4. In this case, for example, after the series of processes shown in Figure 2 are completed, the system may return to the mode of generating a display image of the 3D model corresponding to the operation and outputting it to the display device 4.

[0043] Next, using the flowchart in Figure 3, we will explain the process of generating the display image of the virtual robot in the flowchart in Figure 2 (step S101) and the process of generating the display image of the reachable range (step S105). Note that in step S101, the object of operation and display is the 3D model of the virtual robot equipped with the virtual tool, and in step S105, the object of display is the 3D model showing the reachable range, but in Figure 3, the two are not particularly distinguished in the explanation.

[0044] (Step S201) The reception unit 35 determines whether it has received an operation request for the 3D model. If it has received the operation request, it proceeds to step S203; otherwise, it proceeds to step S202.

[0045] (Step S202) The image generation unit 36 ​​determines whether to generate a display image. If it decides to generate a display image, it proceeds to step S203; otherwise, it returns to the flowchart in Figure 2. The image generation unit 36 ​​may, for example, periodically decide whether to generate a display image. This decision ensures that, for example, even if no operation is performed, if the position or orientation of the display device 4 is changed, the display image corresponding to the changed position or orientation will be displayed on the display device 4.

[0046] (Step S203) The position relationship acquisition unit 32 acquires the relative position relationship between the reference marker 6 and the display device 4.

[0047] (Step S204) The image generation unit 36 ​​generates a display image for displaying the 3D model using the relative positional relationship acquired in step S203. If the display target is a virtual robot equipped with a virtual tool, the 3D model stored in the storage unit 31 may be used to generate the display image. If an operation is accepted, a display image will be generated to show the 3D model after it has been modified according to that operation. If the position or orientation of the display device 4 is changed, a display image of the 3D model corresponding to the changed position and orientation of the display device 4 will be generated. The display image for showing the 3D model of the virtual robot will be generated so as to have a predetermined positional relationship with the reference marker 6.

[0048] (Step S205) The output unit 37 outputs the generated display image to the display device 4. Then, the flowchart in Figure 2 is returned. Depending on this output, for example, a 3D model of a virtual robot equipped with a virtual tool may be displayed, or a 3D model showing the reachable range may be displayed.

[0049] Note that if the flowchart in Figure 3 shows the process in step S105 of Figure 2, the process in step S201 does not need to be performed, because no operation is performed on the reachable range. If the process in step S201 is not performed, the flowchart in Figure 3 may start from step S202.

[0050] Next, the operation of the image generation device 3 according to this embodiment will be explained using a specific example. First, the operator places the sheet 6a with the reference marker 6 displayed on it at the position where the actual robot to be introduced will be placed. Next, the operator puts the head-mounted display device 4 on their head and starts the processing of the image generation device 3. The image generation device 3 then generates a display image of the virtual robot with the virtual tool attached and outputs it to the display device 4 (step S101). Specifically, the image generation unit 36 ​​determines that it is time to generate a display image and gives an instruction to the position relationship acquisition unit 32 to acquire the relative position relationship between the reference marker 6 and the display device 4 (step S202). In response to that instruction, the position relationship acquisition unit 32 acquires the relative position relationship and passes it to the image generation unit 36 ​​(step S203). Upon receiving the relative positional relationship, the image generation unit 36 ​​uses that relative positional relationship and the 3D model of the virtual robot equipped with the virtual tool stored in the memory unit 31 to generate a display image for displaying the 3D model at a predetermined initial position and initial orientation, and passes it to the output unit 37 (step S204). The output unit 37 outputs the received display image to the display device 4 (step S205). As a result, the operator can see the virtual robot equipped with the virtual tool superimposed in real space.

[0051] Subsequently, the operator moves the tip of the displayed virtual tool, the welding torch, to the desired position using input devices such as a teaching pendant or gesture operations (steps S201, S203-S205). At that point, the operator inputs their determination of the desired position via an input device or virtual input interface, and this input is received by the reception unit 35 and passed to the position acquisition unit 33 (step S102). The position acquisition unit 33 then acquires the position of the virtual tool at that point as the desired position and passes it to the identification unit 34 (step S103).

[0052] Upon receiving the desired posture, the identification unit 34 determines, for each grid point 50, whether the angle of each joint of the virtual robot can be calculated by inverse kinematics when the 3D model 20 of the virtual tool is placed at each of the grid points 50 arranged at predetermined intervals, for example as shown in Figure 4, in the robot coordinate system of the virtual robot positioned at the reference marker 6. As a result, it is possible to identify a set of grid points 50 for which the angle of each joint of the virtual robot can be calculated by inverse kinematics, that is, a set of grid points 50 that the virtual tool can reach. The identification unit 34 then identifies information about the contour of the reachable range surface, which is a set of square faces formed by the outermost grid points 50 of the set of grid points 50 that the virtual tool can reach, and passes it to the image generation unit 36 ​​(step S104). This contour information may be, for example, a set of grid points that show the contour of the reachable range surface.

[0053] Upon receiving information indicating the reachable range, the image generation unit 36 ​​determines to generate a display image and instructs the position relationship acquisition unit 32 to acquire the relative position relationship between the reference marker 6 and the display device 4 (step S202). In response to this instruction, the position relationship acquisition unit 32 acquires the relative position relationship and passes it to the image generation unit 36 ​​(step S203). Upon receiving the relative position relationship, the image generation unit 36 ​​generates a 3D model indicating the reachable range based on the information indicating the reachable range, and uses this 3D model indicating the reachable range and the received relative position relationship to generate a display image for displaying the 3D model of the reachable range and passes it to the output unit 37 (step S204). The output unit 37 outputs the received display image to the display device 4 (step S205). As a result, the operator can see the reachable range 51 superimposed on real space, as shown in Figure 5. Although Figure 5 shows the case where the grid points are coarse, a more detailed reachable range can be displayed by setting finer grid points. On the other hand, when finer grid points are used, the processing load on the specific unit 34 increases, so it is preferable to use grid points of a desirable fineness by comparing the degree of processing load with the degree of detail required for the reachable range.

[0054] As described above, the image generation device 3 according to this embodiment can identify the reachable range of a virtual tool in a target posture and generate and output a display image to show that reachable range. Therefore, for example, someone considering the introduction and placement of a new robot can easily find out what range the virtual tool in the target posture can reach by looking at the display image, and can confirm whether the target task can be performed. In addition, normally, even if a three-dimensional range is displayed on a two-dimensional plane, it is not easy to understand the depth, etc. However, with the image generation device 3 according to this embodiment, the viewpoint from which to view the display image of the reachable range can be changed in accordance with the change in the position and orientation of the display device 4, so that the three-dimensional range can be easily grasped.

[0055] Furthermore, by determining whether the angle of each joint of the virtual robot can be calculated using inverse kinematics while the virtual tool attached to the virtual robot is positioned at a grid point in the desired posture, the grid points that the virtual tool can reach in the desired posture can be identified, and the reachable range of the virtual tool in the desired posture can be determined according to the set of grid points.

[0056] In the image generation device 3 according to this embodiment, shortening the spacing between grid points makes it possible to more precisely determine the reachable range of the virtual tool for the target pose. On the other hand, shortening the spacing between grid points increases the number of grid points, which increases the computational load of the inverse kinematics calculation performed for each grid point. Therefore, the identification unit 34 may, for example, use two types of grid points to determine the reachable range. In this case, the plurality of grid points may include a plurality of first grid points arranged at a first spacing and a plurality of second grid points arranged at a second spacing shorter than the first spacing. The first grid points and the second grid points may be located at different positions, or at least some of the grid points may be at the same position. In the latter case, for example, a plurality of first grid points may be included in a plurality of second grid points. This embodiment will mainly describe this case.

[0057] The identification unit 34 may identify a first grid point that the virtual tool can reach, identify a second grid point that the virtual tool can reach between the first grid point that the virtual tool can reach and the first grid point that the virtual tool cannot reach, and identify a range corresponding to the set of first and second grid points that the virtual tool can reach as the reachable range of the virtual tool in the desired orientation. For example, the identification unit 34 may, after determining whether each of the multiple first grid points is reachable by the virtual tool in the desired orientation, determine whether, for each smallest unit cube having eight first grid points at each vertex, both grid points that are reachable and grid points that are not reachable at the eight first grid points at each vertex are included. Furthermore, if a cube contains both reachable and inreachable lattice points, the specific unit 34 determines whether the virtual tool in the desired orientation is reachable for each of the multiple second lattice points contained in the cube. If not, it does not need to determine whether the multiple second lattice points contained in the cube are reachable.

[0058] In this way, by using the first and second grid points, the identification unit 34 can identify a rough reachable range using the first grid point and then identify a finer reachable range near the boundary of that reachable range using the second grid point. Therefore, it becomes possible to identify a finer reachable range with a lower computational load. Furthermore, although the case in which the reachable range is identified using two stages of grid points with different spacings, i.e., the first and second grid points, the reachable range may also be identified using three or more stages of grid points with different spacings, i.e., the first to Nth grid points (where N is an integer of 3 or more). In this case, the reachable range of the virtual tool with respect to finer grid points may be identified by repeating the same process as in the identification of the reachable range using the first and second grid points. Note that the spacing of the grid points is shorter for the (K+1)th grid point than for the Kth grid point. K is any integer from 1 to N-1.

[0059] Furthermore, the virtual tool attached to the virtual robot may be able to rotate by any angle around the Z-axis of the tool coordinate system. Note that the Z-axis of the tool coordinate system may be, for example, the axis extending in the longitudinal direction of the tip of the torch when the tool is a welding torch. In this case, even if the virtual tool is rotated by any angle around the Z-axis of the tool coordinate system, the orientation of the virtual tool will not change. Therefore, the specific unit 34 may define the grid points that the virtual tool can reach as grid points from which the angles of each joint can be calculated by inverse kinematics at any angle obtained by rotating the virtual tool in the desired orientation around the Z-axis of the tool coordinate system. Furthermore, when rotating the virtual tool of the target pose around the Z-axis of the tool coordinate system, the specific unit 34 may, for example, rotate the virtual tool at predetermined fine intervals (e.g., every 10 degrees, every 30 degrees, etc.), or it may sequentially rotate the virtual tool at predetermined coarse intervals (e.g., every 90 degrees, every 120 degrees, etc.), and if it is not possible to calculate the angles of each joint by inverse kinematics, it may shift the position by a finer interval (e.g., 10 degrees or 30 degrees, etc.) and then repeat the process of sequentially rotating the virtual tool again at predetermined coarse intervals. In this way, by rotating the virtual tool by an arbitrary angle around the Z-axis of the tool coordinate system and performing inverse kinematics calculations, the possibility of calculating the angles of each joint by inverse kinematics can be increased, and a more accurate determination of the reachable range can be achieved.

[0060] Furthermore, among multiple grid points from which the angles of each joint can be calculated by inverse kinematics, for example, when a virtual tool moves from one grid point to another, the angles of at least one joint may become discontinuous. More specifically, when a virtual tool in a desired posture moves from one grid point to another, the angle of a certain joint may rotate by nearly 360 degrees. Between such two grid points, there exists a boundary where the angles of at least one joint of the virtual robot become discontinuous. A virtual tool attached to a virtual robot can move across that boundary, but a real tool attached to a real robot cannot. Therefore, if such a boundary exists, the identification unit 34 may, for example, specify an reachable range within the range that does not cross the boundary, or it may specify the boundary itself. That is, for example, if there is a boundary where the angles of at least one joint become discontinuous among multiple grid points from which the angles of each joint can be calculated by inverse kinematics with a virtual tool in a desired posture positioned, the identification unit 34 may define the grid point on the base end side of the virtual robot relative to that boundary as a grid point reachable by the virtual tool. Furthermore, the specific unit 34 may, for example, identify a boundary where the angle of at least one joint becomes discontinuous for a plurality of grid points where the angle of each joint can be calculated by inverse kinematics with a virtual tool representing the target posture in place.

[0061] Whether a boundary exists between two grid points reachable by the virtual tool where the angle of at least one joint of the virtual robot becomes discontinuous may be determined, for example, by calculating the absolute value of the angle difference corresponding to the two grid points for each joint and determining whether there is a joint where the absolute value of the angle difference exceeds a threshold. Specifically, the absolute value of the angle difference corresponding to the two grid points reachable by the virtual tool for each joint may be calculated, and if the absolute value of the angle difference for any joint exceeds a threshold, it may be determined that a boundary exists between those two grid points where the angle of at least one joint of the virtual robot becomes discontinuous. This determination may be made, for example, by the identification unit 34. The threshold may be, for example, a threshold proportional to the distance between the two grid points. For example, the greater the distance between the two grid points, the greater the threshold may be. For example, as shown in Figure 6, if grid points 50-1 to 50-8 are arranged, and the absolute value of the difference in joint angles calculated by inverse kinematics for two grid points 50-1 and 50-4 exceeds a threshold, and the same is true for two grid points 50-3 and 50-4, two grid points 50-5 and 50-8, and two grid points 50-7 and 50-8, then it may be determined that a boundary exists between two grid points 50-1 and 50-4, two grid points 50-3 and 50-4, two grid points 50-5 and 50-8, and two grid points 50-7 and 50-8. Furthermore, if grid points 50-1, 50-3, 50-5, and 50-7 are closer to the base end of the virtual robot than grid points 50-4 and 50-8, then the identification unit 34 does not need to identify grid points 50-4 and 50-8 as grid points reachable by the virtual tool. Furthermore, when the identifying unit 34 identifies a boundary, it may, for example, identify the grid points on the base end side of the virtual robot with respect to the boundary, i.e., grid points 50-1 to 50-3, 50-5 to 50-7, as the boundary; it may identify the grid points on the opposite side of the boundary, i.e., grid points 50-4 and 50-8, as the boundary; or it may identify the area between grid points 50-1, 50-3, 50-5, and 50-7 and grid points 50-4 and 50-8 as the boundary. Thus, identifying a boundary may also involve identifying grid points in the vicinity of that boundary or other points.

[0062] If a boundary is identified, the image generation unit 36 ​​may, for example, generate a display image for showing the identified boundary. The display image for showing the boundary may, for example, be a display image for showing a three-dimensional model that shows the boundary. The three-dimensional model may, for example, be a three-dimensional model that shows a surface composed of identified grid points or other points. When this display image is displayed on the display device 4, the person viewing the display image on the display device 4 will be able to know the location of the boundary. When a boundary is identified, the identification unit 34 may define the reachable range as a range corresponding to the set of all grid points from which the angles of each joint can be calculated by inverse kinematics.

[0063] By identifying grid points inside the boundary, i.e., grid points on the base end side of the virtual robot relative to the boundary, as grid points reachable by the virtual tool, the reachable range that the virtual tool can reach in the desired pose without discontinuities in the angles of all joints is determined. Therefore, when the virtual tool moves within this determined reachable range, the angles of all joints will not be discontinuous. In addition, a display image indicating the boundary is generated and output, allowing the user viewing the image on the display device 4 to know the position of the boundary. Therefore, for example, when performing teaching work using the virtual tool, by avoiding crossing the boundary, it is possible to ensure that the angles of all joints are not discontinuous.

[0064] Furthermore, although this embodiment mainly describes the case where the posture acquisition unit 33 acquires one target posture for the virtual tool, this is not the case. The posture acquisition unit 33 may acquire two or more target postures for the virtual tool. For example, if the posture of the virtual tool differs at the welding start position and the welding end position, the posture acquisition unit 33 may acquire two postures, the welding start position and the welding end position. Alternatively, the posture acquisition unit 33 may acquire three or more postures, including the welding start position, intermediate position, and end position.

[0065] If the posture acquisition unit 33 acquires two or more target postures for the virtual tool, the identification unit 34 may identify the reachable range of the virtual tool for each of the two or more target postures. The identified two or more reachable ranges may be displayed separately, or a range that is included in all of the two or more reachable ranges may be displayed, or a range that is included in any of the two or more reachable ranges may be displayed. In the latter case, for example, the image generation unit 36 ​​may generate a display image of a 3D model showing a range that is included in all of the two or more reachable ranges, or a display image of a 3D model showing a range that is included in any of the two or more reachable ranges. If two or more reachable ranges are identified, for example, a person viewing the display image on the display device 4 may be able to choose whether to display a range that is included in all of the two or more reachable ranges, or a range that is included in any of the two or more reachable ranges. This selection may be made, for example, via an input device such as a teaching pendant that exists in the real environment, or via a virtual input interface such as a virtual button or virtual teaching pendant displayed on the display of the display device 4. The selection result may be received by the reception unit 35 and passed to the image generation unit 36. Furthermore, if two or more reachable ranges are identified, for example, a person viewing the display image on the display device 4 may identify one or more reachable ranges of the displayed object. This identification may also be performed via an input device or virtual input interface, and the identification result may be received by the reception unit 35 and passed to the image generation unit 36. The image generation unit 36 ​​may generate a display image relating to one or more reachable ranges according to the selection result or the identification result. In this way, by identifying two or more reachable ranges corresponding to two or more desired poses of the virtual tool and displaying them, a person viewing the display image on the display device 4 can easily grasp the range in which the virtual tool can assume multiple poses.

[0066] Furthermore, although this embodiment mainly describes the case where the reference marker 6 is a predetermined two-dimensional image, this is not required. Similar to markerless AR, an object existing in the real environment may be used as the reference marker. The object existing in the real environment is not particularly limited, but for example, it may be the base used to install the actual robot or a jig located near the position where the robot is placed. Alternatively, a marker on a three-dimensional object (for example, a rectangular prism shape) with markers attached to each surface may be used as the reference marker. In addition, the reference marker may be three-dimensional, such as a base or a jig.

[0067] Furthermore, although this embodiment mainly describes the case in which the posture acquisition unit 33 acquires the desired posture using a 3D model of the operated virtual tool, this is not required. The posture acquisition unit 33 may, for example, acquire the desired posture indicated by the posture of an operation instruction member present in the real environment. In this case, the desired posture for the virtual tool can be set using an operation instruction member present in the real environment, making the setting easier than when using the display of a 3D model. The shape of the operation instruction member is not particularly limited, but may be a rod-shaped member or a member of other shapes. The rod-shaped member may be, for example, a cylindrical member or a polygonal prism-shaped member. The operation instruction member, which is a rod-shaped member, may have different shapes at both ends in the longitudinal direction, such as one end being pointed, so that its orientation can be determined. A writing instrument such as a pen or pencil may be used as the operation instruction member. The operation instruction member is usually moved by the operator's hand. The operation instruction member may be about the same size as a writing instrument such as a pen, and may be held and moved by the operator in the same way as a writing instrument. The orientation of the operation instruction member may indicate the orientation of a predetermined part, such as the tip of a virtual tool.

[0068] The method by which the attitude acquisition unit 33 acquires the attitude of the operation instruction member is not limited. Acquiring the attitude of the operation instruction member may, for example, mean specifying the attitude of the operation instruction member in a predetermined coordinate system. That coordinate system may be, for example, the world coordinate system, the local coordinate system of the display device 4, or any other coordinate system. When the attitude acquisition unit 33 acquires the attitude of the operation instruction member, for example, it may acquire the longitudinal angle of the rod-shaped operation instruction member. This attitude may also be represented by, for example, roll, pitch, yaw angle, or Euler angle.

[0069] The posture acquisition unit 33 may acquire the posture of the operation instruction member, for example, by using an image of the operation instruction member. In this case, the image may be taken using, for example, a camera on the display device 4, or a camera placed at a predetermined location in the actual environment. If the head-mounted display device 4 is equipped with a camera for capturing images of the actual environment, it is preferable that the camera is positioned to acquire images in the same direction as the line of sight of the worker wearing the display device 4. Also, if a camera is placed in the actual environment, it is preferable that the camera is positioned to capture the area in which the operation instruction member is moved by the worker. The posture acquisition unit 33 may, for example, identify and track the operation instruction member using the captured image, and then acquire the posture of the operation instruction member using the result.

[0070] Furthermore, the attitude acquisition unit 33 may acquire the attitude of the operation instruction member using, for example, the sensing results from a sensor incorporated into the operation instruction member. This sensor may be, for example, a sensor for acquiring the attitude of the operation instruction member. The sensor for acquiring the attitude may be, for example, an azimuth sensor for acquiring the azimuth angle and an inclination sensor for acquiring the inclination angle relative to the vertical, or it may be a gyro sensor (i.e., an angular velocity sensor). Since a gyro sensor can only detect changes in angle, for example, the operator may initially position the operation instruction member in a predetermined reference position. The attitude acquisition unit 33 may then acquire the current attitude of the operation instruction member using the sensing results acquired by the gyro sensor and the reference position. The sensing results from the sensors on the operation instruction member may be transmitted from the operation instruction member to the attitude acquisition unit 33 wirelessly or via a wired connection.

[0071] Furthermore, the attitude acquisition unit 33 may acquire the attitude of the operation instruction member using sensing results indicating the position of the operation instruction member acquired by, for example, a sensor placed in the actual environment or a sensor on the display device 4. This sensor may be, for example, a distance measuring sensor such as a depth sensor. If the sensor is a distance measuring sensor, a 3D scan of the actual environment may be performed using multiple distance measuring sensors. The attitude acquisition unit 33 may then use the sensing results, i.e., the results of the 3D scan, to determine the attitude of the operation instruction member, which has a predetermined shape.

[0072] Furthermore, the acquisition of the orientation of the operation instruction member using captured images or sensing results may be performed by components or devices other than the orientation acquisition unit 33. In this case, the orientation acquisition unit 33 may accept the orientation of the operation instruction member acquired by components or devices other than the orientation acquisition unit 33. In other words, the acquisition of the orientation of the operation instruction member by the orientation acquisition unit 33 may be performed by accepting the orientation of the operation instruction member.

[0073] The posture acquisition unit 33 may, for example, acquire the posture of the operation instruction member when the reception unit 35 receives an instruction to acquire the desired posture, as the desired posture.

[0074] Furthermore, the posture acquisition unit 33 may acquire the desired posture of the virtual tool, which has been numerically input via an input device or a virtual input interface. Acquisition of the numerical value representing this desired posture may, for example, be done by simply receiving that numerical value.

[0075] Furthermore, obtaining the desired posture for a virtual tool may also involve obtaining the posture of a location that is in a predetermined positional relationship with the virtual tool. However, it is assumed that there is a one-to-one relationship between the posture of that location and the posture of the virtual tool. That is, once the posture of that location is determined, the posture of the virtual tool is also uniquely determined. In this case, obtaining the posture of that location is substantially the same as obtaining the desired posture for the virtual tool. Therefore, obtaining the desired posture for a virtual tool may be considered to include obtaining the posture of such a location. In addition, the reachable range may be determined using the posture of that location. That location may be, for example, a location a predetermined distance away from the virtual tool in a predetermined direction. That location may, as an example, be a location on a virtual robot.

[0076] Furthermore, in the above embodiment, each process or function may be implemented by centralized processing by a single device or a single system, or by distributed processing by multiple devices or multiple systems. For example, at least a part of the configuration of the image generation device 3 may be physically included in a device having a display, etc. Therefore, the device divisions shown in Figure 1 may be considered to be for convenience based on function rather than corresponding to physical devices.

[0077] Furthermore, in the above embodiment, each process or function may be implemented by centralized processing by a single device or a single system, or by distributed processing by multiple devices or multiple systems.

[0078] Furthermore, in the above embodiment, if two or more components included in the image generation device 3 have a communication device, an input device, etc., the two or more components may have a single physical device, or they may have separate devices.

[0079] Furthermore, in the above embodiment, each component may be configured with dedicated hardware, or, if it is a component that can be implemented by software, it may be implemented by executing a program. For example, each component can be implemented by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. During execution, the program execution unit may execute the program while accessing the storage unit or recording medium. The program may also be executed by being downloaded from a server or the like, or by being executed by reading a program recorded on a predetermined recording medium (e.g., an optical disk, magnetic disk, semiconductor memory, etc.). Furthermore, this program may be used as a program that constitutes a program product. Furthermore, the computer executing the program may be one or multiple computers. That is, centralized processing may be performed, or distributed processing may be performed.

[0080] Furthermore, the embodiments described above are illustrative examples for specifically carrying out the present invention and do not limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims rather than by the description of the embodiments, and modifications within the literal scope and equivalent meaning of the claims are intended. [Explanation of symbols]

[0081] 3 Image generation device, 4 Display device, 6 Reference marker, 31 Storage unit, 32 Position relationship acquisition unit, 33 Pose acquisition unit, 34 Identification unit, 35 Reception unit, 36 Image generation unit, 37 Output unit

Claims

1. A memory unit that stores 3D models of virtual robots and virtual tools corresponding to actual robots and tools, A position relationship acquisition unit that acquires the relative positional relationship between a reference marker present in the real environment and a display device that overlays an image onto the image of the real environment or the real environment itself, A posture acquisition unit that acquires the desired posture for the virtual tool, A identifying unit that identifies the reachable range of the virtual tool in the target orientation, which is mounted on a virtual robot positioned to have a predetermined positional relationship with the reference marker, based on a three-dimensional model of the virtual robot to which the virtual tool is attached. An image generation unit generates a display image for showing the reachable range identified by the specific unit based on the relative positional relationship, An image generation device comprising an output unit that outputs the display image to the display device.

2. The image generation apparatus according to claim 1, wherein the identifying unit identifies a range corresponding to the reachable range of the virtual tool for the target posture, which is a set of grid points that can be reached by the virtual tool, among a plurality of grid points arranged at predetermined intervals in three-dimensional space, and which are grid points that can be reached by the virtual tool for the target posture, with the virtual tool attached to the virtual robot positioned so as to have a predetermined positional relationship with the reference marker.

3. The plurality of grid points include a plurality of first grid points arranged at a first interval and a plurality of second grid points arranged at a second interval shorter than the first interval. The image generation apparatus according to claim 2, wherein the identifying unit identifies a first grid point that the virtual tool can reach, identifies a second grid point that the virtual tool can reach between the first grid point that the virtual tool can reach and a first grid point that the virtual tool cannot reach, and identifies a range corresponding to the set of first and second grid points that the virtual tool can reach as the reachable range of the virtual tool in the desired orientation.

4. The image generation apparatus according to claim 2, wherein the specified part is a grid point that the virtual tool can reach, which can calculate the angle of each joint by inverse kinematics at any angle obtained by rotating the virtual tool of the target posture around the Z axis of the tool coordinate system.

5. The image generation apparatus according to claim 2, wherein the specified unit, when a boundary exists in which the angle of at least one joint becomes discontinuous among a plurality of grid points in which the angle of each joint can be calculated by inverse kinematics with the virtual tool of the target posture positioned, sets the grid point on the base end side of the virtual robot with respect to the boundary to a grid point that the virtual tool can reach.

6. The specified unit also identifies a boundary where the angle of at least one joint becomes discontinuous, with respect to a plurality of grid points where the angle of each joint can be calculated by inverse kinematics while the virtual tool for the target posture is positioned. The image generation device according to claim 2, wherein the image generation unit also generates a display image for displaying the identified boundary.

7. The posture acquisition unit acquires two or more desired postures for the virtual tool. The image generation apparatus according to any one of claims 1 to 6, wherein the identifying unit identifies the reachable range of two or more virtual tools for the above-mentioned objective postures.

Images