Image processing apparatus and robot placement method

Abstract

To solve the problem that it is difficult to grasp a situation of an arranged robot from information about a size or the like prescribed in a catalog.SOLUTION: An image processing device 3 comprises: a storage part 31 that stores a three-dimensional model of a virtual robot; a receiving part 32 that receives operation to the three-dimensional model of the virtual robot; a positional relation obtaining part 33 that obtains a relative positional relation between a reference marker existing in an actual environment and a display device 4 that displays an image while superimposing the image on an image of the actual environment or on the actual environment; an image generating part 34 that generates a display image for displaying the operated three-dimensional model so that the model is in a predetermined positional relation with the reference marker, on the basis of the relative positional relation with the three-dimensional model; and an output part 35 that outputs the display image to the display device 4. This enables the actual robot to be arranged after confirming a situation of the arranged actual robot, using the virtual robot.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus for displaying a virtual robot and a robot placement method for placing a real robot.

Background Art

[0002] Conventionally, when introducing a new robot, information such as the size and reach of the robot to be introduced has been confirmed in a catalog to select the robot to be introduced and determine the placement position.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, it has been difficult to know from the catalog information whether the reach of the robot, whether the robot can take a desired posture, whether obstacles existing in the environment where the robot is placed can be avoided, etc. There is also a problem that it is difficult to actually place the robot when selecting the robot. Furthermore, although the space where the robot is to be placed can be converted into three-dimensional data and placement can be considered during simulation, there is also a problem that the cost is high in that case.

[0004] The present invention has been made to solve the above problems, and an object thereof is to provide an image processing apparatus for easily confirming the situation when a robot is placed before placing the real robot, and a robot placement method for placing the real robot.

Means for Solving the Problems

[0005] To achieve the above object, an image processing apparatus according to an aspect of the present invention includes a storage unit that stores a three-dimensional model of a virtual robot, a reception unit that receives an operation on the three-dimensional model of the virtual robot, a reference marker existing in a real environment, and a position relationship acquisition unit that acquires a relative position relationship between a display device that displays an image by superimposing it on an image of the real environment or the real environment itself, and an image generation unit that generates a display image for displaying the operated three-dimensional model so that the reference marker has a predetermined position relationship based on the relative position relationship with the three-dimensional model, and an output unit that outputs the display image to the display device.

Effects of the Invention

[0006] According to an image processing apparatus or the like according to an aspect of the present invention, it is possible to easily confirm in advance using a virtual robot the situation when a robot is placed, for example, the reachable range of the robot, whether the robot can take a desired posture, whether the robot can avoid existing obstacles, and the like. Therefore, according to the confirmation result, for example, a robot suitable for the installation location can be selected, or a real robot can be placed at an appropriate position.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 4C

Embodiments for Carrying Out the Invention

[0008] Hereinafter, an image processing apparatus and a robot placement method according to the present invention will be described using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or corresponding, and repeated descriptions may be omitted. The image processing apparatus according to the present embodiment displays a three-dimensional model of a virtual robot so as to have a predetermined positional relationship with a reference marker.

[0009] FIG. 1 is a block diagram showing the configuration of a robot control system 100 according to the present embodiment. The robot control system 100 according to the present embodiment is for operating a virtual robot, and includes an image processing apparatus 3, a display apparatus 4, and a sheet 6a on which a reference marker 6 is displayed. Note that the image processing apparatus 3 and the display apparatus 4 may be connected by wire or wirelessly, for example.

[0010] The virtual robot to be operated is constituted by a three-dimensional model existing in a virtual environment and corresponds to an actual robot. That is, the virtual robot is the same as the actual robot except that it is constituted by a three-dimensional model. For example, it may have the same size and configuration as the actual robot, and may be able to change the angles of joints of a plurality of arms, etc., in the same manner as the actual robot.

[0011] The actual robot is usually an industrial robot, and may be a manipulator having a plurality of arms (links) connected by joints driven by motors. The actual robot may be, for example, a vertical articulated robot or a horizontal articulated robot. Also, the actual robot may be, for example, a transfer robot, a welding robot, an assembly robot, a painting robot, or a robot for other uses. Note that the actual robot is a robot existing in the real environment. The real environment means the environment of the real space.

[0012] The image processing device 3 generates a display image for displaying a three-dimensional model of a virtual robot so as to have a predetermined positional relationship with the reference marker 6, and outputs the display image to the display device 4. Details of the image processing device 3 will be described later.

[0013] The display device 4 overlays and displays an image on the image of the real environment or the real environment itself. That is, an operator who operates the virtual robot can view both the real environment and the image of the virtual environment through the display device 4. The display device 4 may be a wearable display device worn on the head by an operator who operates the virtual robot, or may be a display device that is a portable information processing terminal such as a tablet terminal. The wearable display device may be, for example, a head-mounted display. Further, the display device 4 may have, for example, a transmissive display. In this case, the display device 4 will overlay and display the image on the real environment itself. As the wearable display device 4 having a transmissive display, for example, HoloLens (registered trademark) etc. are known. It can also be considered that the display device 4 having such a transmissive display is a display device for realizing mixed reality (MR). Further, the display device 4 may have, for example, a non-transmissive display. In this case, the display device 4 will overlay and display the image on the image of the real environment. Therefore, it is preferable that the display device 4 having a non-transmissive display has a camera for photographing the real environment or is connected to a camera for photographing the real environment. The image of the real environment photographed by the camera is displayed on the non-transmissive display in real time. As the wearable display device 4 having a non-transmissive display, for example, Oculus Quest etc. are known. It can also be considered that the display device 4 having such a non-transmissive display is a display device for realizing augmented reality (AR). The display device 4 which is a portable information processing terminal such as a tablet terminal may have, for example, a camera and a display, and may display the image of the real environment photographed by the camera on the display in real time. In the present embodiment, the case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

[0014] The reference marker 6 is a two-dimensional predetermined image. The reference marker 6 may be, for example, an AR marker, a QR code (registered trademark), or other two-dimensional images with a predetermined shape. The size of the reference marker 6 may be predetermined, for example. In the present embodiment, the case where the reference marker 6 is displayed on the sheet 6a will be mainly described. The reference marker 6 may be printed on a sheet 6a made of paper or resin, for example. Also, on the sheet 6a, one or more figures 6b used when arranging the actual robot may be displayed, for example. This figure 6b is used for positioning the actual robot and may be, for example, a figure indicating the position of a screw hole for fixing the actual robot, a figure indicating the position of the end portion on the proximal side of the actual robot, or other figures used for positioning when arranging the actual robot. The figure 6b is not particularly limited as long as it is a figure used for positioning, and may be, for example, a figure such as a circle, a square, a triangle, a polygon, or a figure such as an × (multiplication sign) or a + (addition sign), or other figures capable of specifying a position. By arranging the actual robot using this figure 6b, it is possible to prevent the position of the displayed virtual robot from deviating from the arrangement position of the actual robot. The figure 6b is used to arrange the actual robot at the position of the virtual robot displayed so as to have a predetermined positional relationship with the reference marker 6. Therefore, it is assumed that the figure 6b is displayed in a predetermined positional relationship with the reference marker 6 so that the actual robot can be arranged in this way.

[0015] As shown in FIG. 1, the image processing apparatus 3 according to the present embodiment includes a storage unit 31, a reception unit 32, a positional relationship acquisition unit 33, an image generation unit 34, and an output unit 35.

[0016] In the storage unit 31, a three-dimensional model of a virtual robot is stored. This three-dimensional model of the virtual robot may be, for example, a three-dimensional model of a virtual robot corresponding to an actual robot under consideration for introduction. In the storage unit 31, three-dimensional models of a plurality of virtual robots may be stored. The three-dimensional models of the plurality of virtual robots respectively correspond to a plurality of actual robots. For example, when there are a plurality of candidate actual robots for introduction, three-dimensional models of a plurality of virtual robots corresponding to those plurality of actual robots may be stored in the storage unit 31. Also, information other than the three-dimensional model may be stored in the storage unit 31. For example, teaching data or the like may be stored in the storage unit 31. The process by which information is stored in the storage unit 31 is not limited. For example, information may be stored in the storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage unit 31. The storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0017] The reception unit 32 receives operations on the three-dimensional model of the virtual robot. The operation on the three-dimensional model may be, for example, an operation for changing the position and orientation of at least a part of the three-dimensional model. The reception of operations and teaching instructions may be performed, for example, via an input device such as a teaching pendant existing in the real environment, or via a virtual input interface such as a virtual button or a virtual teaching pendant displayed on the display of the display device 4. The virtual input interface is displayed on the display of the display device 4 in response to an operation such as an air tap, for example, and when a button or the like is selected by the operator's finger or pointing device, an input corresponding to the operation of the button or the like is passed from the display device 4 to the reception unit 32. Also, at least a part (for example, the tip of a tool or the like) of the three-dimensional model of the virtual robot may be changed in position and orientation by a gesture such as the operator's hand. In this case, for example, when the operator performs an operation of pinching the three-dimensional model displayed on the display with the hand (hold operation), the part pinched by the hand is specified as the operation target, and by changing the position and orientation of the hand, an operation for changing the position and orientation of the specified target is performed, and by ending the pinching operation, the operation on the specified target may be ended. In this case, for example, information indicating the operation target (for example, information indicating the position and part in the three-dimensional model) and information indicating the content of the operation (for example, information indicating a change in position or orientation) may be passed from the display device 4 to the reception unit 32, or the result of hand tracking of the operator's hand photographed by the camera may be passed to the reception unit 32, and in the image processing device 3, the operation target and the content of the operation may be specified. When the display device 4 acquires information indicating the operation target and information indicating the content of the operation, the display device 4 may be able to access the information of the current three-dimensional model of the virtual robot in the virtual space held by the image processing device 3. When an operation using the operator's hand is performed, the display device 4 has a camera, and by performing hand tracking of the operator's hand photographed by the camera, it may be specified where on the display the operator's hand is located.Also, a method of converting an operation on a display image of a three-dimensional model according to a gesture such as an operator's hand into a change in the position and orientation of the three-dimensional model in a three-dimensional virtual space is already known, and a detailed description thereof will be omitted. Further, the reception unit 32 may receive, for example, an instruction for teaching a three-dimensional model of a virtual robot, or may receive an instruction to change the three-dimensional model of the displayed virtual robot.

[0018] The reception unit 32 may receive, for example, information input from an input device or a display device 4, or may receive information transmitted via a wired or wireless communication line. Note that the reception unit 32 may or may not include a device for reception (for example, an input device or a communication device, etc.). Further, the reception unit 32 may be realized by hardware, or may be realized by software such as a driver for driving a predetermined device.

[0019] The position relationship acquisition unit 33 acquires the relative position relationship between the reference marker 6 existing in the real environment and the display device 4. Acquiring the relative position relationship between the reference marker 6 and the display device 4 may be, for example, acquiring the relative position relationship between the marker coordinate system, which is the local coordinate system of the reference marker 6, and the display coordinate system, which is the local coordinate system of the display device 4. This relative position relationship may be represented by, for example, a homogeneous transformation matrix indicating the transformation between the two coordinate systems. The method by which the position relationship acquisition unit 33 acquires this relative position relationship is not limited. The position relationship acquisition unit 33 may receive, for example, an image captured by a camera of the display device 4, and use three or more feature points of the reference marker 6 included in the image to acquire a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system. The acquisition of the homogeneous transformation matrix may be performed by the display device 4. In this case, the position relationship acquisition unit 33 may receive the homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system from the display device 4. That is, the acquisition of the relative position relationship by the position relationship acquisition unit 33 may be the reception of the relative position relationship.

[0020] The image generation unit 34 generates a display image for displaying the three-dimensional model of the virtual robot based on the three-dimensional model of the virtual robot and the relative positional relationship between the reference marker 6 acquired by the positional relationship acquisition unit 33 and the display device 4 so that the reference marker 6 has a predetermined positional relationship. Further, when an operation on the virtual robot is received, the image generation unit 34 generates a display image for displaying the operated three-dimensional model. The predetermined positional relationship may be, for example, a predetermined positional relationship, or may be a positional relationship that can be changed by an operator who operates the virtual robot. The predetermined positional relationship may be, for example, a positional relationship in which the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6, or may be a positional relationship in which the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6. In the former case, the image generation unit 34 may generate a display image for displaying the three-dimensional model of the virtual robot at the position of the reference marker 6. That the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6 may mean that the three-dimensional model of the virtual robot is displayed so as to virtually reproduce the situation where the actual robot is arranged at the position of the reference marker 6. For example, the end face on the base end side (for example, the mounting surface on the floor surface, etc.) of the three-dimensional model of the virtual robot may be displayed so as to coincide with the surface of the reference marker 6. That the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6 may mean, for example, that the three-dimensional model of the virtual robot is displayed next to the reference marker 6. In addition, when the three-dimensional model of the virtual robot is displayed next to the reference marker 6, usually, on the sheet 6a, a figure 6b is displayed next to the reference marker 6. In any case, the three-dimensional model of the virtual robot is displayed so that the positional relationship with the reference marker 6 arranged in the actual environment does not change. Therefore, even if the operator changes the orientation of the display device 4, the display position of the three-dimensional model of the virtual robot in the actual environment does not change.

[0021] The relative positional relationship between the reference marker 6 and the display device 4 is acquired by the positional relationship acquisition unit 33. Also, the positional relationship between the reference marker 6 and the three-dimensional model of the virtual robot is determined. Therefore, using this information, the image generation unit 34 can specify the relative positional relationship between the three-dimensional model of the virtual robot and the display device 4. For this reason, the image generation unit 34 can arrange the three-dimensional model of the virtual robot in the virtual space and specify the position and orientation of the display device 4 that have the specified positional relationship with respect to the three-dimensional model of the virtual robot. Then, the image generation unit 34 can generate a two-dimensional display image for displaying the three-dimensional model by rendering the three-dimensional model of the virtual robot in the virtual space with reference to the position and orientation of the display device 4. Note that the angle of each joint of the virtual robot is the initial value when no operation is performed, and is the value after the operation when an operation is performed. The angle of each joint after the operation may be calculated, for example, by inverse kinematics using the position and posture of the end effector in the three-dimensional model of the virtual robot after the operation, similar to the real robot. When the virtual robot is operated, as described above, the shape of the three-dimensional model of the virtual robot in the virtual space is changed accordingly. Also, when the position or orientation of the display device 4 changes in the real environment, the position and direction of the viewpoint in the virtual space are changed accordingly. Then, by performing rendering after the change, a display image of the three-dimensional model after the operation and a display image of the three-dimensional model after the change in the position or orientation of the display device 4 are generated. Note that the image generation unit 34 generates the display image so that the size of the display image matches the real environment when the display image of the three-dimensional model of the virtual robot is displayed on the display of the display device 4. That is, the display image is generated so that the three-dimensional model of the virtual robot displayed on the display of the display device 4 and the real robot arranged in the real environment so as to have the same relative positional relationship as that three-dimensional model have the same size when viewed through the display device 4.

[0022] In addition, for example, when the reception unit 32 receives an instruction to change the three-dimensional model of the virtual robot to be displayed, the image generation unit 34 may change the three-dimensional model of the virtual robot to be displayed according to the instruction. For example, when the three-dimensional models of a plurality of virtual robots are stored in the storage unit 31, the image generation unit 34 may generate a display image using the three-dimensional model of a virtual robot different from the previous one according to an instruction to change the display target. In addition, when the instruction includes information for identifying the three-dimensional model of the virtual robot to be displayed, a display image may be generated using the three-dimensional model of the virtual robot identified by the information.

[0023] Also, it may not be possible to operate the virtual robot according to the received operation. For example, an operation to move the end effector of the virtual robot beyond the movable range or an operation to rotate it beyond the rotatable range may be received. In such a case, the image generation unit 34 may not generate a display image corresponding to the operation, or may generate a display image of the three-dimensional model of the virtual robot that has moved or rotated within the possible range.

[0024] The output unit 35 outputs the display image generated by the image generation unit 34 to the display device 4. Note that the output unit 35 may output only the display image. In this case, on the display device 4, the display image will be overlaid and displayed on the image of the real environment or the real environment itself. On the other hand, when the display device 4 has a non-transmissive display and the image of the real environment captured by the display device 4 is received by the image processing device 3, the result of synthesizing the image of the real environment and the display image may be output to the display device 4. This synthesis may be performed, for example, by a synthesis unit (not shown) included in the image processing device 3.

[0025] When the three-dimensional model of the virtual robot is displayed at a position different from the reference marker 6, for example, as shown in FIG. 1, the three-dimensional model 10 of the virtual robot may be displayed next to the reference marker 6. The reference marker 6 and the three-dimensional model 10 of the virtual robot in FIG. 1 schematically show the situation seen by the operator operating the virtual robot through the display device 4. In this case, for example, the figure 6b may not be displayed on the sheet 6a.

[0026] Note that the display of the three-dimensional model of the virtual robot and the operation of the three-dimensional model of the virtual robot using the virtual input interface of the display device, the gestures of the operator, etc. are already known as shown in, for example, the following Patent Documents 1 to 3, and detailed descriptions thereof are omitted.

Prior Art Documents

Patent Documents

[0027]

Patent Document 1

Patent Document 2

Patent Document 3

[0028] Next, the operation of the image processing apparatus 3 will be described with reference to the flowchart of FIG. 2.

[0029] (Step S101) The reception unit 32 determines whether an operation on the three-dimensional model of the virtual robot has been received. If an operation has been received, the process proceeds to step S103; otherwise, the process proceeds to step S102.

[0030] (Step S102) The image generation unit 34 determines whether to generate a display image. If it is determined to generate a display image, the process proceeds to step S103; otherwise, the process returns to step S101. Note that the image generation unit 34 may make this determination periodically, for example, when generating a display image. By making this determination, for example, even if no operation is being performed, when the position or orientation of the display device 4 is changed, a display image corresponding to the changed position or orientation will be displayed on the display device 4.

[0031] (Step S103) The position relationship acquisition unit 33 acquires the relative position relationship between the reference marker 6 and the display device 4.

[0032] (Step S104) Using the relative position relationship acquired in step S103 and the 3D model of the virtual robot, the image generation unit 34 generates a display image for displaying the 3D model such that the reference marker 6 has a predetermined position relationship. Note that when an operation is received, a display image for displaying the 3D model after being changed according to the operation will be generated. Also, when the position or orientation of the display device 4 is changed, a display image of the 3D model corresponding to the changed position or orientation of the display device 4 will be generated.

[0033] (Step S105) The output unit 35 outputs the generated display image to the display device 4. Then, the process returns to step S101. In response to this output, the 3D model of the virtual robot will be displayed such that the reference marker 6 has a predetermined position relationship.

[0034] Note that the order of the processes in the flowchart of FIG. 2 is an example, and if the same result can be obtained, the order of each step may be changed. Also, in the flowchart of FIG. 2, the process ends due to a power-off or a processing end interrupt.

[0035] Next, the method of arranging the actual robot will be described using the flowchart of FIG. 3.

[0036] (Step S201) The operator places the reference marker 6, which is a two-dimensional predetermined image, in the actual environment. The reference marker 6 may be placed, for example, at the position where the actual robot is planned to be placed. Also, for example, if another actual robot is placed at the position where the actual robot is planned to be placed, it may be placed next to the existing actual robot.

[0037] (Step S202) The operator operates the three-dimensional model of the virtual robot displayed on the display device 4 using the image processing device 3 and the display device 4 so that it has a predetermined positional relationship with the reference marker 6. This operation may be performed, for example, for the purpose of checking the reachable range of the robot, checking whether the robot can take a desired posture, checking whether the robot can avoid existing obstacles, etc. When the reference marker 6 is placed at the position where the actual robot is planned to be placed, the three-dimensional model of the virtual robot may be displayed at the position of the reference marker 6. Also, when the reference marker 6 is placed next to the existing actual robot, the three-dimensional model of the virtual robot may be displayed at the new placement position of the new robot that has a predetermined positional relationship with the reference marker 6, for example, at the position where the existing actual robot is placed.

[0038] (Step S203) The operator determines whether the placement of the virtual robot and the type of the virtual robot are appropriate. For example, if there is no problem with the reachable range of the robot, the robot can take the desired posture, and the robot can avoid existing obstacles, the operator may determine that the placement and type of the virtual robot are appropriate. On the other hand, if there is a problem with the reachable range of the robot, the robot cannot take the desired posture, or the robot cannot avoid existing obstacles, the operator may determine that the placement or type of the virtual robot is inappropriate. And if the placement of the virtual robot, that is, the placement of the reference marker 6, is inappropriate, the process returns to step S201, and the placement of the reference marker 6 is done again. Also, if the type of the virtual robot is inappropriate, the type of the virtual robot may be changed and the operation may be performed again. In this case, the type of the virtual robot is changed and the process returns to step S202, and the operation of the new virtual robot is performed. On the other hand, if the placement and type of the virtual robot are appropriate, the process proceeds to step S204.

[0039] (Step S204) After the operation of the 3D model of the virtual robot, the actual robot is placed so as to have a predetermined positional relationship with the reference marker 6 arranged in the actual environment. This actual robot corresponds to the 3D model of the operated virtual robot. For example, if a figure 6b for specifying the placement position of the actual robot is displayed on the sheet 6a on which the reference marker 6 is displayed, a screw hole for placing the actual robot may be provided at the position of the figure 6b, and the actual robot may be fixed to the floor surface or the like using the screw hole. In this way, the actual robot can be placed at the same position as the virtual robot used for the operation confirmation, and the placement of the actual robot at an appropriate position can be realized.

[0040] Although not included in the flowchart of FIG. 3, teaching using a virtual robot may be performed. In this case, for example, in a situation where the reference marker 6 is arranged at an appropriate position, that is, in a situation where the virtual robot is displayed at the same position as the installation position of the actual robot where the virtual robot is to be introduced, the three-dimensional model of the virtual robot may be operated and teaching instructions may be input as appropriate. Then, teaching data may be stored in the storage unit 31 according to the teaching instructions. Further, the teaching data may be passed to a robot control device that controls the actual robot. And after the actual robot is arranged using the reference marker 6, the actual robot may be operated using the teaching data. In this case, teaching data can be prepared in advance before the introduction of the actual robot, and a playback operation using the teaching data can be started immediately after the introduction of the actual robot.

[0041] Next, the robot placement method according to the present embodiment and the operation of the image processing apparatus 3 will be described using a specific example. In this specific example, it is assumed that a three-dimensional model of a virtual robot corresponding to the actual robot 1 to be introduced is stored in advance in the storage unit 31 of the image processing apparatus 3.

[0042] First, as shown in FIG. 4A, the operator attaches the sheet 6a on which the reference marker 6 is displayed to the position where the actual robot 1 to be introduced is to be arranged (step S201). It is assumed that four figures 6b for specifying the arrangement position of the robot 1 are displayed on the sheet 6a.

[0043] Next, the operator wears the head-mounted display device 4 on the head and operates the three-dimensional model 10 of the virtual robot displayed by the image processing device 3 (step S202). Specifically, when it is determined that the display image is generated by the image generation unit 34 (step S102), the relative positional relationship between the reference marker 6 and the display device 4 is acquired accordingly (step S103). Using the three-dimensional model stored in the storage unit 31 and the relative positional relationship, a display image for displaying the three-dimensional model is generated and output to the display device 4 (steps S104, S105). In this way, as shown in FIG. 4B, the three-dimensional model 10 of the virtual robot is displayed at the position of the reference marker 6.

[0044] Also, when the operator operates the three-dimensional model of the virtual robot by using a teaching pendant, a virtual input interface, or a gesture operation using the hand, a display image corresponding to the operation is generated and displayed (steps S101, S103 to S105). This operation may be, for example, an operation of the end effector of the virtual robot. The end effector of the virtual robot may be, for example, a TCP (Tool Center Point). In this way, the operator can use the virtual robot to confirm the situation when the robot is placed, for example, the reachable range of the robot, whether the robot can take a desired posture, and whether the robot can avoid existing obstacles. If the placement position of the robot is not appropriate, the attachment position of the sheet 6a displaying the reference marker 6 is changed, and the placement situation of the robot is confirmed again. Also, if the type of the robot is not appropriate, the type of the virtual robot to be displayed is changed, and the placement situation of the robot is confirmed again.

[0045] If there are no problems with the arrangement status and type of the virtual robot (step S203), the operator ends the display of the virtual robot using the image processing device 3 and the display device 4, and provides bolt holes for fixing the robot 1, which is the actual robot, at the positions of the figures 6b of the sheet 6a on which the reference marker 6 is displayed. Then, as shown in FIG. 4C, the robot 1 to be introduced is arranged at the position of the reference marker 6, and the end portion on the proximal end side of the robot 1 is fixed to the bolt hole with the bolt 1a (step S204). In this way, the robot 1, which is the actual robot, can be arranged at the position where the operation has been confirmed in advance by the virtual robot.

[0046] As described above, according to the image processing device 3 and the robot arrangement method according to the present embodiment, the situation where the robot to be introduced is arranged can be confirmed using the virtual robot before the actual robot is arranged. Therefore, after confirming in advance the arrangement position of the robot to be introduced and the type of the robot to be introduced, etc., the actual robot can be arranged, and an unexpected situation can be prevented from occurring after the actual robot is arranged. Further, when the reference marker 6 is a two-dimensional image, the display position of the virtual robot can be easily changed according to the change in the arrangement position of the reference marker 6 in the actual environment. Further, when the reference marker 6 is displayed on the sheet 6a and the figure 6b for specifying the arrangement position of the actual robot is also displayed on the sheet 6a, the actual robot can be arranged at the same position as the display position of the virtual robot where the operation has been confirmed by using the figure 6b.

[0047] In the present embodiment, the case where the three-dimensional model of the virtual robot is displayed at the position of the reference marker 6 has been mainly described, but it is not necessary. For example, the three-dimensional model of the virtual robot may be displayed at a predetermined position in the marker coordinate system of the reference marker 6. Further, the three-dimensional model of the virtual robot may be displayed at a predetermined position and a predetermined posture in the marker coordinate system. In this case, when the actual robot is arranged, the actual robot is also arranged so as to be at a predetermined position and a predetermined posture with respect to the reference marker 6.

[0048] Also, in this embodiment, although the case where the 3D model of the virtual robot is displayed after the reference marker 6 is arranged in the real environment has been mainly described, it may not be so. For example, after the 3D model of the virtual robot is displayed and the operator moves the display position to a desired position, the reference marker 6 may be arranged. In this case, the positional relationship between the reference marker 6 when the reference marker 6 is arranged and the 3D model of the virtual robot may be the predetermined positional relationship between the reference marker and the 3D model of the virtual robot. Also, until the reference marker 6 is arranged, the positional relationship acquisition unit 33 acquires the relative positional relationship between a predetermined object existing in the real environment and the display device 4, and uses the relative positional relationship to display the 3D model of the virtual robot so that the positional relationship between the 3D model of the virtual robot and the predetermined object becomes constant. Note that the positional relationship between the 3D model of the virtual robot and the predetermined object can be appropriately changed by the operator's operation. The predetermined object may be, for example, a real robot existing in the real environment, a jig, or other objects.

[0049] Also, in this embodiment, the type of the virtual robot to be displayed may be specified by the reference marker 6. For example, when the reference marker 6 is a 2D code such as an AR marker or a QR code (registered trademark), the identifier of the 2D code is associated with the type of the virtual robot, and the image generation unit 34 receives the image of the reference marker 6 from the display device 4 or the identifier of the 2D code that is the reference marker 6, and may generate a display image using the 3D model of the virtual robot corresponding to the identifier of the 2D code that is the reference marker 6. In this case, the operator has a plurality of sheets 6a on which the reference marker 6 for each virtual robot is displayed, and can easily change the virtual robot to be displayed by appropriately changing the sheet 6a to be arranged in the real environment. Note that the identifier of the 2D code may be, for example, information obtained by reading the 2D code, or an identifier associated with the 2D code.

[0050] Also, in this embodiment, the case where the reference marker 6 is a two-dimensional predetermined image has been mainly described, but it may not be so. Similar to markerless AR, an object existing in the real environment may be used as the reference marker. The object existing in the real environment is not particularly limited, and for example, it may be a base portion used for installing a real robot, a jig existing in the vicinity of the position where the robot is arranged, or the like. Also, the marker on an object having a three-dimensional shape (for example, a rectangular parallelepiped shape) with markers attached to each surface may be used as the reference marker. Further, the reference marker may be three-dimensional, such as a base or a jig.

[0051] Also, when the three-dimensional model of the virtual robot is being operated, the image processing device 3 may perform a collision determination between the three-dimensional model and an object existing in the real environment. By performing such a collision determination, an operator operating the virtual robot can easily know whether at least a part of the three-dimensional model of the virtual robot has interfered with an obstacle in the real environment.

[0052] Also, in the above embodiment, each process or each function may be realized by being centrally processed by a single device or a single system, or may be realized by being distributedly processed by a plurality of devices or a plurality of systems. For example, at least a part of the configuration of the image processing device 3 may physically be included in a device having a display or the like. Therefore, it may be considered that the separation of the devices shown in FIG. 1 is for convenience according to the function, not according to the physical device.

[0053] In the above-described embodiment, when the information transfer between each component is performed, for example, if the two components that transfer the information are physically different, it may be performed by the output of information by one component and the reception of information by the other component. Or, if the two components that transfer the information are physically the same, it may be performed by shifting from the processing phase corresponding to one component to the processing phase corresponding to the other component.

[0054] In the above-described embodiment, information related to the processing executed by each component, for example, information received, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, mathematical formulas, addresses, etc. used in the processing by each component may be temporarily or permanently stored in a recording medium not shown even if not specified in the above description. Also, the accumulation of information on the recording medium not shown may be performed by each component or an accumulation unit not shown. Further, the reading of information from the recording medium not shown may be performed by each component or a reading unit not shown.

[0055] In the above-described embodiment, when information used in each component, etc., for example, information such as thresholds, addresses, and various setting values used in the processing by each component may be changed by the user, even if not specified in the above description, the user may be allowed to appropriately change that information, or not. When the user can change that information, the change may be realized, for example, by a reception unit not shown that receives a change instruction from the user and a change unit not shown that changes the information in response to the change instruction. The reception of the change instruction by the reception unit not shown may be, for example, reception from an input device, reception of information transmitted via a communication line, or reception of information read from a predetermined recording medium.

[0056] In addition, in the above-described embodiment, when two or more components included in the image processing apparatus 3 have a communication device, an input device, or the like, the two or more components may physically have a single device, or may have separate devices.

[0057] In addition, in the above-described embodiment, each component may be configured by dedicated hardware, or for components that can be realized by software, they may be realized by executing a program. For example, each component can be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. At the time of its execution, the program execution unit may execute the program while accessing a storage unit or a recording medium. Also, the program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). Further, this program may be used as a program constituting a program product. Also, the computer that executes the program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

[0058] Also, the present invention is not limited to the above-described embodiments, and various modifications are possible, and it goes without saying that those are also included within the scope of the present invention.

Description of Reference Numerals

[0059] 3 Image processing apparatus, 4 Display device, 6 Reference marker, 31 Storage unit, 32 Reception unit, 33 Position relationship acquisition unit, 34 Image generation unit, 35 Output unit, 100 Robot control system

Claims

1. a storage unit that stores a 3D model of a virtual robot; a reception unit that receives an operation on the 3D model of the virtual robot; a reference marker existing in the real environment, and a position relationship acquisition unit that acquires the relative position relationship between the display device that overlays and displays an image on the image of the real environment or the real environment itself; an image generation unit that generates a display image for displaying the operated 3D model so that the reference marker has a predetermined position relationship based on the 3D model and the relative position relationship; an output unit that outputs the display image to the display device, and is provided with, the reference marker is a two-dimensional predetermined image displayed on the sheet, and is arranged so as to have a predetermined position relationship with the position where the real robot is to be arranged, the predetermined position relationship is a predetermined position relationship or a position relationship that can be changed by an operator who operates the virtual robot, an image processing apparatus.

2. The image processing apparatus according to claim 1, wherein a figure for specifying the arrangement position of the real robot is displayed on the sheet.

3. The image processing apparatus according to claim 1 or claim 2, wherein the image generation unit generates a display image for displaying the 3D model of the virtual robot at the position of the reference marker.

4. a step of arranging a reference marker, which is a two-dimensional predetermined image, in the real environment; a step of operating the 3D model of the virtual robot so as to be displayed in a predetermined position relationship with the reference marker on a display device that overlays and displays an image on the image of the real environment or the real environment itself; after the operation of the 3D model, a step of arranging a real robot corresponding to the 3D model of the virtual robot so as to have the predetermined position relationship with the reference marker in the real environment, and comprising, the predetermined position relationship is a predetermined position relationship or a position relationship that can be changed by an operator who operates the virtual robot, a robot arrangement method.

Images