Image Processing Device

Abstract

To solve the problem that is difficult to intuitively perform operation to a virtual robot, because an operator cannot sense touch to the virtual robot.SOLUTION: An image processing device 3 comprises: a storing part 31 that stores a three-dimensional model of a virtual robot; an operation obtaining part 34 that obtains a position of an operation instructing member existing in an actual environment and obtains operation to a portion corresponding to the operation instructing member in the three-dimensional model; a positional relation obtaining part 32 that obtains a relative positional relation between the actual environment and a display device 4 that displays an image while superimposing the image on an image of the actual environment or on the actual environment; an image generating part 35 that generates a display image for displaying the operated three-dimensional model, on a predetermined position in the actual environment, on the basis of the three-dimensional model and the relative positional relation; and an image output part 36 that outputs the display image to the display device 4. Thus, the virtual robot can be operated using the operation instructing member in the actual environment, which enables operation to the virtual robot to be performed more intuitively.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that generates a display image of a three-dimensional model of a virtual robot to be operated. [Background technology]

[0002] A worker may operate a real robot by operating a three-dimensional model of a virtual robot through gesture operations (see, for example, Patent Document 1). Then, for example, the worker may press or grab and move a tool at the tip of the three-dimensional model or other parts with his / her fingers, thereby directly teaching the real robot in the virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-055075 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when operating a 3D model of a virtual robot, the operator does not have the sense of touch, making it difficult to intuitively operate the robot. For example, when directly teaching a robot by moving a tool of the real robot by hand, the operator feels as if he is holding the tool in his hand, and when changing the position or posture of the tool by hand, the operator can check whether the intended change has been realized by the sense of his hand. On the other hand, when operating a 3D model in a virtual space, the operator cannot feel the grip of the tool, so he has to judge whether the intended change has been realized only by looking at the shape of the displayed 3D model, making it difficult to judge whether the intended change has been realized. As a result, the operator's operation may not have the intended result.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing device that enables an operator to perform operations that are closer to those of a real robot, even when operating a three-dimensional model of a virtual robot in a virtual space. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, an image processing device according to one embodiment of the present invention includes a memory unit in which a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment is stored; an operation acquisition unit that acquires the position of an operation instruction member existing in the real environment and acquires an operation for a location in the three-dimensional model of the virtual robot corresponding to the operation instruction member in accordance with the position; a positional relationship acquisition unit that acquires the relative positional relationship between the real environment in which the real robot exists and a display device that displays an image superimposed on an image of the real environment or the real environment itself; an image generation unit that generates a display image for displaying the operated three-dimensional model at a predetermined position in the real environment based on the relative positional relationship with the three-dimensional model; and an image output unit that outputs the display image to the display device. Effect of the Invention

[0007] According to an image processing device of an aspect of the present invention, an operation on a three-dimensional model of a virtual robot can be acquired in response to an operation by an operator using an operation instructing member present in a real environment, and a display image can be generated accordingly. Therefore, since the operator can operate the three-dimensional model of the virtual robot by changing the position and posture of the operation instructing member, the operator can feel closer to operating a real robot, even though the operation is performed on the three-dimensional model of the virtual robot in a virtual space. As a result, the possibility that the result of the operation by the operator will be unexpected can be reduced. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a robot control system according to an embodiment of the present invention; [Diagram 2] A flowchart showing the operation of the image processing device according to the embodiment. [Figure 3A] FIG. 13 is a diagram for explaining an example of an operation in the embodiment; [Figure 3B] FIG. 13 is a diagram for explaining an example of an operation in the embodiment; [Figure 3C] FIG. 13 is a diagram for explaining an example of an operation in the embodiment; [Figure 4] FIG. 2 is a diagram showing an example of a relationship between an operation instruction member and a three-dimensional model in the embodiment; [Diagram 5] FIG. 2 is a diagram showing an example of a relationship between an operation instruction member and a three-dimensional model in the embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The image processing device according to the present invention will be described below using an embodiment. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated description may be omitted. The image processing device according to this embodiment acquires operations for a three-dimensional model of a virtual robot according to the position of an operation instruction member, etc.

[0010] FIG. 1 is a schematic diagram showing a configuration of a robot control system 100 according to the present embodiment. The robot control system 100 according to the present embodiment is for operating a virtual robot, and includes an image processing device 3 and a display device 4. The image processing device 3 and the display device 4 may be connected, for example, by wire or wirelessly. In addition, in the case where a robot 1, which is a real robot, is operated by operating a three-dimensional model of a virtual robot, the robot control system 100 may further include, for example, the robot 1 and a robot control device that controls the operation of the robot 1. The operation of the robot 1 via the three-dimensional model of the virtual robot may be performed, for example, for teaching purposes, or may be performed for confirmation of a reachable range, etc. In the case where the robot 1 is operated by operating a three-dimensional model, for example, the robot 1 may be operated so that the angle of each joint of the robot 1 is the same as the angle of each joint of the three-dimensional model.

[0011] The virtual robot to be operated is configured as a three-dimensional model existing in a virtual environment, and corresponds to a real robot existing in a real environment. In other words, the virtual robot is the same as a real robot except that it is configured as a three-dimensional model, and may, for example, be the same size and configuration as a real robot, and may be able to change the angles of the joints of multiple arms, just like a real robot.

[0012] The real robot is usually an industrial robot, and may be a manipulator having a plurality of arms (links) connected by joints driven by a motor. The real robot may be, for example, a vertical articulated robot or a horizontal articulated robot. The real robot may be, for example, a transport robot, a welding robot, an assembly robot, a painting robot, or a robot for other purposes. The real robot is a robot that exists in a real environment. The real environment is an environment in a real space. The robot control device controls the operation of the robot 1. The robot control device that operates the robot 1 is already known, and a detailed description thereof will be omitted. In addition, when the robot control system 100 has the robot 1 that is, for example, a welding robot, the robot control system 100 may further have a welding power source, a wire feeder, and the like, as necessary.

[0013] The image processing device 3 generates a display image for displaying a three-dimensional model of a virtual robot at a predetermined position in the real environment, and outputs the display image to the display device 4. The display image of the three-dimensional model of the virtual robot may be generated so that the three-dimensional model of the virtual robot and the robot 1, which is a real robot existing in the real environment, have a predetermined positional relationship. Details of the image processing device 3 will be described later.

[0014] The display device 4 displays an image by superimposing it on an image of the real environment or the real environment itself. That is, the worker who operates the virtual robot can see both the real environment and the virtual environment images by the display device 4. The display device 4 may be a wearable display device that the worker who operates the virtual robot wears on his head, or may be a display device that is a portable information processing terminal such as a tablet terminal. The wearable display device may be, for example, a head-mounted display. The display device 4 may also have, for example, a transmissive display. In this case, the display device 4 displays an image by superimposing it on the real environment itself. As a wearable display device 4 having a transmissive display, for example, HoloLens (registered trademark) is known. The display device 4 having such a transmissive display can also be considered as a display device for realizing mixed reality (MR). The display device 4 may also have, for example, a non-transmissive display. In this case, the display device 4 displays an image by superimposing it on an image of the real environment. Therefore, it is preferable that the display device 4 having a non-transmissive display has a camera for photographing the real environment or is connected to a camera for photographing the real environment. An image of the real environment captured by the camera is displayed on a non-transmissive display in real time. For example, Oculus Quest is known as a wearable display device 4 having a non-transmissive display. Such a display device 4 having a non-transmissive display can be considered as a display device for realizing augmented reality (AR). The display device 4, which is a portable information processing terminal such as a tablet terminal, may have, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. In this embodiment, a case where the display device 4 is a head-mounted display having a transmissive display will be mainly described.

[0015] As shown in FIG. 1, the image processing device 3 according to this embodiment includes a storage unit 31, a positional relationship acquisition unit 32, a reception unit 33, an operation acquisition unit , an image generation unit 35, and an image output unit .

[0016] The storage unit 31 stores a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment. This three-dimensional model of the virtual robot may be, for example, a three-dimensional model of a virtual robot corresponding to a real robot to be operated. The three-dimensional model of the virtual robot may be composed of, for example, a plurality of partial objects. Each partial object may correspond to each arm of the three-dimensional model of the manipulator, a tool at the end of the hand, or the like. In addition, information other than the three-dimensional model may be stored in the storage unit 31. For example, teaching data or the like may be stored in the storage unit 31. The process by which information is stored in the storage unit 31 does not matter. For example, information may be stored in the storage unit 31 via a recording medium, or information transmitted via a communication line or the like may be stored in the storage unit 31. The storage unit 31 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0017] The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment in which the real robot exists and the display device 4. For example, the robot 1, which is a real robot, may exist in the real environment. Acquiring the relative positional relationship between the real environment and the display device 4 may be, for example, acquiring the relative positional relationship between a world coordinate system, which is a coordinate system of the real environment, and a display coordinate system, which is a local coordinate system of the display device 4. The relative positional relationship may be indicated, for example, by a homogeneous transformation matrix indicating the transformation between both coordinate systems. Note that the relative positional relationship between the real environment and the display device 4 may be, for example, the relative positional relationship between the robot 1 or other objects existing in the real environment and the display device 4, or may be information indicating the position and attitude of the display device 4 in the real environment.

[0018] The method by which the positional relationship acquisition unit 32 acquires this relative positional relationship is not limited. For example, when an environmental map of the real environment is prepared, the positional relationship acquisition unit 32 may acquire the relative positional relationship, which is information indicating the position and orientation of the display device 4 in a three-dimensional real environment, using the environmental map of the real environment and the distance to the surrounding objects and the surrounding image acquired by the display device 4, using a method such as SLAM (Simultaneous Localization and Mapping) or Visual-SLAM. In this case, for example, the environmental map is held in the image processing device 3, and the positional relationship acquisition unit 32 may acquire the relative positional relationship using the distance to the surrounding objects and the surrounding image received from the display device 4 and the environmental map. Also, for example, the acquisition of the relative positional relationship using the method such as SLAM or Visual-SLAM may be performed by the display device 4, and the relative positional relationship may be passed to the positional relationship acquisition unit 32. In this case, the acquisition of the relative positional relationship by the positional relationship acquisition unit 32 may be acceptance of the relative positional relationship. In this case, the display device 4 may have a depth sensor capable of measuring the distance to surrounding objects and a camera capable of acquiring images of the surroundings. The positional relationship acquisition unit 32 may receive an image captured by the camera of the display device 4, and acquire a homogeneous transformation matrix indicating the transformation between the world coordinate system and the display coordinate system using three or more feature points of the robot 1 or other objects included in the image.

[0019] The positional relationship acquisition unit 32 may acquire the relative positional relationship between the real environment and the display device 4 by a method other than the above. For an example of the method, see, for example, Patent Document 1. The relative positional relationship may be acquired by the positional relationship acquisition unit 32 as described above for each generation of a display image described later, or may not be acquired. In the latter case, for example, the relative positional relationship between the real environment and the display device 4 may be updated using a change in the position or attitude of the display device 4 after the relative positional relationship is acquired. This update may be performed by, for example, the positional relationship acquisition unit 32. When this update is performed, the display device 4 may include, for example, an acceleration sensor or a sensor for acquiring the orientation of the display device 4, and the change in position or attitude may be acquired using these sensors. Note that the values ​​acquired by these sensors may be passed to a component that updates the relative positional relationship, such as the positional relationship acquisition unit 32. The sensor for acquiring the orientation of the display device 4 may be, for example, a gyro sensor or a direction sensor.

[0020] The receiving unit 33 receives a start instruction that indicates the start point of an operation using the operation instructing member 5, and an end instruction that indicates the end point of the operation. The start instruction and the end instruction may be input by voice such as "start operation" and "end operation", or may be input via an input device or a virtual input interface of the display device 4.

[0021] When voice input is performed, the reception unit 33 may receive voice corresponding to the operator's utterance of "start operation" or "end operation" through a microphone or the like, convert the voice into a signal, perform voice recognition on the voice signal, and determine that a start instruction has been received when the text information resulting from the voice recognition is "start operation." The same applies to an end instruction. Note that the reception unit 33 may receive a start instruction or an end instruction corresponding to the result of the voice recognition from another component or another device.

[0022] The input device may have, for example, a physical button for inputting a start instruction and a physical button for inputting an end instruction. For example, when the worker presses the physical button for inputting a start instruction, the reception unit 33 may determine that the start instruction has been received. The same applies to the end instruction.

[0023] Furthermore, the worker may cause a virtual input interface such as a virtual button to be displayed on the display of the display device 4 in response to an action such as an air tap. Then, for example, when a virtual button for inputting a start instruction is selected by the worker's finger or a pointing device, and a signal indicating that the virtual button has been selected is input to the reception unit 33, the reception unit 33 may determine that the start instruction has been received. The same applies to an end instruction.

[0024] The reception unit 33 may or may not include a device for receiving (for example, a microphone, an input device, a communication device, etc.). The reception unit 33 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0025] The operation acquisition unit 34 acquires the position of the operation instruction member 5 present in the real environment. The operation acquisition unit 34 may acquire, for example, the position and the attitude of the operation instruction member 5. The shape of the operation instruction member 5 is not particularly limited, and may be, for example, a rod-shaped member or a member of other shapes. The rod-shaped member may be, for example, a cylindrical member or a polygonal prism-shaped member. Also, as shown in FIG. 1, the operation instruction member 5, which is a rod-shaped member, may be configured such that the shape of both ends in the longitudinal direction is different, for example, one end side is pointed, so that the orientation can be determined. For example, a writing instrument such as a pen or a pencil may be used as the operation instruction member 5. In this embodiment, the case where the operation instruction member 5 is a rod-shaped member will be mainly described, and other cases will be described later. The operation instruction member 5 is usually moved by the hand of the operator. The operation instruction member 5 is, for example, about the same size as a writing instrument such as a pen, and may be held and moved by the operator in the hand in the same way as a writing instrument. The position and posture of the operation instructing member 5 may indicate, for example, the position and posture of a tool attached to the hand of the 3D model of the virtual robot, the position and posture of a TCP (tool center point) of the 3D model of the virtual robot, or the position and posture of a predetermined part (e.g., a predetermined arm) in the 3D model of the virtual robot. The part of the 3D model corresponding to the operation instructing member 5 may be determined in advance, or may be determined by the operator. In this embodiment, a case where the position and posture of a tool at the hand of the 3D model are indicated by the position and posture of the operation instructing member 5 will be mainly described.

[0026] The method by which the operation acquisition unit 34 acquires the position and attitude of the operation instruction member 5 is not limited. Acquiring the position and attitude of the operation instruction member 5 may, for example, specify the position and attitude of the operation instruction member 5 in a predetermined coordinate system. By repeating the acquisition of the position and attitude in real time, the movement of the operation instruction member 5 is acquired. The coordinate system may, for example, be a world coordinate system, a local coordinate system of the display device 4, or another coordinate system. When the operation acquisition unit 34 acquires the position of the operation instruction member 5, for example, the position of a predetermined part of the operation instruction member 5 (for example, the tip of the lower side of the operation instruction member 5 shown in FIG. 1) may be acquired. This position may be indicated by, for example, coordinate values ​​in a three-dimensional orthogonal coordinate system. When the operation acquisition unit 34 acquires the attitude of the operation instruction member 5, for example, the angle of the longitudinal direction of the rod-shaped operation instruction member 5 and the angle of rotation about the longitudinal axis of the rod-shaped operation instruction member 5 may be acquired. This attitude may also be indicated by, for example, a roll, pitch, yaw angle, Euler angle, or the like. In addition, when the rotation angle about the longitudinal axis of the rod-shaped operation instructing member 5 is also acquired, or when angles with three degrees of freedom according to the posture are acquired, a predetermined pattern, protrusion, recess, or the like may be provided on the surface of the operation instructing member 5 (for example, the circumferential surface of the cylindrical operation instructing member 5) so that, for example, the rotation angle about the longitudinal axis of the operation instructing member 5 can be known. In this embodiment, the case where the operation acquiring unit 34 acquires both the position and posture of the operation instructing member 5 will be mainly described.

[0027] The operation acquisition unit 34 may acquire the position, etc., of the operation instruction member 5, for example, by using a captured image of the operation instruction member 5. In this case, the image may be captured by, for example, a camera included in the display device 4, or may be captured by using a camera disposed at a predetermined position in the real environment. When a camera for capturing an image of the real environment is attached to the head-mounted display device 4, the camera is preferably disposed so as to capture an image in the same direction as the line of sight of the worker wearing the display device 4. When a camera is disposed in the real environment, the camera is preferably disposed so as to capture an image of the area in which the operation instruction member 5 is moved by the worker. The operation acquisition unit 34 may, for example, identify and track the operation instruction member 5 by using the captured image, and acquire the position, etc., of the operation instruction member 5 by using the result.

[0028] The operation acquisition unit 34 may acquire the position and the like by using, for example, a sensing result by a sensor incorporated in the operation instruction member 5. The sensor may be, for example, an acceleration sensor or a sensor for acquiring the attitude of the operation instruction member 5. The sensor for acquiring the attitude may be, for example, an orientation sensor for acquiring an azimuth angle and an inclination sensor for acquiring an inclination angle with respect to the vertical direction, or a gyro sensor (i.e., an angular velocity sensor). Since the acceleration sensor and the gyro sensor can only detect changes in position and angle, for example, the operator may first place the operation instruction member 5 at a predetermined reference position and reference attitude. Then, the operation acquisition unit 34 may acquire the current position and attitude of the operation instruction member 5 by using the sensing result acquired by the acceleration sensor or the gyro sensor and the reference position and reference attitude thereafter. The sensing result of the sensor included in the operation instruction member 5 may be transmitted from the operation instruction member 5 to the operation acquisition unit 34 wirelessly or by wire.

[0029] Furthermore, the operation acquisition unit 34 may acquire the position of the operation instructing member 5, for example, by using a sensing result indicating the position of the operation instructing member 5 acquired by a sensor arranged in the real environment or a sensor included in the display device 4. This sensor may be, for example, a distance measurement sensor such as a depth sensor. When the sensor is a distance measurement sensor, a three-dimensional scan of the real environment may be performed using multiple distance measurement sensors. Then, the operation acquisition unit 34 may specify the position and orientation of the operation instructing member 5, which has a predetermined shape, by using the sensing result, i.e., the result of the three-dimensional scan.

[0030] Furthermore, acquisition of the position and the like of the operation instructing member 5 using the captured image or the sensing result may be performed by a component or device other than the operation acquisition unit 34. In this case, the operation acquisition unit 34 may accept the position and the orientation of the operation instructing member 5 acquired by a component or device other than the operation acquisition unit 34. In other words, acquisition of the position and the orientation of the operation instructing member 5 by the operation acquisition unit 34 may be acceptance of the position and the orientation of the operation instructing member 5.

[0031] The operation acquisition unit 34 acquires an operation for a part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot according to the acquired position and posture of the operation instructing member 5. The part corresponding to the operation instructing member 5 may be a part in the three-dimensional model whose position and posture are indicated by the operation instructing member 5. For example, an operation for making the part corresponding to the operation instructing member 5 in the three-dimensional model overlap with the operation instructing member 5 may be acquired. The acquisition of this operation may be, for example, acquisition of the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot after the operation. For example, in the case where the operation instructing member 5 is a rod-shaped member and indicates the position and posture of a tool attached to the hand of the three-dimensional model of the virtual robot, the operation acquisition unit 34 may acquire an operation for the tool of the three-dimensional model of the virtual robot according to the position of the operation instructing member 5, etc. That is, the operation acquisition unit 34 may acquire an operation in which the tool of the three-dimensional model becomes the position of the operation instructing member 5. The tool is disposed at the hand of the three-dimensional model of the virtual robot, and may be, for example, a welding torch, a gripping unit for gripping a transport target, a suction unit for suctioning a transport target, a placement unit for placing a transport target, or the like. For example, when the tool is a welding torch, the rod-shaped operation instruction member 5 may indicate the position of the tip of the welding torch and the attitude of the tip. In this case, the operation acquisition unit 34 acquires an operation to change the position and attitude of the welding torch in the same manner as the position and attitude of the operation instruction member 5, according to the position and attitude of the operation instruction member 5. Also, for example, when the tool is a gripping unit or a suction unit, the rod-shaped operation instruction member 5 may indicate the reference position and the attitude of the reference direction of the unit. In this case, the operation acquisition unit 34 acquires an operation to change the reference position and the attitude of the gripping unit or the suction unit in the same manner as the position and attitude of the operation instruction member 5, according to the position and attitude of the operation instruction member 5.

[0032] In addition, when the position and posture of the operation instructing member 5 directly become the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot, the operation acquisition unit 34 may treat the position and posture of the operation instructing member 5 as an operation for the part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot. In this embodiment, this case will be mainly described. Even in this case, the operation acquisition unit 34 may convert the position and posture of the operation instructing member 5 in the display coordinate system into the position and posture in the local coordinate system of the virtual robot using a homogenous transformation matrix between the world coordinate system and the display coordinate system and a homogenous transformation matrix between the world coordinate system and the local coordinate system of the three-dimensional model of the virtual robot, which are acquired by the positional relationship acquisition unit 32, and use the result as an operation for the three-dimensional model of the virtual robot. On the other hand, when the position and posture of the operation instructing member 5 are different from the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot, the operation acquisition unit 34 may generate the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model of the virtual robot based on the position and posture of the operation instructing member 5, and the generated result may be the operation for that part. In this case, the relationship between the position and posture of the operation instructing member 5 and the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model is usually determined in advance, and the operation acquisition unit 34 may acquire the position and posture of the part corresponding to the operation instructing member 5 in the three-dimensional model from the position and posture of the operation instructing member 5 using the relationship, thereby acquiring the operation of that part. This case will be described later.

[0033] The operation according to the position and the like of the operation instructing member 5 is acquired from the time when the start instruction is accepted by the accepting unit 33 to the time when the end instruction is accepted. That is, the operation acquiring unit 34 acquires the operation according to the position and the like of the operation instructing member 5 from the start instruction to the end instruction. The acquired operation may indicate, for example, the position of the tip of the welding torch and the attitude of the tip in real time.

[0034] The image generation unit 35 generates a display image for displaying the three-dimensional model at a predetermined position in the real environment based on the three-dimensional model of the virtual robot stored in the storage unit 31 and the relative positional relationship acquired by the positional relationship acquisition unit 32. Furthermore, when an operation on the virtual robot is acquired, the image generation unit 35 generates a display image for displaying the operated three-dimensional model.

[0035] The predetermined position in the real environment where the three-dimensional model of the virtual robot is displayed may be, for example, a predetermined position in the world coordinate system, or may be a position where the robot 1 or other object existing in the real environment and the three-dimensional model of the virtual robot have a predetermined positional relationship. The position in the real environment where the three-dimensional model of the virtual robot is displayed, or the predetermined positional relationship between the robot 1 or the like and the three-dimensional model of the virtual robot, may be, for example, determined in advance, or may be changeable by the operator. The three-dimensional model of the virtual robot may be displayed, for example, at the position of the robot 1, or at a position different from the robot 1. Displaying the three-dimensional model of the virtual robot at the position of the robot 1 means displaying the three-dimensional model so that the virtual robot is placed at the same position as the robot 1, and may be, for example, displaying the end portion of the base end side of the three-dimensional model of the virtual robot (for example, the part attached to the floor surface, etc.) so as to overlap the end portion of the base end side of the robot 1. When the three-dimensional model of the virtual robot is displayed at a position different from the robot 1, for example, the three-dimensional model may be displayed next to the robot 1. In the image processing device 3 according to the present embodiment, the location of the three-dimensional model of the virtual robot corresponding to the operation instruction member 5 moved by the worker in the real environment is displayed at the position of the operation instruction member 5. Therefore, when the three-dimensional model of the virtual robot is displayed at the position of the robot 1, since the operation instruction member 5 may hit the arm or the like of the robot 1, it is preferable that the three-dimensional model of the virtual robot is displayed at a position different from that of the robot 1. In any case, the three-dimensional model of the virtual robot is displayed so that its position in the real environment does not change. For example, the three-dimensional model of the virtual robot is displayed so that its positional relationship with the robot 1, which is the real robot, does not change. Therefore, even if the worker changes the orientation of the display device 4, the display position of the three-dimensional model of the virtual robot in the real environment does not change.

[0036] The position of the three-dimensional model of the virtual robot in the world coordinate system is fixed. Therefore, the image generating unit 35 can place the three-dimensional model of the virtual robot in the virtual space so as to have the same positional relationship as in the real environment. For example, when the robot 1 is placed in the real environment, the image generating unit 35 may place the three-dimensional model of the virtual robot so as to have a predetermined positional relationship with the placement position of the robot 1 in the virtual space where the placement position of the robot 1 is predetermined. The angles of each joint of the virtual robot are set to initial values ​​when no operation is performed, and are set to values ​​after the operation when an operation is performed. The angles of each joint after the operation may be calculated by inverse kinematics using the position and posture of the hand in the three-dimensional model of the virtual robot after the operation, for example, as with the real robot. The position and posture of the hand in the three-dimensional model may be indicated by the acquisition result of the operation acquisition unit 34. Furthermore, the image generating unit 35 can know the relationship between the world coordinate system and the display coordinate system, which is the local coordinate system of the display device 4, by the relative positional relationship acquired by the positional relationship acquisition unit 32, and therefore can specify the position and orientation of the display device 4 in the virtual space. Therefore, the image generating unit 35 can generate a two-dimensional display image for displaying the three-dimensional model by rendering the three-dimensional model of the virtual robot in the virtual space based on the position and orientation of the display device 4. When the virtual robot is operated, the shape of the three-dimensional model of the virtual robot in the virtual space is changed accordingly, as described above. Also, when the position or orientation of the display device 4 changes in the real environment, the position and direction of the viewpoint in the virtual space are changed accordingly. Then, rendering is performed after the change, so that a display image of the three-dimensional model after the operation and a display image of the three-dimensional model after the change in the position or orientation of the display device 4 are generated. Note that the image generating unit 35 generates a display image such that the size of the display image of the three-dimensional model of the virtual robot matches the real environment when the display image is displayed on the display of the display device 4.In other words, a display image is generated so that the three-dimensional model of the virtual robot displayed on the display of display device 4 and the real robot placed in the real environment so as to have the same relative positional relationship as the three-dimensional model appear to be the same size when viewed through display device 4.

[0037] In addition, it may not be possible to operate the virtual robot according to the accepted operation. For example, an operation to move the hand of the virtual robot beyond the movable range or rotate it beyond the rotatable range may be accepted. In such a case, the image generating unit 35 may not generate a display image according to the operation, or may generate a display image of a three-dimensional model of the virtual robot that has been moved or rotated within the possible range. In addition, when an operation to move the hand beyond the movable range or rotate it beyond the rotatable range is accepted, for example, an alert may be output to the operator to the effect that the accepted operation exceeds the operable range.

[0038] The image output unit 36 ​​outputs the display image generated by the image generation unit 35 to the display device 4. The image output unit 36 ​​may output only the display image. In this case, the display image is displayed on the display device 4 by being superimposed on the image of the real environment or the real environment itself. On the other hand, when the display device 4 has a non-transparent display and the image of the real environment captured by the display device 4 is accepted by the image processing device 3, the result of compositing the image of the real environment and the display image may be output to the display device 4. This composition may be performed, for example, by a composition unit (not shown) included in the image processing device 3. In addition, the generated display image may be displayed on the display device 4 so that, for example, the operation instruction member 5 and a portion corresponding to the operation instruction member 5 in the three-dimensional model of the virtual robot are displayed so as to be superimposed on each other.

[0039] When the three-dimensional model of the virtual robot is displayed at a position different from that of the robot 1, which is the actual robot, for example, as shown in Fig. 1, the three-dimensional model 10 of the virtual robot may be displayed next to the robot 1. The robot 1 and the three-dimensional model 10 of the virtual robot in Fig. 1 are schematic diagrams showing the situation as viewed by an operator operating the virtual robot via a display device 4. Note that, for convenience of explanation, in Fig. 1, the three-dimensional model 10 of the virtual robot and the operation instruction member 5 are in different positions, but, for example, at least a part of the three-dimensional model 10 may be displayed so as to overlap with the operation instruction member 5.

[0040] Note that displaying a three-dimensional model of a virtual robot and displaying the three-dimensional model after manipulation of the position or posture of a specific portion of the three-dimensional model of the virtual robot in accordance with the manipulation are already publicly known, for example as shown in Patent Document 1 mentioned above, and detailed description thereof will be omitted.

[0041] Next, the operation of the image processing device 3 will be described with reference to the flowchart of FIG.

[0042] (Step S101) The reception unit 33 determines whether or not an instruction to start an operation has been received. If an instruction to start an operation has been received, the process proceeds to step S102, and if not, the process proceeds to step S109.

[0043] (Step S102) The operation acquisition unit 34 judges whether or not to acquire the position and / or posture of the operation instructing member 5. If the position, etc. of the operation instructing member 5 is to be acquired, the process proceeds to step S103. If not, the process repeats the process of step S102 until it is determined that the position, etc. of the operation instructing member 5 is to be acquired. Note that the operation acquisition unit 34 may, for example, periodically judge that the position, etc. of the operation instructing member 5 is to be acquired. Also, the operation acquisition unit 34 may, for example, immediately judge that the position, etc. of the operation instructing member 5 is to be acquired when proceeding from step S101 to step S102.

[0044] (Step S103) The operation acquisition unit 34 acquires the position and posture of the operation instructing member 5. The process of this step S103 may be periodically repeated, so that the movement of the operation instructing member 5 is acquired as a result.

[0045] (Step S104) The operation acquisition unit 34 acquires an operation for a location in the three-dimensional model of the virtual robot corresponding to the operation instructing member 5, based on the acquisition result regarding the position of the operation instructing member 5 in step S103, etc. The acquisition of this operation may be, for example, acquisition of the position and orientation of a predetermined location in the three-dimensional model. The position and orientation acquired in this manner are usually the position and orientation after the operation. Note that, for example, if the position and orientation of the operation instructing member 5 directly become an operation for a location in the three-dimensional model corresponding to the operation instructing member 5, the operation acquisition unit 34 may use the position and orientation of the operation instructing member 5 acquired in step S103 as an operation for the location in the three-dimensional model corresponding to the operation instructing member 5.

[0046] (Step S105) The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment and the display device 4.

[0047] (Step S106) The image generation unit 35 generates a display image of the operated three-dimensional model by using the three-dimensional model of the virtual robot, the relative positional relationship acquired in step S105, and the operation acquired in step S104.

[0048] (Step S107) The image output unit 36 ​​outputs the display image generated in step S106 to the display device 4. As a result, the worker can see the 3D model of the virtual robot after operation. Usually, a part of the 3D model corresponding to the operation instructing member 5 is displayed superimposed on the position of the operation instructing member 5 that the worker is operating in the real environment. In this way, it becomes possible to operate the 3D model of the virtual robot using the operation instructing member 5 that exists in the real environment.

[0049] (Step S108) The accepting unit 33 determines whether or not an instruction to end the operation has been accepted. If an instruction to end the operation has been accepted, the process returns to step S101, and if not, the process returns to step S102.

[0050] (Step S109) The image generating unit 35 judges whether or not to generate a display image. If a display image is to be generated, the process proceeds to step S110; if not, the process returns to step S101. The image generating unit 35 may, for example, periodically judge to generate a display image. By making this judgment, for example, even if no operation is performed, when the position or orientation of the display device 4 is changed, a display image corresponding to the changed position or orientation is displayed on the display device 4.

[0051] (Step S110) The positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment and the display device 4.

[0052] (Step S111) The image generating unit 35 generates a display image of the three-dimensional model by using the three-dimensional model of the virtual robot and the relative positional relationship acquired in step S110.

[0053] (Step S112) The image output unit 36 ​​outputs the display image generated in step S111 to the display device 4. As a result, the worker can see the three-dimensional model of the virtual robot. Then, the process returns to step S101.

[0054] Note that the order of the processes in the flowchart of Fig. 2 is an example, and the order of the steps may be changed as long as the same results can be obtained. In addition, in the flowchart of Fig. 2, the process ends when the power is turned off or an interrupt is issued to end the process.

[0055] Next, the operation of the image processing device 3 according to this embodiment will be described using a specific example.

[0056] Assume that an operator wearing the display device 4 on his / her head displays a three-dimensional model of a virtual robot on the display (steps S109 to S112). Then, as shown in FIG. 3A, the operator moves the rod-shaped operation instructing member 5 by hand to a position in the real environment to which the operator wants to move the three-dimensional model tool 10a, and utters "operation start" in that state. Then, the voice is collected by the microphone of the reception unit 33, and voice recognition is performed, and it is determined that an instruction to start the operation has been input (step S101). Then, the operation acquisition unit 34 acquires the position and orientation of the tip side of the operation instructing member 5 using the photographed image of the operation instructing member 5 acquired by the display device 4, and this directly becomes an operation indicating the position and orientation of the tip of the tool 10a (steps S102 to S104). The operation is passed to the image generation unit 35.

[0057] Thereafter, the positional relationship acquisition unit 32 acquires the relative positional relationship between the real environment and the display device 4, and transfers it to the image generation unit 35 (step S105). When the image generation unit 35 receives the relative positional relationship and the operation of the tool 10a, it updates the position and orientation of the display device 4 in the virtual space according to the relative positional relationship, calculates the angle of each joint of the three-dimensional model of the virtual robot by inverse kinematics based on the operation to the tool 10a, i.e., the position and orientation of the tool 10a, and changes the angle of each joint of the three-dimensional model according to the calculated joint angle. Thereafter, the image generation unit 35 generates a display image by rendering the three-dimensional model, and transfers it to the image output unit 36 ​​(step S106). The display image is output to the display device 4 by the image output unit 36 ​​(step S107). As a result, the operator can see the three-dimensional model displayed so that the tool 10a overlaps the operation instruction member 5, as shown in FIG. 3B.

[0058] Thereafter, it is assumed that the operator moves the operation instructing member 5 from the position indicated by the dashed line in FIG. 3C to the position indicated by the solid line. Then, the tool 10a of the three-dimensional model of the virtual robot also moves accordingly (steps S102 to S107). In this way, the tool 10a of the three-dimensional model of the virtual robot follows the operation instructing member 5, so that the operator can easily operate the three-dimensional model of the virtual robot. In particular, the operator can easily change a minute angle by using the operation instructing member 5. In addition, the operator can appropriately operate the three-dimensional model of the virtual robot by placing the base material to be welded in the real environment and moving the operation instructing member 5 so that the tip of the operation instructing member 5 is aligned with the weld line on the base material.

[0059] In addition, when teaching is performed according to this operation, the worker may input a teaching instruction to the image processing device 3 as appropriate using a virtual input interface or other input device. Then, for example, the angle of each joint at that time may be recorded as teaching data in response to the instruction. The teaching data generated in this way may be passed to a robot control device that controls the robot 1, which is a real robot, and may be used for playback operation of the robot 1. In this way, teaching data can also be generated using a three-dimensional model of a virtual robot. It is assumed that the worker utters "operation end" after a series of operations on the three-dimensional model of the virtual robot is completed. Then, the voice is collected by the microphone of the reception unit 33, voice recognition is performed, and it is determined that an instruction to end the operation has been input (step S108). Then, the series of operations is completed, and the tool 10a of the three-dimensional model does not follow the operation instruction member 5.

[0060] 3A to 3C, for convenience of explanation, the hand of the operator holding the operation instructing member 5 is omitted. For example, in the case where a pattern, protrusion, recess, or the like for acquiring the attitude of the operation instructing member 5 is provided on the circumferential surface of the operation instructing member 5, and the position and attitude of the operation instructing member 5 are acquired using a captured image of the operation instructing member 5, it is preferable that the operator holds the operation instructing member 5 so that the pattern, or the like, on the circumferential surface of the operation instructing member 5 is included in the captured image.

[0061] In addition, in the present embodiment, the case where the position of the tip of the operation instructing member 5 is the position of the tip of the tool 10a has been mainly described, but this is not necessary. For example, as shown in FIG. 4, an operation may be acquired in which the position of the tip of the tool 10a is the position of the tip of the operation instructing member 5 that is closer to the other end side by a distance L from the tip side of the operation instructing member 5. In this case, for example, when operating the tool 10a in consideration of the extension length of the welding wire, the tip of the operation instructing member 5 may be moved along the welding line of the base material, and the distance L may be the extension length. Then, the operation acquiring unit 34 may acquire an operation in which the position of the tip of the tool 10a is the position of the tip of the operation instructing member 5 that is closer to the other end side by a distance L from the tip of the operation instructing member 5. In this way, an operation that takes into consideration the extension length regarding the three-dimensional model of the virtual robot can be realized. In this case, the position of the acquired operation instructing member 5 and the position of the tool 10a indicated by the acquired operation are different. In this way, the position indicated by the operation on the part corresponding to the operation instructing member 5 in the three-dimensional model may be a position that is a predetermined length away from the acquired position of the operation instructing member 5. In addition, the position corresponding to the tip of the tool 10a in the rod-shaped operation instructing member 5 may be indicated by a color, a ring, or the like. For example, a position close to the other end side from the tip of the operation instructing member 5 by a distance L may be painted in a different color from the others in the circumferential direction. Then, an operation may be acquired in which the part of that color corresponds to a predetermined position in the three-dimensional model, for example, the position of the tip of the tool 10a. Also, a ring-shaped member movable in the longitudinal direction may be attached to the rod-shaped operation instructing member 5. Then, an operation may be acquired in which the part of the ring corresponds to a predetermined position in the three-dimensional model, for example, the position of the tip of the tool 10a. In this case, for example, the position of the tip of the tool 10a can be arbitrarily changed by the operator moving the position of the ring.

[0062] In addition, in the present embodiment, the case where the tool 10a is operated by the operation instructing member 5 has been mainly described, but this is not necessarily the case. The operation instructing member 5 may be used to operate a partial object such as an arm constituting the 3D model 10 of the virtual robot. That is, the part corresponding to the operation instructing member 5 in the 3D model 10 of the virtual robot may be a predetermined arm, not a tool, a hand, or a TCP. More specifically, as shown in FIG. 5, among the partial objects 10-1 to 10-8 included in the 3D model 10 of the virtual robot, the partial object 10-7 may be operated by the operation instructing member 5. That is, the position and the orientation of a certain partial object 10-7 may be determined according to the position and the orientation of the operation instructing member 5 in the real environment. In this case, the image generating unit 35 may generate, for example, a display image in which the partial object 10-7 operated by the operation instructing member 5 can be specified. More specifically, a display image in which the color or brightness of the partial object 10-7 is different from the colors or brightness of the other partial objects may be generated. The partial object to be operated may be selected by the operator as appropriate. In this case, for example, when a worker selects a specific partial object in the three-dimensional model of the virtual robot displayed on the display of the display device 4 by a gesture operation such as an air tap, the selected partial object may become the target of operation using the operation instructing member 5.

[0063] As described above, according to the image processing device 3 of this embodiment, a predetermined part of a three-dimensional model of a virtual robot in a virtual space can be operated by operating the operation instruction member 5 existing in a real environment. Therefore, a more realistic and intuitive operation can be realized than a case where a predetermined part of a virtual robot is virtually pinched by hand (holding operation) in a virtual space and the pinched part is moved. In particular, when an operation of moving along a base material existing in a real environment or an operation of changing an angle only slightly is performed, a more accurate operation can be realized. In addition, by acquiring an operation by the operation acquisition unit 34 during the period from the start instruction to the end instruction of the operation accepted by the acceptance unit 33, it is possible to distinguish a period in which the three-dimensional model follows the operation instruction member 5 from a period in which it does not, and to realize an appropriate operation.

[0064] In the present embodiment, the case where the instruction to start and the instruction to end the operation are received by the receiving unit 33 has been described, but this is not essential. For example, it may be determined that the start of the operation or the end of the operation is instructed in response to the operation of the operation instructing member 5 by the worker. In this case, for example, the operation acquiring unit 34 may determine that the start of the operation is instructed when the rod-shaped operation instructing member 5 included in the captured image is swung for the first time, and may determine that the end of the operation is instructed when the rod-shaped operation instructing member 5 included in the captured image is swung for the second time. Also, for example, the operation acquiring unit 34 may determine that the start of the operation is instructed when the operation instructing member 5 included in the captured image is superimposed on a portion of the three-dimensional model corresponding to the operation instructing member 5, and may determine that the end of the operation is instructed when the operation instructing member 5 is swung. Also, in the case where the operation instructing member 5 has a light source, for example, the operation acquiring unit 34 may determine that the start of the operation is instructed when the light source of the operation instructing member 5 included in the captured image is turned on, and may determine that the end of the operation is instructed when the light source is turned off. In this way, when the reception unit 33 does not receive an instruction to start or end an operation, the image processing device 3 does not need to have the reception unit 33.

[0065] In addition, in the present embodiment, the case where the rod-shaped operation instruction member 5 is used has been mainly described, but this is not necessary. For example, the operation instruction member may be a planar member. And, for example, the tip of the tool of the three-dimensional model may be located at a predetermined position (for example, the center or center of gravity, or a position where a predetermined figure, a convex part, a concave part, etc. exists) on the planar member, and an operation on the tool may be acquired so that the attitude of the tip side of the tool is in the normal direction of the planar member. In this case, when the display image of the three-dimensional model is displayed on the display device 4, the part corresponding to the operation instruction member in the three-dimensional model (for example, the tip part of the tool) and the planar operation instruction member may be displayed so as not to overlap. For example, the tool may be displayed so that the tip of the tool contacts a predetermined position on the surface of the planar operation instruction member.

[0066] In the present embodiment, the case where the position and posture of the operation instruction member 5 are acquired by the operation acquisition unit 34, and the operation of the part of the three-dimensional model of the virtual robot corresponding to the operation instruction member 5 is acquired according to the acquisition result has been mainly described, but this is not necessary. The operation acquisition unit 34 may acquire only the position of the operation instruction member 5, and acquire an operation indicating only the position of the part of the three-dimensional model of the virtual robot corresponding to the operation instruction member 5 according to the acquisition result. Even in this case, for example, the tip of the tool, the TCP, or the position of another part of the three-dimensional model can be operated using the operation instruction member 5. In this case, for example, a predetermined posture may be used for the posture of the three-dimensional model of the virtual robot, or a posture input by another method may be used. For example, when the worker selects a predetermined part of the three-dimensional model by air tapping or the like, the operation acquisition unit 34 may acquire an operation in which the selected position becomes the position of the tip of the operation instruction member 5.

[0067] In the above embodiment, each process or function may be realized by centralized processing by a single device or a single system, or may be realized by distributed processing by multiple devices or multiple systems. For example, at least a part of the configuration of the image processing device 3 may be physically included in a device having a display. Therefore, the division of the devices shown in FIG. 1 may be considered to be for convenience according to function, not according to physical device.

[0068] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0069] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0070] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0071] In addition, in the above embodiment, when two or more components included in the image processing device 3 have a communication device, an input device, etc., the two or more components may physically have a single device, or may have separate devices.

[0072] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. During execution, the program execution unit may execute the program while accessing a storage unit or a recording medium. The program may be executed by being downloaded from a server or the like, or may be executed by reading a program recorded on a predetermined recording medium (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.). The program may be used as a program constituting a program product. The computer that executes the program may be single or multiple. That is, centralized processing may be performed, or distributed processing may be performed.

[0073] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Explanation of symbols]

[0074] 3 image processing device, 4 display device, 5 operation instruction member, 31 storage unit, 32 positional relationship acquisition unit, 33 reception unit, 34 operation acquisition unit, 35 image generation unit, 36 image output unit

Claims

1. A storage unit for storing a three-dimensional model of a virtual robot corresponding to a real robot existing in a real environment; an operation acquisition unit that acquires a position of an operation instructing member that exists in a real environment, and acquires an operation for causing a portion of a three-dimensional model of the virtual robot that corresponds to the operation instructing member to overlap with the operation instructing member in accordance with the acquired position; a positional relationship acquisition unit that acquires a relative positional relationship between a real environment in which the real robot exists and a display device that displays an image of the real environment or the real environment itself by superimposing the image; an image generating unit that generates a display image for displaying the operated three-dimensional model at a predetermined position in a real environment based on the three-dimensional model and the relative positional relationship; and an image output unit that outputs the display image to a display device.

2. the operation instruction member is a rod-shaped member, The image processing apparatus according to claim 1 , wherein the operation acquisition unit acquires an operation for a tool of the three-dimensional model in accordance with a position of the operation instruction member.

3. a reception unit that receives a start instruction that instructs a start point of an operation using the operation instructing member and an end instruction that instructs a end point of the operation, The image processing apparatus according to claim 1 , wherein the operation acquisition unit acquires an operation corresponding to a position of the operation instructing member from the start instruction to the end instruction.

Images