Image processing device

Abstract

To solve the problem in which: a user cannot easily grasp an area in which a robot can be safely arranged.SOLUTION: An image processing apparatus 1 comprises: a positional relation acquisition unit 12 which acquires a relative positional relation between an actual environment and a display device 2 that displays an image so as to be superimposed on the actual environment itself; a reception unit 13 which receives a position with the display device 2 as a reference; an acquisition unit 14 which acquires an entry inhibition area on the actual environment by using the received position indicating the entry inhibition area and the relative positional relation; a specification unit 15 which specifies a safe arrangement possible area of a robot for preventing entry to the entry inhibition area by using the acquired entry inhibition area and a movable area of the robot; an image generation unit 16 which generates a display image indicating the safe arrangement possible area by using the safe arrangement possible area and the relative positional relation; and an output unit 17 which outputs the display image to the display device 2. Consequently, a user can easily grasp the safe arrangement possible area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device for displaying an image superimposed on a real environment. [Background technology]

[0002] Conventionally, there is known a display device that can display a virtual robot together with objects in the real space before the robot is installed at the work site (see, for example, Patent Document 1). By using such a display device, it is possible to check the situation of the work site where the virtual robot is installed, and for example, to check in advance whether the virtual robot will interfere with objects or workers placed at the work site. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-088028 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional display devices, while it is possible to check the work of a virtual robot at a work site, it is necessary to visually confirm whether the virtual robot will interfere with objects or workers located at the work site. Therefore, for example, even if no interference occurs when the virtual robot is performing a specific task, it is not possible to determine whether interference will occur when the virtual robot performs a different task until the virtual robot performs that task. Furthermore, if it is determined that interference will occur and the position of the virtual robot is changed, it is necessary to visually confirm again whether interference will occur, which results in the problem of having to repeatedly set up the virtual robot and check for interference.

[0005] The present invention has been made to solve the above problem, and aims to provide an image processing device that can easily grasp an area where a robot can be safely placed. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, an image processing device according to one aspect of the present invention includes a positional relationship acquisition unit that acquires the relative positional relationship between the real environment and a display device that displays an image superimposed on an image of the real environment or the real environment itself; a reception unit that accepts a position based on the display device; an acquisition unit that acquires a no-entry area in the real environment using the position indicating the no-entry area accepted by the reception unit and the relative positional relationship; an identification unit that identifies a safe placeable area, which is a placement area for the robot to be placed to prevent it from entering the no-entry area, using the acquired no-entry area and the movable area of ​​the robot to be placed; an image generation unit that generates a display image indicating the safe placeable area using the identified safe placeable area and the relative positional relationship; and an output unit that outputs the display image to the display device. [Effects of the Invention]

[0007] According to an image processing device of one aspect of the present invention, it is possible to easily grasp a safe placement area, which is an area where a robot can be placed to prevent the robot from entering a prohibited area. Therefore, for example, by determining the placement position of the robot within the safe placement area, it becomes possible to place the robot in a position that will not cause it to enter the prohibited area. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of an image processing system according to an embodiment of the present invention; [Figure 2] A flowchart showing the operation of the image processing device according to the embodiment. [Figure 3] FIG. 10 shows an example of a marker arranged in the embodiment. [Figure 4A]FIG. 10 is a diagram illustrating the acceptance of a no-entry area in the embodiment. [Figure 4B] FIG. 10 is a diagram illustrating the acceptance of a no-entry area in the embodiment. [Figure 5] FIG. 10 is a diagram for explaining the specification of a safe arrangement area in the embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a display of a safe arrangement area in the embodiment; [Figure 7] FIG. 10 is a diagram showing another example of a no-entry area in the embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0009] An image processing device according to the present invention will be described below using an embodiment. In the following embodiments, components and steps denoted with the same reference numerals are the same or equivalent, and repeated description may be omitted. The image processing device according to this embodiment identifies a safe placement area, which is a robot placement area where the robot will not enter an accepted no-entry area, and outputs an image showing that area to a display device.

[0010] 1 is a schematic diagram showing the configuration of an image processing system 100 according to this embodiment. The image processing system 100 according to this embodiment is used to determine the placement position of a robot, and includes an image processing device 1 and a display device 2. Note that the image processing device 1 and the display device 2 may be connected, for example, by wire or wirelessly.

[0011] In this embodiment, the robot to be placed is called a real robot, and a robot configured using a three-dimensional model corresponding to the real robot is sometimes called a virtual robot. A real robot is typically an industrial robot, and may be a manipulator having multiple arms (links) connected by joints driven by motors. A real robot may be, for example, a vertical articulated robot or a horizontal articulated robot. Furthermore, the use of a real robot may be, for example, transportation, welding, assembly, painting, or other uses. Note that a real robot is a robot that exists in a real environment. A real environment refers to an environment in real space.

[0012] A virtual robot is configured as a 3D model existing in a virtual environment and corresponds to a real robot. That is, a virtual robot is the same as a real robot except that it is configured as a 3D model. For example, the virtual robot may have the same size and configuration as a real robot, and may be able to change the angles of the joints of its multiple arms, just like a real robot.

[0013] The image processing device 1 receives a no-entry area for a robot from a user, identifies a safe placement area that is a placement area for a robot to prevent the robot to be placed from entering the no-entry area, generates an image showing the area, and outputs it to the display device 2. Details of the image processing device 1 will be described later.

[0014] The display device 2 displays an image superimposed on an image of the real environment or the real environment itself. That is, the user can see both the real environment and the virtual environment image through the display device 2. The display device 2 may be a wearable display device worn by the user on the head, or may be a display device that is a portable information processing terminal such as a tablet terminal. The wearable display device may be, for example, a head-mounted display. The display device 2 may also have, for example, a transmissive display. In this case, the display device 2 displays an image superimposed on the real environment itself. Known examples of wearable display devices 2 having a transmissive display include HoloLens (registered trademark). Such a display device 2 having a transmissive display can also be considered a display device for realizing mixed reality (MR). The display device 2 may also have, for example, a non-transmissive display. In this case, the display device 2 displays an image superimposed on an image of the real environment. Therefore, it is preferable that the display device 2 having a non-transmissive display has a camera for capturing images of the real environment or is connected to a camera for capturing images of the real environment. An image of the real environment captured by the camera is displayed on a non-transmissive display in real time. For example, Oculus Quest is known as a wearable display device 2 having a non-transmissive display. Such a display device 2 having a non-transmissive display can also be considered a display device for realizing augmented reality (AR). The display device 2, which is a portable information processing terminal such as a tablet terminal, may have, for example, a camera and a display, and may display an image of the real environment captured by the camera on the display in real time. This embodiment will mainly describe a case where the display device 2 is a head-mounted display having a transmissive display. The display device 2 may have sensors such as a depth sensor capable of measuring the distance to surrounding objects and a camera capable of acquiring images of the surroundings, as necessary.

[0015] In this embodiment, as shown in FIG. 3, a case will be mainly described in which the placement position of a robot to be placed is determined using a marker 4 placed in real space. The marker 4 is a predetermined two-dimensional image. The marker 4 may be, for example, an AR marker, a QR code (registered trademark), or any other two-dimensional image with a predetermined shape. The size of the marker 4 may be, for example, predetermined. As will be described later, when the orientation of the robot to be placed is determined depending on the orientation of the marker 4, the marker 4 is one whose orientation can be specified, i.e., one that can be rotated. name It is preferable that the image is not a marker 4. In this embodiment, the case where the marker 4 is displayed on a sheet will be mainly described. The marker 4 may be printed on, for example, a paper or resin sheet. Furthermore, the sheet may display, for example, one or more figures used when placing the actual robot. This figure is used for positioning the actual robot, and may be, for example, a figure indicating the position of a screw hole for fixing the actual robot, a figure indicating the position of the end of the base end of the actual robot, or any other figure used for positioning the actual robot when placing it.

[0016] As shown in FIG. 1, the image processing device 1 according to this embodiment includes a memory unit 11, a positional relationship acquisition unit 12, a reception unit 13, an acquisition unit 14, an identification unit 15, an image generation unit 16, and an output unit 17.

[0017] The storage unit 11 may store information associating the type of robot, the type of tool attached to the robot, and the movable area when that type of tool is attached to that type of robot. The movable area may be an area through which at least a part of the robot passes when each link of the robot is moved within the movable range of the joint. The movable area may, for example, indicate an area in a planar direction (i.e., horizontal direction), or may be a three-dimensional area. If the no-entry area is a two-dimensional area, the movable area may, for example, indicate an area in a planar direction. Furthermore, if the no-entry area is a three-dimensional area, it is preferable that the movable area is also a three-dimensional area. The movable area may be obtained using, for example, a maximum reachable range or a P-point operating area. Specifically, the movable area may be an extension of the P-point operating area of ​​the robot depending on the type of tool attached to the hand of the robot. The movable area can also be calculated, for example, by knowing the length of each link of the robot, the movable range of each joint, and the shape and size of the tool attached to the hand. Therefore, the movable area stored in storage unit 11 may be calculated in this manner. Furthermore, when processing using the movable area is performed, a movable area according to the type of robot or the type of tool may be calculated. Furthermore, when processing to identify a safe placement area is performed only for a specific type of robot equipped with a specific type of tool, only the movable area according to that may be stored in storage unit 11.

[0018] Furthermore, as will be described later, information indicating the no-entry area acquired by the acquisition unit 14 may be stored in the storage unit 11. Furthermore, when a display image showing a virtual robot is generated, a three-dimensional model of the virtual robot may be stored in the storage unit 11. The three-dimensional model of the virtual robot may be, for example, a three-dimensional model of a virtual robot corresponding to a real robot whose introduction is being considered. The storage unit 11 may store three-dimensional models of multiple virtual robots corresponding to multiple real robots, respectively. Furthermore, information other than the above may be stored in the storage unit 11. The storage unit 11 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory, a magnetic disk, an optical disk, or the like.

[0019] The positional relationship acquisition unit 12 acquires the relative positional relationship between the real environment and the display device 2. In this embodiment, a case where the positional relationship acquisition unit 12 acquires the relative positional relationship between a marker 4 present in the real environment and the display device 2 will be mainly described. Acquiring the relative positional relationship between the marker 4 and the display device 2 may, for example, mean acquiring the relative positional relationship between a marker coordinate system, which is a local coordinate system of the marker 4, and a display coordinate system, which is a local coordinate system of the display device 2. This relative positional relationship may be represented by, for example, a homogeneous transformation matrix indicating the transformation between the two coordinate systems. The method by which the positional relationship acquisition unit 12 acquires this relative positional relationship is not important. For example, the positional relationship acquisition unit 12 may receive an image captured by a camera of the display device 2 and acquire a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system using three or more feature points of the marker 4 included in the image. The homogeneous transformation matrix may be acquired in the display device 2. In this case, the positional relationship acquisition unit 12 may receive the homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system from the display device 2. That is, the acquisition of the relative positional relationship by the positional relationship acquisition unit 12 may be acceptance of the relative positional relationship.

[0020] The receiving unit 13 receives a position based on the display device 2. This position may be, for example, a position indicating a no-entry area or a placement position of a robot to be placed. The position indicating a no-entry area may be, for example, a position indicating the outline of the no-entry area. In this embodiment, a case where the no-entry area is a two-dimensional area extending in a horizontal direction will be mainly described, and a case where the no-entry area is a three-dimensional area extending in a horizontal direction and a height direction will be described later. The no-entry area is an area where a robot must not enter, and may be, for example, a passageway where people pass or an area where existing objects are located. The receiving unit 13 may receive the position by, for example, a gesture of the user's hand or finger, or through an input device such as a teaching pendant present in the real environment, or through a virtual input interface such as a virtual button or virtual teaching pendant displayed on the display of the display device 2. The position received by the receiving unit 13 may be, for example, a position in a display coordinate system. In this embodiment, a case where the receiving unit 13 receives a position in a display coordinate system will mainly be described.

[0021] When a position is specified by a gesture, for example, the intersection of a line passing through the position of the user's fingertip and extending in the user's line of sight with the floor surface or the like in the real environment may be accepted as the specified position. Alternatively, the intersection of a line extending in the direction of the user's finger or the longitudinal direction of a rod-shaped device (e.g., a pen-shaped device) held by the user with the floor surface or the like in the real environment may be accepted as the specified position. Furthermore, when a position is specified by an input device or a virtual input interface, for example, a pointer may be displayed on the display device 2, and a position corresponding to the pointer in the real environment may be accepted as the specified position. In this case, for example, the intersection of a line passing through the pointer displayed on the display device 2 and extending in the user's line of sight with the floor surface or the like in the real environment may be accepted as the specified position. In accepting the position, if necessary, the position, orientation, and gaze direction of a finger or rod-shaped device may be acquired using sensing results (e.g., distance to surrounding objects, captured images, gaze direction, etc.) acquired by sensors such as a depth sensor, camera, and gaze sensor included in the display device 2. Alternatively, the position of the floor surface or the like in the real environment may be acquired and used to identify the position designated by the user. The process of identifying the position may be performed, for example, by the reception unit 13 or by the display device 2. In the former case, the reception unit 13 may also accept the sensing results and identify the position designated by the user using the sensing results. In the latter case, the reception unit 13 may accept coordinate values ​​indicating the position identified on the display device 2. In addition, when input is performed using a virtual input interface, the input result may be passed from the display device 2 to the reception unit 13. Alternatively, the hand tracking results acquired on the display device 2 may be passed to the reception unit 13, and the input result may be identified by the reception unit 13.

[0022] The receiving unit 13 may receive information other than the above. For example, the receiving unit 13 may receive information indicating the type of robot to be placed, information indicating the type of tool to be attached to the hand of the robot, and the like. The receiving unit 13 may also receive information indicating the final placement position of the robot, for example. Such information may be input, for example, from a virtual input interface in the display device 2, an input device present in the real environment, or the like.

[0023] Receiving unit 13 may, for example, receive information input from an input device or display device 2, or may receive information transmitted via a wired or wireless communication line. Note that receiving unit 13 may or may not include a device for receiving the information (for example, an input device or a communication device). Receiving unit 13 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0024] The acquisition unit 14 acquires the no-entry area in the real environment using the position indicating the no-entry area received by the reception unit 13 and the relative positional relationship acquired by the positional relationship acquisition unit 12. If the relative positional relationship acquired by the positional relationship acquisition unit 12 is a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the display coordinate system, the acquisition unit 14 may acquire the no-entry area in the real environment, for example, by using the homogeneous transformation matrix to convert the position in the display coordinate system into a position in the marker coordinate system. Note that, since the marker 4 is placed in the real environment, the marker coordinate system is considered to be the coordinate system in the real environment, i.e., the world coordinate system. If a coordinate system other than the marker coordinate system is used as the world coordinate system, for example, the no-entry area in the world coordinate system may also be acquired using a homogeneous transformation matrix indicating the transformation between the marker coordinate system and the world coordinate system. Acquiring the no-entry area in the real environment may mean, for example, acquiring information indicating the no-entry area in the real environment (e.g., coordinate values ​​indicating the position of the outline of the no-entry area).

[0025] The acquisition unit 14 may also acquire the placement position of the robot to be placed in the real environment using the placement position of the robot accepted by the acceptance unit 13 and the relative positional relationship. This placement position may also be acquired using a homogeneous transformation matrix, similar to the acquisition of no-entry areas in the real environment. Note that, since methods for identifying positions in real space using the display device 2 are already known, detailed descriptions of how to acquire no-entry areas in the real environment and the placement position of the robot will be omitted. Furthermore, as long as the position in real space can be acquired using the display device 2 as a result, the acquisition may be performed by a method other than the above.

[0026] The identification unit 15 uses the no-entry area acquired by the acquisition unit 14 and the movement area of ​​the robot to be placed to identify a safe placeable area, which is a placement area for the robot to be placed so that the robot to be placed does not enter the no-entry area. This safe placeable area may be, for example, an area in which the robot will not enter the no-entry area if the robot is present within the safe placeable area. In other words, the safe placeable area may be a placement area for the robot in which the movement area of ​​the robot does not overlap with the no-entry area. Note that if a specific position of the robot is within the safe placeable area, the robot may not enter the no-entry area. The specific position of the robot may be, for example, the center point or center of gravity of the end face of the base end of the robot (e.g., the mounting surface on a floor or the like). The specific position may correspond to, for example, a reference position (e.g., the center position) in the movement area of ​​the robot.

[0027] Furthermore, when the movable area is not isotropic, for example, when the two-dimensional movable area is not circular, the safe disposition area may be a robot placement area that prevents a robot placed in a predetermined orientation from entering a no-entry area. The orientation of the robot may be, for example, input by a user or may correspond to the orientation of the marker 4. In the latter case, the identification unit 15 may identify the safe disposition area when the robot to be placed is placed according to the orientation of the marker 4. In this case, the identification unit 15 may, for example, identify the safe disposition area when the robot is placed in a predetermined orientation in a marker coordinate system. Furthermore, the orientation of the robot input by the user may, for example, be an orientation in a world coordinate system. Furthermore, the orientation in the world coordinate system may, for example, be input as a numerical value or indicated by a gesture or the orientation of a finger or a stick-shaped device.

[0028] For example, the specification unit 15 may determine an area where the reference position of the robot movement area can be located in a situation where the robot movement area and the restricted area do not overlap, as the safe placement area. Specifically, in a situation where the robot movement area and the restricted area do not overlap but the outer edges of both areas are in contact, the specification unit 15 may specify a line along which the reference position of the robot movement area can be located, and specify an area that is farther away from the restricted area than the line as the safe placement area.

[0029] In addition, when the acquisition unit 14 acquires a placement position in the real environment corresponding to the placement position of the robot to be placed that has been accepted by the acceptance unit 13, the identification unit 15 may, for example, determine whether the placement position is within a safe placement area.

[0030] The image generating unit 16 generates a display image indicating the safe placement area using the safe placement area identified by the identifying unit 15 and the relative positional relationship. The image generating unit 16 may also generate a display image for displaying a 3D model of the virtual robot at the position of the marker 4 or at a position specified by the user. The virtual robot corresponds to the real robot to be placed. Displaying the 3D model of the virtual robot at the position of the marker 4 may mean displaying the 3D model of the virtual robot so as to virtually reproduce a situation in which the real robot is placed at the position of the marker 4. More specifically, the 3D model of the virtual robot may be displayed so that the base end face of the 3D model of the virtual robot coincides with the face of the marker 4 and a predetermined position (e.g., center point, center of gravity, etc.) on the base end face coincides with a predetermined position (e.g., center point, etc.) of the marker 4. The same applies when displaying the 3D model of the virtual robot at a position specified by the user. The orientation of the 3D model of the virtual robot is ,example For example, the orientation may be input by the user, or may correspond to the orientation of the marker 4. The display image generated by the image generating unit 16 is an image to be displayed on the display device 2.

[0031] The image generation unit 16 may generate a 2D display image by, for example, placing a 3D model of a safe placement area or a virtual robot in a virtual space corresponding to the real environment, and rendering the 3D model of the safe placement area or the virtual robot based on the position and orientation of the display device 2 in the virtual space. Since the virtual space corresponds to the real environment, the position and orientation of the display device 2 in the virtual space are indicated by the relative positional relationship acquired by the positional relationship acquisition unit 12. The angles of each joint in the 3D model of the virtual robot placed in the virtual space may be, for example, initial values ​​or may be freely changeable by the user. In the latter case, the angles of each joint may be set by operation or input using, for example, a teaching pendant or a virtual teaching pendant present in the real environment. Furthermore, for example, a tool of a type selected by the user may be attached to the hand of the 3D model of the virtual robot. The image generation unit 16 generates the display image so that the size of the display image of the 3D model of the virtual robot matches the size of the display image displayed on the display device 2. In other words, the display image is generated so that the three-dimensional model of the virtual robot displayed on the display of display device 2 and the actual robot placed in the real environment so as to have the same relative positional relationship as the three-dimensional model appear to be the same size when viewed through display device 2.

[0032] The output unit 17 outputs the display image generated by the image generation unit 16 to the display device 2. Note that the output unit 17 may output only the display image. In this case, the display image is displayed on the display device 2, superimposed on an image of the real environment or the real environment itself. On the other hand, if the display device 2 has a non-transmissive display and an image of the real environment captured by the display device 2 is accepted by the image processing device 1, the result of combining the image of the real environment and the display image may be output to the display device 2. This combination may be performed, for example, by a combination unit (not shown) included in the image processing device 1.

[0033] The display image output from the output unit 17 is displayed on the display device 2, allowing the user to know the safe placement area in real space. In addition, the three-dimensional model of the virtual robot is displayed at the position of the marker 4 or a position specified by the user, allowing the user to confirm the situation in which the robot is placed in real space.

[0034] When the identification unit 15 determines whether the placement position of the robot in the real environment is within the safe placement area, the output unit 17 may, for example, perform output according to the determination result. For example, a display image for displaying a 3D model of the virtual robot in a different color may be output depending on whether the placement position of the robot in the real environment is within the safe placement area. In this case, such a display image may be generated by the image generation unit 16. Furthermore, for example, the output unit 17 may output information indicating the determination result of whether the placement position of the robot in the real environment is within the safe placement area by displaying, outputting audio, or the like. The display output may, for example, be an output for displaying a character string indicating "inside the area" or "outside the area" on the display device 2.

[0035] Furthermore, when the acquisition unit 14 acquires the placement position of the robot to be placed in the real environment, the output unit 17 may output, for example, information indicating the placement position acquired by the acquisition unit 14 in terms of a position relative to the marker 4. This output may be performed, for example, in response to receipt of the placement position or in response to receipt of information indicating that the placement position of the robot has been finally determined. The information indicating the placement position in terms of a position relative to the marker 4 may be information indicating a position relative to the marker 4 (for example, a position XX centimeters to the right and XX centimeters below the center of the marker 4). When the world coordinate system is a marker coordinate system, the position relative to the marker 4 is indicated by the world coordinate system, i.e., the placement position in the real environment. When the world coordinate system is a coordinate system different from the marker coordinate system, the position relative to the marker 4 can be acquired by using the positional relationship between the marker coordinate system and the world coordinate system and the placement position in the world coordinate system. This position acquisition may be performed, for example, by the identification unit 15 or other components. This information may be output to the display device 2, transmitted to a predetermined device via a communication line, output as audio through a speaker, stored in a recording medium, or passed to another component. The output unit 17 may or may not include a device that performs the output. The output unit 17 may be realized by hardware, or may be realized by software such as a driver that drives such a device.

[0036] Next, the operation of the image processing device 1 will be described with reference to the flowchart of Fig. 2. It is assumed that the storage unit 11 of the image processing device 1 stores the robot's movable area in association with the type of robot and the type of tool.

[0037] (Step S101) The reception unit 13 determines whether or not the type of robot and the type of tool to be attached to the tip of the robot have been received. If the type of robot, etc. have been received, the process proceeds to step S102; if not, the process of step S101 is repeated until the type of robot, etc. have been received.

[0038] (Step S102) The reception unit 13 determines whether or not a plurality of positions indicating a no-entry area relative to the display device 2 have been received. If the plurality of positions have been received, the process proceeds to step S103; if not, the process of step S102 is repeated until the plurality of positions have been received. When the reception unit 13 receives a position indicating a no-entry area, a display image for displaying a figure indicating the position (for example, a dotted figure indicating the position) may be generated by the image generation unit 16 and output to the display device 2. Such a display allows the user to input a position while checking the position being input.

[0039] (Step S103) The positional relationship acquisition unit 12 acquires the relative positional relationship between the marker 4 and the display device 2 using the captured image of the marker 4 captured by the display device 2.

[0040] (Step S104) The acquisition unit 14 acquires the no-entry area in the real environment by converting the multiple positions indicating the no-entry area accepted in step S102 into positions in the real environment using the relative positional relationship acquired in step S103.

[0041] (Step S105) The identification unit 15 reads out from the memory unit 11 the movable area corresponding to the type of robot and the type of tool received in step S101, and identifies a safe placement area using that movable area and the no-entry area obtained in step S104.

[0042] (Step S106) The image generation unit 16 generates a display image for displaying the safe placement area identified in step S105 and a three-dimensional model of the virtual robot. Note that before the placement position is accepted, a display image for displaying the three-dimensional model of the virtual robot at the position of the marker 4 may be generated, and after the placement position is accepted, a display image for displaying the three-dimensional model of the virtual robot at the latest placement position may be generated.

[0043] (Step S107) The output unit 17 outputs the display image generated in step S106 to the display device 2. As a result, the display image is generated on the display device 2, and the user can confirm the position of the safe placement area and the placement position of the robot.

[0044] (Step S108) The reception unit 13 determines whether or not information indicating that the placement position of the robot has been finally determined has been received. If the information has been received, the process proceeds to step S109; if not, the process proceeds to step S110.

[0045] (Step S109) The output unit 17 outputs information indicating the placement position of the robot based on the marker 4. Then, the process returns to step S101. The placement position may be, for example, the most recent placement position among the accepted placement positions. Also, if a placement position has not been accepted, information indicating the position of the marker 4 as the placement position may be output.

[0046] (Step S110) The reception unit 13 determines whether or not a new placement position of the robot has been received. If a new placement position has been received, the process proceeds to step S111, and if not, the process returns to step S108.

[0047] (Step S111) The acquisition unit 14 acquires the placement position in the real environment using the placement position and the relative positional relationship accepted in step S110, and then returns to step S106.

[0048] If the relative positional relationship between the real environment and the display device 2 is likely to change while a new placement position is being accepted, a process of acquiring the relative positional relationship may be performed again before generating a display image or before acquiring a placement position in the real environment, and the display image may be generated using the latest relative positional relationship. The order of the processes in the flowchart of FIG. 2 is an example, and the order of the steps may be changed as long as the same results are obtained. In the flowchart of FIG. 2, the process ends when the power is turned off or an interrupt occurs to end the process.

[0049] Next, the operation of the image processing system 100 according to this embodiment will be described using a specific example. Fig. 3 is a diagram showing the situation in which the user views the robot placement area via the display device 2. In Fig. 3, markers 4 are placed in the real environment. Also, a passage 5 for people to move through exists in the real environment.

[0050] First, the user inputs the type of robot to be placed and the type of tool to be attached to the hand of the robot using the virtual input interface of the display device 2. Then, the information is received by the receiving unit 13 and passed to the identifying unit 15 (step S101). Note that the identifying unit 15 may store the information in the storage unit 11, for example.

[0051] Next, while viewing the real environment via the display device 2, the user points with his / her finger at point 6a on the passage 5 as shown in FIG. 4A. The position of point 6a in the display coordinate system is acquired by the display device 2 and input to the image processing device 1. A similar operation is performed for points 6b to 6d shown in FIG. 4B. As a result, the receiving unit 13 receives information on points 6a to 6d that indicate the outline of the no-entry area. Upon receiving the coordinate values ​​of the four points in the display coordinate system, the receiving unit 13 passes these coordinate values ​​to the acquiring unit 14 (step S102).

[0052] Thereafter, the positional relationship acquisition unit 12 acquires the relative positional relationship between the marker 4 and the display device 2 using the image of the marker 4 captured by the display device 2, and passes the relative positional relationship to the acquisition unit 14 and the image generation unit 16 (step S103). Upon receiving the relative positional relationship, the acquisition unit 14 converts the coordinate values ​​of the four points received from the reception unit 13 into coordinate values ​​in the marker coordinate system, and passes the converted coordinate values ​​to the identification unit 15 (step S104). 。 In this way, the position of the no-entry area in the world coordinate system (i.e., the marker coordinate system) is acquired.

[0053] Upon receiving the coordinate values ​​in the world coordinate system indicating the outline of the no-entry area, the identification unit 15 reads out from the storage unit 11 the movable area corresponding to the robot type and tool type received from the reception unit 13, and identifies a safe placement area using the movable area and the no-entry area (step S105). For example, if a no-entry area 6 and a movable area 7 are shown in a plan view as shown in FIG. 5, the identification unit 15 may identify a safe placement area 8, which is a hatched area. Note that in this specific example, as shown in FIG. 5, the movable area 7 is circular, with its center at the center point 4a of the end face on the base end side of the robot to be placed. Also, in FIG. 5, the movable area 7 is displayed so that the center point 4a is located at the center of the marker 4.

[0054] Thereafter, the image generation unit 16 generates a display image showing a 3D model of the virtual robot placed at the position of the marker 4 and the safe placement area 8 (step S106). The virtual robot corresponds to the type of robot accepted by the acceptance unit 13. The output unit 17 outputs the display image to the display device 2 (step S107). As a result, the safe placement area 8 and a 3D model 10 of the virtual robot are displayed on the display device 2 as shown in FIG. 6. A user wearing the display device 2 can know the safe placement area 8 from the display image. When the user specifies a new placement position in the safe placement area 8 with a finger or the like, the placement position is accepted by the acceptance unit 13 and converted into a placement position in the real environment (steps S110 and S111). Thereafter, a display image showing the 3D model of the virtual robot at the new placement position and the safe placement area 8 is generated again and output to the display device 2 (steps S106 and S107). Furthermore, when the user inputs information to finally determine the placement position of the robot, the information is received by the receiving unit 13 (step S108), and the output unit 17 outputs information indicating the placement position of the robot based on the position of the marker 4 (step S109). This information indicates the position at which the real robot should be placed relative to the marker 4. Therefore, by placing the real robot according to the output information, it is possible to place the real robot so that it does not enter the no-entry area.

[0055] As described above, the image processing device 1 according to this embodiment generates and outputs a display image indicating a safe placement area, making it easy to know where to place the real robot so that the real robot will not enter a no-entry area. Placing a marker 4 in the real environment also makes it possible to specify the safe placement area when placing the robot, for example, depending on the orientation of the marker 4. Furthermore, the output unit 17 outputs information indicating the placement position of the robot based on the position of the marker 4, making it easy to know where to place the robot in the real environment.

[0056] In the present embodiment, the user can input the placement position of the robot regardless of whether the position of the marker 4 is within the safe placement area. However, this is not necessarily the case. When the position of the marker 4 is within the safe placement area, the position of the marker 4 becomes the placement position of the robot. When the position of the marker 4 is not within the safe placement area, the placement position of the robot within the safe placement area may be received from the user and output. In this case, the receiving unit 13 may receive the placement position of the robot to be placed within the safe placement area from the user when the robot to be placed at the position of the marker 4 enters the prohibited area, i.e., when the position of the marker 4 is not within the safe placement area. Furthermore, the acquiring unit 14 may acquire the placement position of the robot to be placed in the real environment using the placement position received by the receiving unit 13 and the relative positional relationship. Then, the output unit 17 may output information indicating the acquired placement position by the position relative to the marker 4. The processing of the acquiring unit 14 and the output unit 17 is similar to that described above.

[0057] Furthermore, in the image processing device 1 according to this embodiment, the case where the placement position of the robot is input by the user has been described, but this is not necessarily the case. For example, if a robot to be placed at the position of marker 4 is likely to enter a prohibited area, the identification unit 15 may determine the placement position of the robot to be placed in a safe placement area. Note that if a robot to be placed at the position of marker 4 is not likely to enter a prohibited area, the position of marker 4 may be the placement position of the robot. For example, the identification unit 15 may determine the placement position of the robot to be a position in the safe placement area that is closest to the position of marker 4, may determine the placement position of the robot randomly in the safe placement area, or may determine the placement position of the robot based on other criteria. Note that the output unit 17 may output information indicating the determined placement position by its position relative to the marker.

[0058] The identification unit 15 may also identify a safe placement area within a predetermined placement candidate area. Similar to the no-entry area, the placement candidate area in the real environment may be acquired by the acquisition unit 14 using the position indicating the placement candidate area accepted by the acceptance unit 13 and a relative positional relationship. The position accepted by the acceptance unit 13 may be a position in a display coordinate system. Typically, when considering the placement of a robot, the range in which the target robot is to be placed is generally determined. Therefore, by setting an area including that range as the placement candidate area, it becomes possible to identify a safe placement area within a necessary and sufficient range. Furthermore, in some cases, identifying a safe placement area within a placement candidate area can reduce the processing load for identifying the safe placement area.

[0059] In this embodiment, the type of robot or the type of tool may be identified by the marker 4. For example, if the marker 4 is a two-dimensional code such as an AR marker or a QR code (registered trademark), the identifier of the two-dimensional code may be associated with the type of robot and the type of tool, and the identification unit 15 and the image generation unit 16 may use the type of robot and the type of tool to identify a safe placement area or generate a display image of a three-dimensional model of a virtual robot. Note that the identifier of the two-dimensional code may be, for example, information obtained by reading the two-dimensional code, or may be an identifier associated with the two-dimensional code.

[0060] Furthermore, in the present embodiment, the case where the positional relationship acquisition unit 12 acquires the relative positional relationship between the marker 4 arranged in the real environment and the display device 2 has been mainly described, but this is not necessarily the case. When the marker 4, which is a predetermined two-dimensional image, is not used, the positional relationship acquisition unit 12 may use, for example, an object existing in the real environment (e.g., a robot controller, a welding power source, a jig, etc.) as a marker, or may acquire the relative positional relationship between the real environment and the display device 2 by a method that does not use a marker. In the former case, an object existing in the real environment may be used as a marker, as in so-called markerless AR. Furthermore, when acquiring the relative positional relationship by a method that does not use a marker, for example, if an environmental map of the real environment is prepared, the positional relationship acquisition unit 12 may acquire the relative positional relationship, which is information indicating the position and orientation of the display device 2 in the three-dimensional real environment, by using a method such as SLAM (Simultaneous Localization and Mapping) or Visual-SLAM using the environmental map of the real environment and distances to surrounding objects and surrounding images acquired by the display device 2. In this case, for example, an environmental map may be stored in the image processing device 1, and the positional relationship acquisition unit 12 may acquire the relative positional relationship using the environmental map and the distances to surrounding objects and images of the surroundings received from the display device 2. Alternatively, for example, the relative positional relationship may be acquired by the display device 2 using a technique such as SLAM or Visual-SLAM, and the relative positional relationship may be passed to the positional relationship acquisition unit 12. In this case, the acquisition of the relative positional relationship by the positional relationship acquisition unit 12 may be performed by receiving the relative positional relationship. In this case, the display device 2 may have a depth sensor capable of measuring the distance to surrounding objects and a camera capable of acquiring images of the surroundings. The positional relationship acquisition unit 12 may also receive an image captured by the camera of the display device 2 and acquire a homogeneous transformation matrix indicating a transformation between the world coordinate system and the display coordinate system using three or more feature points of the robot controller or other objects included in the image.

[0061] Furthermore, as described above, the no-entry area may be a three-dimensional area. In this case, for example, as shown in FIG. 7 , a quadrangle having points 6a to 6d as vertices may be input from the user to point 6e, which is obtained by extending point 6a, the vertex closest to the user, upward. The no-entry area 6 may then be a quadrangular prism-shaped area whose base is a quadrangle having points 6a to 6d as vertices and whose height is the length from point 6a to point 6e. This quadrangular prism shape is typically a right prism. In this way, a three-dimensional no-entry area in the real environment may be acquired in response to receiving the upper limit position in the height direction of the three-dimensional no-entry area. Although the bottom of the no-entry area 6 is the floor in this case, this need not be the case. A three-dimensional no-entry area that is above the floor in the real environment may be acquired in response to receiving the upper limit position and the lower limit position in the height direction of the three-dimensional no-entry area. The upper limit position and the lower limit position may be received by a method other than specifying the position. For example, the upper limit and lower limit heights may be input as numerical values ​​or may be input using an input interface such as a slider.

[0062] The shape of the forbidden area is not limited. For example, the shape of the two-dimensional forbidden area may be a polygonal shape such as a square, rectangle, triangle, quadrangle, or pentagon, or may be a circle, sector, or ellipse. In the case of a polygonal shape, the forbidden area may be acquired by receiving the positions of each vertex. In the case of a circular forbidden area, the forbidden area may be acquired by, for example, receiving the position of the center and any position on the circumference. In the case of a sector-shaped forbidden area, the forbidden area may be acquired by, for example, receiving the position of the center and the positions of both ends of the arc. In the case of an elliptical forbidden area, the forbidden area may be acquired by, for example, receiving the position of the center (i.e., the intersection of the major axis and minor axis), the position of at least one of the two intersections of the ellipse and the major axis, and the position of at least one of the two intersections of the ellipse and the minor axis. If the no-entry area can have various shapes, the shape of the no-entry area may be selected before specifying the position, and the position may be input according to the selected shape. The three-dimensional no-entry area may also have a shape other than a polygonal prism. For example, it may have a spherical shape, a cylindrical shape, an ellipsoidal shape, or the like.

[0063] Furthermore, in the above-described embodiments, each process or function may be realized by centralized processing by a single device or a single system, or may be realized by distributed processing by multiple devices or multiple systems. For example, at least a portion of the configuration of the image processing device 1 may be physically included in a device having a display (e.g., display device 2). Therefore, the division of devices shown in FIG. 1 may be considered to be for convenience based on function rather than based on physical devices.

[0064] In the above embodiments, each component may be configured with dedicated hardware, or components that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. During execution, the program execution unit may execute the program while accessing a storage unit or recording medium. The program may also be executed by being downloaded from a server or the like, or by being read from a predetermined recording medium. The program may also be executed by a single computer or multiple computers. That is, centralized processing or distributed processing may be performed.

[0065] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Explanation of symbols]

[0066] 1 image processing device, 2 display device, 11 storage unit, 12 positional relationship acquisition unit, 13 reception unit, 14 acquisition unit, 15 identification unit, 16 image generation unit, 17 output unit

Claims

1. a positional relationship acquisition unit that acquires a relative positional relationship between the real environment and a display device that displays an image of the real environment or the real environment itself by superimposing the image; a reception unit that receives a position based on the display device; an acquisition unit that acquires a no-entry area in a real environment using the position indicating the no-entry area accepted by the acceptance unit and the relative positional relationship; an identification unit that identifies a safe placement area, which is a placement area for the robot to be placed, to prevent the robot from entering the no-entry area, using the acquired no-entry area and a movement area of ​​the robot to be placed; an image generating unit that generates a display image showing the identified safe placeable area using the relative positional relationship and the identified safe placeable area; an output unit that outputs the display image to the display device.

2. the positional relationship acquisition unit acquires a relative positional relationship between a marker present in a real environment and the display device; The image processing device according to claim 1 , wherein the specifying unit specifies a safe placement area when the target robot is placed, according to the orientation of the marker.

3. the specifying unit, when the position of the marker is not within the safe placement area, determines a placement position of the robot to be placed within the safe placement area; The image processing device according to claim 2 , wherein the output unit also outputs information indicating the determined placement position by a relative position with respect to the marker.

4. the receiving unit receives a placement position of the robot to be placed in the safe placement area when the position of the marker is not within the safe placement area; the acquisition unit acquires a placement position of the robot to be placed in a real environment using the placement position accepted by the acceptance unit and the relative positional relationship; and The image processing device according to claim 2 , wherein the output unit also outputs information indicating the acquired placement position by a relative position with respect to the marker.

Images