Ascertaining the spatial location of target objects for autonomous robots
Patent Information
- Authority / Receiving Office
- EP · EP
- Patent Type
- Applications
- Current Assignee / Owner
- SIEMENS AG
- Filing Date
- 2023-09-05
- Publication Date
- 2026-05-06
Smart Images
Figure EP2023074277_13032025_PF_FP_ABST
Abstract
Description
[0001] Description
[0002] Determination of the spatial position of target objects for autonomous
[0003] robot
[0004] The invention relates to a method for navigating an autonomous robot to a target object in an environment within a technical system. Furthermore, the invention relates to an autonomous robot for use in a technical system.
[0005] To use functions such as capturing images of a specific object on an Autonomous Mobile Robot (AMR), the first step is to scan the area of operation or the underlying environment. For this purpose, a 3D map is created from laser scans. During operation, the Autonomous Mobile Robot continuously takes additional environmental scans and compares them with the existing 3D map of the entire environment. This provides the Autonomous Mobile Robot with positioning information about its relative position within the environment.
[0006] In the final step, specific objects necessary for mission completion must be inserted into the 3D environment. Depending on their location in space, these objects are not always freely accessible or are simply difficult to reach.
[0007] It is known that an autonomous mobile robot uses tags to determine the position of objects in space when creating a mission. These tags must be manually attached to the object by a human. This is a complex process. The tag is then scanned by the autonomous mobile robot. This determines the distance to the object. The position of the tag in space can be determined based on the autonomous mobile robot's own position, its orientation, and the orientation of its camera, for example.
[0008] DE 10 2005 014 146 A1 discloses a target object detection system for detecting a target object in a surrounding area of a detector using an identifier provided on the target object.
[0009] The invention is based on the object of providing an efficient navigation method of an autonomous, mobile robot to a target object, and a correspondingly designed autonomous, mobile robot which can navigate particularly easily to the specific target object.
[0010] This object is achieved by a method for navigating an autonomous, mobile robot to a target object in an environment within a technical system according to claim 1 and by a method for navigating an autonomous, mobile robot to a target object in an environment within a technical system according to claim 2. Furthermore, the object is achieved by an autonomous, mobile robot for use in a technical system according to claim 8 and by an autonomous, mobile robot for use in a technical system according to claim 9. Advantageous further developments arise from the dependent claims.
[0011] A method according to the invention comprises the following method steps: a) Creating a three-dimensional map of the environment by the autonomous, mobile robot, wherein the autonomous, mobile robot also determines its own position in the environment, b) Capturing the environment by means of capturing means of the autonomous, mobile robot and transmitting currently captured image information to a remote human operator, c) Specifying the target object by the remote operator with the aid of the image information by directing a laser beam emitted by the autonomous, mobile robot by means of a laser onto the target object based on instructions from the operator, d) Determining a distance between the target object and the autonomous, mobile robot using a runtime method, e) Determining a position of the target object in the environment taking into account the determined distance, an orientation of the laser and a position of the autonomous,mobile robot in the environment, f) navigation of the autonomous mobile robot to the target object or sensory detection of the target object by sensor means of the autonomous mobile robot based on the previously determined position of the target object.
[0012] An autonomous, mobile robot is defined here as a robot that can move independently within its environment within a technical system. Furthermore, the robot may have devices that enable it to independently perform actions such as grasping, cutting, stirring, or welding. As explained in detail, a key task that such an autonomous, mobile robot (hereinafter referred to as "robot") is to perform is navigation to a specific target object within the technical system. To do this, the robot must know both its own position and the position of the target object within its environment.
[0013] In a first step, the robot creates a three-dimensional map of the environment in which it is located. In doing so, the robot determines its own position within the environment. The robot can create the three-dimensional map by scanning the environment based on emitted radiation and detecting the radiation reflected by the environment. For this purpose, the robot has a suitable radiation source that emits radiation into the robot's environment. The robot can capture a three-dimensional map of the environment using corresponding reflections from objects such as machines, pipes, or walls in the environment, which are detected by a detector on the robot.
[0014] At least part of the radiation can have a frequency in the visible spectrum. In other words, the radiation can be at least partially light. However, it is also possible that non-visible radiation such as infrared radiation is used to map the environment.
[0015] The robot can also create the three-dimensional map using a laser scanning technique. As is well known, the environment is scanned in a grid pattern using a laser beam, and the map of the environment is created using the laser beam's reflections.
[0016] After the robot has created the three-dimensional map of its environment, it transmits currently captured image or video information, which it acquires using image capture devices such as a video camera, to a human operator located remotely from the robot. An operator is defined as a human operator of the technical system. The operator interacts with the technical system or robot using special user interfaces and controls specific technical functions of the technical system or robot.
[0017] The remote operator then uses this image information and a laser integrated into the robot to direct the laser beam emitted by the laser precisely onto the specific target object that is to be the aim of the navigation. The operator does not have to be on site, which is a great advantage, especially in hazardous environmental conditions of the technical system. The robot aligns the laser according to the operator's instructions and the operator checks whether the laser is aiming at the target object correctly. If this is the case, the operator can give the robot the appropriate instruction to determine the position of the target object within the three-dimensional map it has created. To do this, the robot first determines the distance between the target object and the robot using a time-of-flight method. Using knowledge of its own position and the alignment of the laser orThe robot then calculates the exact position of the target object based on the path of the laser beam. Finally, it navigates to this target object based on the previously determined position.
[0018] Alternatively, the robot can also use the position of the target object to perform sensor detection of the target object. For this purpose, the robot can use special sensor devices. These could be, for example, a camera to capture an image, a thermal camera to detect heat radiation from the target object, a laser to perform gas analysis, or a directional microphone to detect sound emitted by the target object.
[0019] Preferably, the robot can change the orientation of the laser independently of the orientation of the entire robot. In other words, the laser is mounted on the robot so that it can rotate or pivot. The robot therefore does not have to laboriously change its entire orientation in order to aim the laser beam at the target object. Because the robot controls a corresponding servomotor of the laser, for example, the robot also knows the precise orientation of the laser that it needs to localize the target object. The robot can also change the orientation of the laser by changing its entire orientation. A related advantage of this is that no additional electromechanical components such as a servomotor are required for rotating / pivoting the laser.
[0020] The previously formulated problem is also solved by a method for navigating an autonomous, mobile robot to a target object in an environment within a technical system, with the following method steps: a) Creating a three-dimensional map of the environment by the autonomous robot, whereby the autonomous, mobile robot also determines its own position in the environment, b) Capturing the environment by depth camera means of the autonomous, mobile robot and transmitting currently captured image information to a remote human operator, c) Specifying the target object by the remote operator with the aid of the image information, in which the operator informs the autonomous, mobile robot at which position in the image information currently captured by the depth camera means the target object is located, d) Determining a distance between the target object and the autonomous,mobile robot using a runtime method of the depth camera means, e) determining a position of the target object in the environment taking into account the determined distance, the specifications of the depth camera means and a position of the autonomous mobile robot in the environment, f) navigating the autonomous mobile robot to the target object or sensory detection of the target object by sensor means of the autonomous mobile robot based on the previously determined position of the target object.
[0021] Instead of a separate laser used to locate the target object, this method uses depth cameras. These depth cameras are also known as time-of-flight cameras and determine the distance to objects in the robot's environment by measuring the respective time of flight of emitted radiation, particularly light. In this method, the depth cameras are used, on the one hand, to transmit a current image of the environment to the remote operator. After the operator has identified the specific target object in the camera image and transmitted this information to the robot, the robot can use the information from the depth camera to immediately calculate the distance between the robot and the target object.
[0022] The robot knows the specifications of the depth camera, in particular their spatial resolution, the aperture angle of the camera optics and the orientation of the depth camera, and can use this to determine the exact position of the target object. This method has the advantage that only one depth camera needs to be used and an additional laser is not (mandatorily) required. However, it can be useful to additionally provide a laser with the help of which the operator can determine the bearing of the target object more easily, especially at greater distances between the target object and the robot. The background to this is that depth cameras are usually designed for a closer range of a few meters and cannot resolve objects or can only resolve them less accurately at greater distances to the target object. For greater distances between the robot and the target object, the advantages of a laser with its highly focused radiation can come into play.
[0023] In addition, the task is solved by an autonomous, mobile robot for use in a technical system, which is trained to
[0024] - create a three-dimensional map of the autonomous mobile robot's environment and determine its own position in the environment; - capture the environment using image capture means and transmit currently captured image information to a remote human operator;
[0025] - to receive a target object for navigation from the remote operator by directing a laser beam emitted by the autonomous mobile robot by means of a laser onto the target object by the autonomous mobile robot in accordance with instructions from the operator;
[0026] - to determine a distance between the target object and itself using a time-of-flight method;
[0027] - to determine a position of the target object in the environment taking into account the determined distance, an orientation of the laser and a position of the autonomous mobile robot in the environment;
[0028] - to navigate to the target object based on the previously determined position of the target object or to perform sensor detection of the target object using sensor means of the autonomous, mobile robot.
[0029] In addition, the task is solved by an autonomous, mobile robot that is trained to:
[0030] - to create a three-dimensional map of the autonomous mobile robot's environment and to determine its own position in the environment;
[0031] - to capture the environment using depth camera means and to transmit currently captured image information to a remote human operator;
[0032] - to receive a target object for navigation from the remote operator, in which the remote human operator informs the autonomous mobile robot at which position the target object is located in the image information currently captured by the depth camera means;
[0033] - to determine a distance between the target object and itself using a time-of-flight method:
[0034] - to determine a position of the target object in the environment taking into account the determined distance, an orientation of the laser and a position of the autonomous mobile robot in the environment;
[0035] - to navigate to the target object based on the previously determined position of the target object or a sensory detection of the target object by sensor means of the autonomous, mobile robot.
[0036] The above-described properties, features, and advantages of this invention, as well as the manner in which they are achieved, will become clearer and more readily understood in connection with the following description of an embodiment, which is explained in more detail in conjunction with the drawings.
[0037] FIG 1 shows an autonomous, mobile robot within a technical installation environment; and
[0038] FIG 2 shows a flow diagram of a method according to the invention.
[0039] FIG 1 shows an autonomous, mobile robot 1 which is located within an environment 2 of a technical system, in particular a manufacturing or processing system. The robot 1 comprises image capture means 3 and a laser 4. For the sake of better readability, details of the internal structure of the robot 1 are omitted here. It should only be pointed out that the robot 1 is capable of moving within the environment 2. Furthermore, the robot 1 has a communication interface 5, by means of which the robot 1 can communicate with a remote operator 6 or can exchange data 9.
[0040] The robot 1 is designed to emit a laser beam 7 by means of the laser 4, which can be directed at a target object 8, which will be explained in more detail below with reference to the description of FIG 2.
[0041] In a first step I, the robot 1 creates a three-dimensional map of the environment 2. For this purpose, it can use, for example, the image capture device 3 or other means such as a laser scanner. In doing so, the robot 1 also determines its own position in the environment 2. This step I can be repeated as often as required, should this be necessary due to a movement of the robot 1 or due to the expiration of a certain waiting time.
[0042] In a subsequent second step II, the robot 1 captures the environment 2 using the image capture means 3. The robot 1 transmits the current image information or video information generated in this process to the remote human operator 6 via the communication interface 5. The transmission can be carried out, for example, on the basis of a 5G radio network.
[0043] In a third step III, the remote human operator 9 specifies a target object 8 (e.g. a container) to which the robot 1 should navigate (e.g. for the purpose of loading a workpiece) or which it should perform sensor detection. The operator uses the image information provided by the robot 1. He therefore does not have to be in the (potentially dangerous) environment 2 of the robot 1 to give the robot 1 instructions regarding the target object 8. The operator 9 aligns the laser beam 7, within the scope of his specifications, onto the target object 8 to which the robot 1 should navigate. For this purpose, the robot 1 provides a rotational or pivoting movement of the laser 4.Based on the targeted target object 8, the robot 1 then determines a distance between the robot 1 and the target object 8 in a fourth step IV by measuring a travel time of the emitted laser beam 7, which is reflected by the target object 8 and detected again by a detector 10 of the robot.
[0044] Based on the determined distance between the robot 1 and the target object 8, an alignment of the laser 4 and a position of the robot 1 in the environment 2, the robot 1 determines the position of the target object 8 in the environment 2 in a fifth step V.
[0045] In a final sixth step VI, the robot 1 finally navigates to the target object 8 based on the previously determined position of the target object 8 or carries out a sensory detection of the target object by sensor means of the robot.
[0046] Although the invention has been illustrated and described in detail by the preferred embodiment, the invention is not limited to the disclosed example, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention.
Claims
Patent claims 1. A method for navigating an autonomous, mobile robot (1) to a target object (8) or for sensory detection of the target object (8) by the autonomous, mobile robot (1) in an environment (2) within a technical system, comprising: a) creating a three-dimensional map of the environment (2) by the autonomous, mobile robot (1), wherein the autonomous, mobile robot (1) also determines its own position in the environment (2), b) detecting the environment (2) by image capture means (3) of the autonomous, mobile robot (1) and transmitting currently detected image information to a remote human operator (6), c) specifying the target object (8) by the remote human operator (6) with the aid of the image information, by directing a laser beam (7) emitted by the autonomous, mobile robot (1) by means of a laser (4) onto the target object (8) based on specifications from the remote human operator (56),d) Determining a distance between the target object (8) and the autonomous, mobile robot (1) by means of a runtime method, e) Determining a position of the target object (8) in the environment (2) taking into account the determined distance, an orientation of the laser (4) and a position of the autonomous, mobile robot (1) in the environment (2), f) Navigation of the autonomous, mobile robot (1) to the target object (8) or sensory detection of the target object (8) by sensor means of the autonomous, mobile robot (1) on the basis of the previously determined position of the target object (8).
2. Method for navigating an autonomous, mobile robot (1) to a target object (8) or for sensory detection of the target object (8) by the autonomous, mobile robot ter (1) in an environment (2) within a technical system, comprising: a) creating a three-dimensional map of the environment (2) by the autonomous robot (1), wherein the autonomous, mobile robot (1) also determines its own position in the environment (2), b) capturing the environment (2) by depth camera means (3) of the autonomous, mobile robot (1) and transmitting currently captured image information to a remote human operator (6), c) specifying the target object (8) by the remote human operator (6) with the aid of the image information, in that the remote human operator (6) informs the autonomous, mobile robot (1) at which position in the image information currently captured by the depth camera means (3) the target object (8) is located, d) determining a distance between the target object (8) and the autonomous, mobile robot (1) by means of a runtime method of the depth camera means (3),e) Determining a position of the target object (8) in the environment taking into account the determined distance, the specifications of the depth camera means (3) and a position of the autonomous, mobile robot (1) in the environment (2), f) Navigation of the autonomous, mobile robot (1) to the target object (8) or sensory detection of the target object (8) by sensor means of the autonomous, mobile robot (1) on the basis of the previously determined position of the target object (8).
3. Method according to claim 1 or 2, wherein the autonomous mobile robot (1) creates the three-dimensional map by means of scanning the environment (2) based on emitted radiation and detection of the radiation reflected by the environment (2).
4. The method of claim 3, wherein at least a portion of the radiation has a radiation frequency in the visible spectrum.
5. Method according to claim 3 or 4, without claim 2, wherein the autonomous mobile robot (1) creates the three-dimensional map by means of a laser scanner method.
6. Method according to claim 1 or according to one of claims 3 to 5, without claim 2, in which the autonomous mobile robot (1) the orientation of the laser (4) can change independently of the orientation of the entire autonomous mobile robot (1).
7. Method according to claim 1 or according to any one of claims 3 to 5, without claim 2, in which the alignment of the laser (4) is changed by the autonomous mobile robot (1) changing its entire orientation.
8. Autonomous, mobile robot (1) for use in a technical system, which is designed to - to create a three-dimensional map of an environment (2) of the autonomous mobile robot (1) and to determine its own position in the environment (2); - to capture the environment (2) by image capture means (3) and to transmit currently captured image information to a remote human operator (6); - to receive a target object (8) from the remote human operator (6) for navigation or for sensory detection by directing a laser beam (7) emitted by the autonomous mobile robot (1) by means of a laser (4) onto the target object (8) by the autonomous mobile robot (1) following instructions from the remote human operator (6); - to determine a distance between the target object (8) and itself by means of a time-of-flight method; - to determine a position of the target object (8) in the environment (2) taking into account the determined distance, an orientation of the laser (4) and a position of the autonomous, mobile robot (1) in the environment (2); - to navigate to the target object (8) on the basis of the previously determined position of the target object (8) or to detect the target object (8) by sensor means of the autonomous, mobile robot (1).
9. Autonomous mobile robot (1) which is trained to - to create a three-dimensional map of an environment (2) of the autonomous mobile robot (1) and to determine its own position in the environment (2); - to capture the environment (2) by depth camera means (3) and to transmit currently captured image information to a remote human operator (6); - to receive a target object (8) from the remote human operator (6) for navigation or for sensory detection, in which the remote human operator (6) informs the autonomous mobile robot (1) at which position the target object (8) is located in the image information currently captured by the depth camera means (3); - to determine a distance between the target object (8) and itself using a time-of-flight method: - to determine a position of the target object (8) in the environment (2) taking into account the determined distance, an orientation of the laser (4) and a position of the autonomous, mobile robot (1) in the environment (2); - to navigate to the target object (8) on the basis of the previously determined position of the target object (8) or to detect the target object (8) by sensor means of the autonomous, mobile robot (1).
10. Autonomous mobile robot (1) according to claim 8 or 9, which is designed to create the three-dimensional map by means of scanning the environment (2) based on emitted radiation and detection of the radiation reflected from the environment (2).
11. The autonomous mobile robot (1) according to claim 10, wherein at least a portion of the radiation has a radiation frequency in the visible spectrum.
12. The autonomous mobile robot (1) according to claim 10 or 11, excluding claim 9, which is configured to create the three-dimensional map using a laser scanning method.
13. The autonomous mobile robot (1) according to claim 8 or any one of claims 10 to 12, excluding claim 9, which is configured to change the orientation of the laser (4) independently of the orientation of the entire autonomous mobile robot (1).
14. Autonomous mobile robot (1) according to claim 8 or any one of claims 10 to 12, excluding claim 9, which is adapted to change the orientation of the laser (4) by changing its entire orientation.