Methods for determining a pose of a mobile device

Abstract

The invention relates to methods for determining a pose of a mobile device, in particular of an at least partially automatedly moving vehicle or robot, in particular a robotic lawn mower, which moves or is intended to move in an environment, comprising the following steps: providing (400) a reference data set (402) which has multiple reference images of the environment captured by means of a sensor of the mobile device and a respective reference pose of the mobile device when the respective reference image is captured, wherein the reference poses correspond at least within predefined tolerances, and wherein the multiple reference images have each been captured under different capturing and / or environmental conditions (404); providing (410) a current data set (412) which has a current image of the environment captured by means of the sensor of the mobile device; determining (420) a selected reference image (428) from the multiple reference images; determining (430) a new pose (432) of the mobile device based on the reference pose, the selected reference image and the current image; and providing (440) the new pose of the mobile device, in particular for navigation of the mobile device.

Description

[0001] Description

[0002] title

[0003] Method for determining the pose of a mobile device

[0004] The present invention relates to a method for determining the pose of a mobile device, in particular a vehicle or robot that moves at least partially automatically, especially a robotic lawnmower, which moves or is intended to move in an environment, a computing unit and a computer program for carrying out this method, and a mobile device.

[0005] Background of the invention

[0006] Mobile devices or work equipment, such as vehicles or robots that move at least semi-automatically, typically move in an environment, in particular an environment to be processed or a work area, such as an apartment, in a garden, in a factory hall or on the street, in the air or in water.

[0007] Disclosure of the invention

[0008] According to the invention, a method for determining the pose of a mobile device, a computing unit and a computer program for its execution, as well as a mobile device with the features of the independent claims, are proposed. Advantageous embodiments are the subject of the dependent claims and the following description.

[0009] The invention relates generally to mobile devices that move, or at least can move, within an environment, such as a work area. These can also be referred to as mobile work equipment. Examples of such mobile devices (or mobile work equipment) include robots and / or drones and / or vehicles that move semi-automatically or fully automatically (on land, water, or in the air). Examples of robots include household robots such as cleaning robots (e.g., vacuuming and / or mopping robots), floor or street cleaning equipment, construction robots, or robotic lawnmowers, as well as other so-called service robots. Examples of vehicles that move at least partially automatically include passenger transport vehicles or goods transport vehicles (also known as industrial trucks, e.g., in warehouses), but also aircraft such as drones or watercraft.

[0010] Such a mobile device includes, in particular, a control unit and a drive unit for moving the device, enabling it to be moved within its environment, especially along a path. Navigation information can be determined based on the path and / or the device's pose; for example, specific instructions on the direction the device should move to follow the path. This information can then be implemented as control signals via the control unit and the drive unit.

[0011] For navigation, an environmental map can be used, which may have been obtained or determined using SLAM. SLAM (Simultaneous Localization and Mapping) is a robotics technique in which a mobile device, such as a robot, can simultaneously create a map of its environment and estimate its spatial position within that map. This facilitates obstacle detection and thus supports autonomous navigation.

[0012] This illustrates that one aspect of navigating such a mobile device is determining its pose, i.e., its position and orientation, and being able to refer back to it later. Furthermore, a mobile device or work tool can have one or more sensors that can capture its surroundings or information within them. These can be, for example, cameras, lidar sensors, or inertial sensors, which can be used to capture the environment and / or the movement of the mobile device, for example, in two or three dimensions.

[0013] Furthermore, such a mobile device can be configured to receive and / or send data via a communication link, in other words, to communicate and exchange data. This allows communication with the mobile device, for example, to give it instructions, transmit other data to it, or receive data or information from it. Wireless communication links are particularly relevant in this context. For this purpose, the mobile device can, for example, have corresponding (possibly different) modules for wireless communication, which can also be integrated into a processing unit.

[0014] With such a mobile device, it is also common for it to have functions that it can perform; that is, the mobile device is configured to carry out one or more, preferably different, functions. Instead of functions, one can also speak of applications. Such functions could be, for example, a work function or a training function.

[0015] The work function includes, for example, the automated movement of the mobile device within an environment and the execution of a work process within that environment, at least temporarily while moving. In the case of a robotic lawnmower, the work function could therefore include, for example, mowing (while moving). For this purpose, the mobile device can be given a start command to perform the work function, for example, by sending corresponding data via the wireless communication link.

[0016] One aspect of navigation can also be the initial learning of the boundaries or borders of the work area, such as the edge of a lawn or other area to be worked on, or other edges. This can also be referred to as a learning function.

[0017] The learning function includes, for example, manually controlling the mobile device's movement within its environment, particularly without performing a task (such as mowing the lawn) within that environment. In the case of a robotic lawnmower, the learning function can therefore include, for example, manually moving or controlling it along the perimeter of the working area to teach the robotic lawnmower the boundaries. For this purpose, the mobile device can be given (continuous) control commands to navigate or move within its environment, for example, by sending corresponding data via the wireless communication connection. This process is also referred to as learning or a "teach-in."

[0018] One aspect of this training process is finding or defining a boundary (or limit) that surrounds the work area, such as a garden or lawn. This boundary can then be used during operation to limit the automated movement of the mobile device.

[0019] When navigating a mobile device, especially after a boundary or limit has been defined during the learning process, it is usually necessary to determine a pose (i.e., position and orientation) of the mobile device. This can be done as part of normal navigation, for example, to follow a predefined movement path or trajectory, or to correct a drift that has occurred. In the latter case, a corrected pose is determined.

[0020] Determining a pose can be based on two images, captured, for example, by a sensor or camera of the mobile device. Particularly when correcting drift, reference images can be used, each associated with a reference pose—that is, a pose in which the reference image was captured. Such reference images can be captured at various reference poses distributed throughout the environment, especially at prominent locations or in relation to the work area. The pose can then be determined based on a current image and a reference image, for example, by tracking one or more features from the reference image in the current image. In this way, a difference pose can be determined between the reference pose of the reference image and the current pose of the current image.

[0021] However, as has now become apparent, tracking such features is often difficult. Surprisingly, this is due to sometimes significantly different acquisition and / or environmental conditions for the current image and the reference image.

[0022] Therefore, a method for determining a pose is proposed for a mobile device. For this purpose, a reference dataset is provided, containing several reference images of the environment captured by a sensor on the mobile device, along with a corresponding reference pose of the mobile device (or the relevant sensor) at the time each reference image was captured. The aforementioned camera is the primary suitable sensor, although images can also be captured using other sensors.

[0023] The reference poses here correspond at least within predefined tolerances, meaning that all reference images are assigned at least approximately the same pose. Ideally, all reference images would be captured at the same reference pose, but this is often not possible in practice and, as has been shown, also not necessary. The multiple reference images were each captured under different acquisition and / or environmental conditions.

[0024] Examples of such acquisition and / or environmental conditions include, for example, a time of day; a time of day, in particular differentiated by day, night, and twilight; a date; a season, in particular differentiated by spring, summer, autumn, and winter; an exposure situation, in particular differentiated by at least two or at least three different exposure situations; and a weather situation, in particular differentiated by rain, sun, cloud cover, snow, and wind. The reference images may differ in only one of these acquisition and / or environmental conditions or in several of them.

[0025] Furthermore, a current dataset is provided, containing a real-time image of the environment captured by the mobile device's sensor. A selected reference image is then chosen from among several reference images. This selection can be made taking into account the capture and / or environmental conditions of the current image and the multiple reference images, as will be explained in more detail later. In this way, a reference image can be selected that most closely matches the current image in terms of the capture and / or environmental conditions, thus enabling, for example, significantly better and more accurate feature tracking.

[0026] Based on the reference pose, the selected reference image, and the current image, a new pose of the mobile device is then determined and provided, particularly for navigating the mobile device.

[0027] For example, a difference pose can be determined based on the selected reference image and the current image; the new pose then results from the reference pose and the difference pose. Determining the difference pose can be done, in particular, as mentioned, by tracking one or more features between the current image and the reference image.

[0028] Tracking involves attempting to locate one or more features from one image in another. This is primarily based on an optical flow. A feature from one image will be found in a slightly different position in the other. Based on this shift of the feature from one image to the other, it can be determined how the pose of the mobile device must have changed from one image to the other; this corresponds to the difference pose. In the case of multiple features, individual difference poses can also be determined, which are then averaged to obtain a single difference pose.

[0029] It should also be mentioned here that feature tracking between images is different from feature matching. Feature matching involves only certain features that exist for a given pose; for example, there are reference features for a reference pose and actual features for the current pose. However, if the actual features, or many of them, have no corresponding reference features, a difference pose cannot be determined, or at least not very accurately.

[0030] Nevertheless, a comparison can also be made between the selected reference image and the current image in order to obtain a difference pose.

[0031] In one embodiment, the reference data set further includes information for each of the multiple reference images about the acquisition and / or environmental conditions under which the respective reference image was acquired. This information could, for example, be stored along with the data. Such information is already available for the current image. Preferably, determining the selected reference image then comprises determining or providing the acquisition and / or environmental conditions of the current image, and comparing the acquisition and / or environmental conditions of the current image with the acquisition and / or environmental conditions of the multiple reference images. Thus, the selected reference image is determined by directly considering the acquisition and / or environmental conditions of the current image and the multiple reference images.

[0032] Thus, each reference image can be assigned a specific date and time at which it was captured. Based on the date and time of the current image, the reference image captured at a similar date and time can then be selected. In this way, for example, the date can be used to ensure that the selected reference image was captured under similar seasonal conditions to the current image. If, for example, the current image was captured in summer, it will be difficult to trace features from a reference image captured in winter when there is snow on the ground. The time, on the other hand, can be used to ensure, at least approximately, that the selected reference image was captured under similar exposure conditions to the current image. For example, if the current image was captured in summer, it will be difficult to trace features from a reference image captured in winter when there is snow on the ground.Since the current image was captured at midday, it will be difficult to track features from a reference image captured at dusk.

[0033] While the date and time can be easily recorded when capturing the images, weather conditions can be determined, for example, by accessing a weather service or similar.

[0034] It should be noted that these explanations are merely examples and that very similar images may be found depending on the type of recording and / or environmental conditions used.

[0035] In one embodiment, the selected reference image is determined based on whether one or more conditions of the acquisition and / or environmental conditions (for the current image and the selected or to-be-selected reference image) are identical or at least correspond within predefined tolerances. This can vary depending on the condition. For example, the time of day should be identical, while the time of day can have a tolerance. In particular, a wide variety of combinations are also possible. In this way, a suitable reference image can be selected quickly and easily.

[0036] In one embodiment, determining the selected reference image involves comparing the current image with multiple reference images, particularly using a machine learning method. In other words, a suitable reference image can be found by directly comparing the images. For example, reference images captured at dusk can be easily and quickly excluded if the current image was captured in bright sunlight. Unlike the previous variant, the capture and / or environmental conditions are not directly considered here, but indirectly, since they affect the images themselves. Thus, the selected reference image is determined by indirectly considering the capture and / or environmental conditions of the current image and the multiple reference images, because the capture and / or environmental conditions do indeed have an effect on the images themselves.

[0037] In one embodiment, determining the selected reference image comprises, in a first step, determining or providing the acquisition and / or environmental conditions of the current image, and comparing the acquisition and / or environmental conditions of the current image with the acquisition and / or environmental conditions of the multiple reference images in order to obtain multiple preselected reference images. In a second step, this then comprises comparing the current image with the multiple preselected reference images, in particular using a machine learning method, to obtain the selected reference image.

[0038] In other words, this combines the two previously mentioned methods. In the first step, specific capture and / or environmental conditions can be taken into account, such as the date and / or the season. This allows for a simple and quick pre-selection and also simplifies image comparison in the second step.

[0039] In one embodiment, the reference data set comprises one or more reference features for at least two, preferably all, of the multiple reference images, each of which is identifiable in the two or at least two of the multiple reference images. The selection of the chosen reference image from the multiple reference images is then carried out taking into account one or at least one of the multiple reference features. In particular, the reference data set can have a weighting for the at least two of the multiple reference features that is related to the reliability of the identifiability of the reference feature.

[0040] These reference features can be determined, or have been determined, based on the multiple reference images, in particular as follows: In a first of the multiple reference images, one or more features are determined; then, it is checked one or more times whether the one or at least one of the multiple features can be determined in at least one further of the multiple reference images. If this is the case, the one or at least one of the multiple features is determined as the one or the multiple reference features.

[0041] In one embodiment, checking whether one or at least one of the several features can be determined in at least one further of the several reference images includes determining a quality measure according to which one or at least one of the several features can be determined in at least one further of the several reference images, wherein the weighting is assigned to the one or the several reference features based on the quality measure.

[0042] By weighting the reference features in relation to their reliability, specific features can be defined that can be tracked between two images—a reference image and a current image—independently of one or more of the acquisition and / or environmental conditions. Thus, the selected reference image can be chosen solely based on the reference features, or the reference features can be used as a selection criterion in addition to the acquisition and / or environmental conditions and / or image comparison.

[0043] The following section will explain a possibility or procedure for determining a reference data set for use in the navigation of a mobile device, wherein the reference images have been captured under different acquisition and / or environmental conditions, in such a way that the reference images differ sufficiently in the acquisition and / or environmental conditions.

[0044] For this purpose, one or more reference images of the environment captured by the sensor of the mobile device and a respective reference pose of the mobile device at the time of capture of the respective reference image are provided, whereby the reference poses agree at least within specified tolerances.

[0045] Furthermore, a new data set is provided, comprising a new image of the environment captured by the mobile device's sensor and a new pose of the mobile device at the time the new image was captured. The new pose must correspond, at least within predefined tolerances, to one or at least one of the several reference poses.

[0046] The process then checks whether one or more features can be traced or compared between the new image and one or at least one of the several reference images. This can be verified using the aforementioned tracing or matching methods.

[0047] If this is not the case, i.e., if no features can be traced or compared between the new image and one or at least one of the several reference images, the new dataset is added to the reference dataset. In this case, it can be assumed that the acquisition and / or environmental conditions in the new image differ significantly from those of the one or more (existing) reference images. Conversely, if one or more features can be traced or compared between the new image and one or at least one of the several reference images, it can be assumed that the acquisition and / or environmental conditions in the new image differ only slightly from those of the one or more (existing) reference images.

[0048] This can be repeated repeatedly, for example whenever a pose is reached during the movement of the mobile device in the environment that corresponds to a reference pose from the reference data set, at least within specified tolerances.

[0049] The resulting reference data set can then be provided, in particular for use in navigation of the mobile device.

[0050] The resulting reference dataset is such that no feature can be traced or compared between any two different reference images within the reference dataset. This applies in particular to any two of the multiple reference images, i.e., any different pair of reference images that can be formed.

[0051] A computing unit according to the invention (i.e., generally a system for data processing), e.g., a control unit or a control unit of a mobile device, or a central server or other computing system, is, in particular in terms of programming, equipped to carry out a method according to the invention.

[0052] The invention also relates to a mobile device configured to receive control information as described above. Furthermore, or alternatively, the mobile device comprises a computing unit according to the invention. The mobile device also includes a control unit and a drive unit for moving the device. Additionally, the mobile device may include sensor means for acquiring environmental information from the environment in which the mobile device is moving or is intended to move, such as the aforementioned camera or multiple cameras.

[0053] Preferably, the mobile device is designed as a vehicle that moves at least partially automatically, in particular as a passenger transport vehicle or a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g., a cleaning robot, floor or street cleaning device, or lawnmower robot, and / or as a drone. Implementing a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, as this incurs particularly low costs, especially if an executing control unit is also used for other tasks and is therefore already available. Finally, a machine-readable storage medium is provided with a computer program stored on it as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical storage media, such as...Examples include hard drives, flash memory, EEPROMs, DVDs, etc. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0054] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

[0055] The invention is schematically illustrated in the drawing using an exemplary embodiment and is described below with reference to the drawing.

[0056] Brief description of the drawings

[0057] Figure 1a schematically shows a mobile device to illustrate the invention.

[0058] Figure 1b schematically shows the mobile device from Figure 1a in a different view.

[0059] Figure 2 schematically shows an environment with a mobile device to illustrate the invention.

[0060] Figure 3 schematically shows two images to illustrate the invention. Figure 4 schematically shows a process flow in one embodiment.

[0061] embodiment(s) of the invention

[0062] Figure 1a schematically illustrates a mobile device 100, in particular a work tool, to explain the invention. Figure 1b shows the mobile device 100 in a different view and with different aspects. Figures 1a and 1b will be described together below.

[0063] The mobile work device 100 is, for example, a robotic lawnmower with a control unit 102 and a drive unit 104 (with wheels) for navigating or moving the robotic lawnmower 100 in an environment 120, and in particular in or on a work area 122, e.g., a lawn or garden. The robotic lawnmower 100 can, for example, move or be moved along a movement path or trajectory 130. Furthermore, the robotic lawnmower 100 has, for example, a sensor 106 designed as a camera. Images of the environment can be captured by means of the camera 106 and used for navigation. In addition, a docking station 110 is provided, for example, where the robotic lawnmower can be charged.

[0064] Figure 1b shows, by way of example, a building 124, a tree 126 and a person 128 in the surroundings, which may need to be taken into account as objects or obstacles when navigating the robotic lawnmower.

[0065] Furthermore, the robotic lawnmower 100 has a processing unit 108, e.g., a control unit, by means of which data can be received and / or sent. As already mentioned, this can be done via, for example, various types of wireless communication connections. These types of wireless communication connections are labeled 170 and 172 in Figure 1a. Examples include a mobile communication connection 170 and a Bluetooth connection 172. The processing unit 108 can, for example, have corresponding radio modules or be connected to such modules, which are then part of the robotic lawnmower 100.

[0066] Figure 1a shows a mobile input device 140, e.g., a smartphone, and a central computing system 150 (or a server, which can represent the so-called cloud). Various types of wireless communication links 170, 172 are provided between the robotic lawnmower 100 or its computing unit 108, the mobile input device 140, and the computing system 150. It should be noted that the mobile communication link 170 is established via a mobile transmitter 152, which in turn is connected to the computing system 150.

[0067] The connection between the robotic lawnmower 100, or rather its processing unit 108, and the mobile input device 140 can be either Bluetooth 172 or mobile network 170. In the latter case, the connection runs via the server 150. It goes without saying that appropriate radio modules are required for this.

[0068] The mobile work device, or the robotic lawnmower 100, can be configured to perform one or more functions. In one embodiment, these functions include a work function, e.g., a mowing function, and a learning function.

[0069] The work function includes, for example, the automated movement of the robotic lawnmower 100 in the environment 120 and the performance of a work operation (or processing operation) in the environment, e.g., mowing the lawn, at least temporarily while moving in the environment. Depending on the situation, it may also be that the robotic lawnmower initially drives to a specific location without mowing, in order to begin or continue mowing there.

[0070] The robotic lawnmower 100 can, for example, move independently within the working area 122 or the surrounding area 120, navigating and mowing the lawn. As already mentioned, various objects can be detected by the robotic lawnmower 100 or its camera 106 (i.e., they are visible in images captured by the camera) and then taken into account during navigation, i.e., when determining navigation information. For this purpose, the captured images or corresponding data can be transmitted to the computer system 150, for example, via the mobile network connection 170.

[0071] The learning function includes, for example, manually moving the robotic lawnmower around its surroundings, particularly along the perimeter of the working area 122, to teach the robotic lawnmower the boundaries. The actual mowing function does not need to be (or should not be) used during this process. For this purpose, the robotic lawnmower 100 can be continuously sent control commands (or commands) for navigation and movement. These control commands and corresponding data are sent from the mobile input device 140 to the robotic lawnmower via the Bluetooth connection 172. Likewise, the robotic lawnmower 100, or rather its processing unit 108, can send data back to the mobile input device 140 via the Bluetooth connection. This process is also referred to as learning or "teach-in."

[0072] Figure 2 illustrates the invention by showing an environment 220 with a building 224 and a tree 226, comparable to the environment shown in Figures 1a and 1b. Figure 222 also shows, by way of example, a work area, e.g., a lawn to be mowed. A mobile device or robotic lawnmower 200 and a docking station 210 are also shown by way of example.

[0073] Typically, the desired outcome is for the mobile device to cover the entire work area 222 as precisely as possible during processing, ensuring that the entire lawn is mowed. To achieve this, a boundary 221 must be defined for the work area, allowing the mobile device to process the area within this boundary. Images of the surroundings, particularly reference images, can be captured during the initial training or later. Specifically, certain images from several continuously or quasi-continuously captured images can be selected as reference images and saved.

[0074] This can occur, for example, at specific positions. Poses 240, 242, and 244a are shown as examples, with pose 244a shown in an enlarged detail view. These poses 240, 242, and 244a are located, for example, at curves or turns in the trajectory 230, and are also positioned such that the orientation of the mobile device and its camera (which is typically aligned in the direction of travel) is parallel to the boundary 223.

[0075] The pose (position and orientation) of the mobile device assumed during the capture of such reference images can then be assigned as a reference pose to the respective reference image. Such reference images can later be used—for example, during a subsequent processing operation that restarts from docking station 210 after a certain limit—to correct the pose of the mobile device and reduce drift.

[0076] Such a drift is indicated in Figure 2 with the modified trajectory 232. Here it can be seen that trajectory 232 deviates from the specified trajectory 230. Without correction, the drift would generally increase further.

[0077] The enlarged detail view in Figure 2 shows pose 244a, which is also a reference pose in which a reference image was captured and which is stored, for example, together with the reference pose as a reference data set.

[0078] As the mobile device moves along trajectory 232, it will assume pose 244b as its current pose at a certain point. Here, the current pose can be corrected by checking, for example, whether there is a reference image with a similar pose – in this case, it would be the reference image with pose 244a. The difference between reference pose 244a and the current pose 244b is indicated here by 234.

[0079] Therefore, the correct pose 244b can be determined as the new pose. As mentioned, this can be done using the reference image at reference pose 244a and the current image captured at pose 244a, for example, by tracking features between the two images. This also applies generally to determining a (new) pose, regardless of any drift correction.

[0080] As mentioned, due to different capture and / or environmental conditions when capturing the reference image and the current image, such tracking may be difficult or impossible.

[0081] Against this background, it is proposed, as mentioned, to provide multiple reference images in a reference dataset for a specific reference pose, or at least within certain tolerances of a reference pose, e.g., reference pose 244a, with each of the multiple reference images having been acquired under different acquisition and / or environmental conditions. This can be done not only for one reference pose, but for several or all reference poses present in an environment (or each within the certain tolerances).

[0082] Figure 3 schematically shows two images to illustrate the invention. Image 300a may be a first reference image at reference pose 244a according to Figure 2, while image 300b may be a second reference image at reference pose 244a or a very similar pose.

[0083] Accordingly, images 300a and 300b show, in particular, a building 324 in front and side views, as well as a tree 326 and, in the lower area, grass. To determine a pose, features between two images—a current image and a reference image—will now be compared or tracked as examples.

[0084] Figure 300a shows three features F1, F2, and F3 as examples. Feature F1 is, for example, a corner of building 324, feature F2 is part of the crown of tree 326, and feature F3 is part of the trunk or root of tree 326.

[0085] Furthermore, image 300a shows a sun 340, which is intended to illustrate that image 300a was taken during the day and in sunshine in terms of recording and / or environmental conditions.

[0086] In image 300b, only features F1 and F2 are visible, but also feature F4. Features F1 and F2 are the same as in image 300a, while feature F4 is a corner of a side window of building 324.

[0087] Furthermore, image 300b shows a cloud 342, which is intended to illustrate that image 300 may have been captured during the day or at dusk and under cloud cover in terms of recording and / or environmental conditions.

[0088] Accordingly, images 300a and 300b show different features, but also some of the same features, making them particularly useful for tracking purposes. For example, one reason why feature F4 is not visible in image 300a is due to sunlight reflecting off the window, and another reason why feature F3 is not visible in image 300b is due to insufficient illumination caused by a lack of sunlight. If there were snow, feature F1 might also be invisible. In autumn, feature F2 would be invisible because the tree would have little to no foliage. While these are just simple examples, they illustrate that the capture and / or environmental conditions of images can influence which features are visible and therefore usable for tracking or comparison.

[0089] Therefore, several such images, each with different capture and / or environmental conditions, can be used as reference images. The most suitable one can then be selected for tracking with a current image. As mentioned earlier, the reference images can also be such that no feature can be tracked or compared between two reference images; that is, for example, any two reference images might not share any common feature.

[0090] Furthermore, the example in Figure 3 shows that different features are visible to varying degrees in multiple images. For example, features F1 and F2 are visible in both images 30a and 300b, while features F3 and F4 are not. Therefore, features F1 and F2 could be used as the reference features mentioned earlier, if images 300a and 300b are the reference images.

[0091] Figure 4 schematically illustrates the process in one embodiment. The mobile device, or the robotic lawnmower, can, for example, move around its surroundings as described above and follow a path. For navigation purposes, it will need to repeatedly determine new poses.

[0092] In step 400, a reference data set 402 can be created, which contains several reference images of the environment captured by a sensor of the mobile device and a respective reference pose of the mobile device when the respective reference image was captured.

[0093] The reference images can correspond, for example, to images according to Figure 3, and the reference poses can correspond, for example, to pose 244a according to Figure 2. The reference poses agree at least within specified tolerances, and the multiple reference images were each acquired under different acquisition and / or environmental conditions 404, as explained with reference to Figure 3.

[0094] In particular, the reference images can be such that no feature can be traced or compared between two different reference images of the reference data set 602.

[0095] In step 410, the current data set 412 is then provided, which contains a current image of the environment captured by the mobile device's sensor. Steps 400 and 410 can be performed, for example, when the mobile device is at pose 244b as shown in Figure 2, although the specific pose there must first be determined.

[0096] In step 420, a selected reference image is then determined from the multiple reference images. This can, for example, in a first step 422, include providing the acquisition and / or environmental conditions of the current image, so that the acquisition and / or environmental conditions of the current image are then compared with the acquisition and / or environmental conditions of the multiple reference images in order to obtain several preselected reference images 424.

[0097] In a second step 426, the current image is then compared with the several preselected reference images, in particular using a machine learning method to obtain the selected reference image 428.

[0098] In step 430, a new pose 432 of the mobile device is then determined based on the reference pose (i.e., that of the selected reference image), the selected reference image and the current image.

[0099] The new pose is then made available in step 440, specifically for navigating the mobile device. Here, navigation information for the mobile device can be determined based on the new pose, and control information for moving the mobile device can be derived from that.

Claims

Claims 1. Method for determining the pose of a mobile device (100), in particular a vehicle or robot that moves at least partially automatically, especially a robotic lawnmower, which moves or is intended to move in an environment, comprising: Providing (400) a reference data set (402) which includes several reference images of the environment captured by a sensor (106) of the mobile device and a respective reference pose of the mobile device at the time of capture of the respective reference image, wherein the reference poses are consistent at least within specified tolerances, and wherein the several reference images have each been captured under different capture and / or environmental conditions (404); Providing (410) a current data set (412) which contains a current image of the environment captured by the sensor of the mobile device; Determine (420) a selected reference image (428) from the multiple reference images Determine (430), based on the reference pose, the selected reference image, and the current image, a new pose (432) of the mobile device; and Providing (440) the new pose of the mobile device, especially for navigation of the mobile device.

2. Method according to claim 1, wherein the reference data set further includes, for each of the multiple reference images, information about the acquisition and / or environmental conditions (404) under which the respective reference image was acquired.

3. The method of claim 2, wherein determining the selected reference image comprises: Determining or providing the capture and / or environmental conditions of the current image, and Comparing the acquisition and / or environmental conditions of the current image with the acquisition and / or environmental conditions of the multiple reference images.

4. Method according to one of the preceding claims, wherein the selected reference image is determined based on whether one or more conditions of the acquisition and / or environmental conditions are identical or at least correspond within specified tolerances.

5. Method according to any of the preceding claims, wherein determining the selected reference image comprises: Comparing the current image with the multiple reference images, especially using a machine learning method.

6. A method according to claim 5 and claim 3, wherein determining the selected reference image comprises: in a first step (422): determining or providing the acquisition and / or environmental conditions of the current image, and comparing the acquisition and / or environmental conditions of the current image with the acquisition and / or environmental conditions of the multiple reference images to obtain multiple preselected reference images (424); and in a second step (426): comparing the current image with the multiple preselected reference images, in particular using a machine learning method, to obtain the selected reference image (428).

7. Method according to any of the preceding claims, wherein the detection and / or environmental conditions (404) comprise at least one of the following conditions: a time of day; a time of day, in particular differentiated by day, night and twilight; a date; a season, in particular differentiated by spring, summer, autumn and winter; an exposure situation, in particular differentiated by at least two or at least three different exposure situations; a weather situation, in particular differentiated by rain, sun, cloud cover, snow and wind.

8. Method according to one of the preceding claims, wherein the reference data set comprises one or more reference features for at least two, preferably all, of the multiple reference images, each of which can be determined in the two or at least two of the multiple reference images, and wherein the determination of the selected reference image from the multiple reference images is carried out taking into account one or at least one of the multiple reference features.

9. Method according to claim 8, wherein the reference data set for the at least two of the several reference features has a weighting that is related to the reliability of a determinability of the reference feature.

10. Method according to one of the preceding claims, wherein no feature is traceable or comparable between two different reference images of the reference data set.

11. Method according to any of the foregoing claims, further comprising: Determine, based on the new pose, navigation information for the mobile device, Determine, based on navigation information, control information for moving the mobile device, and Providing control information and / or moving the mobile device based on the control information.

12. Computing unit comprising means for carrying out the method according to any of the preceding claims.

13. Mobile device (100) configured to receive control information determined by a method according to claim 11, and / or comprising a computing unit according to claim 12, and comprising a drive system and a control or regulation unit for controlling the drive system, and comprising a sensor, in particular a camera, for capturing images of an environment in which the mobile device moves or is intended to move, wherein the mobile device is configured in particular as a vehicle that moves at least partially automatically, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. cleaning robot, floor or street cleaning device or lawn mowing robot, and / or as a drone.

14. Computer program comprising instructions which, when the program is executed by a computer, cause it to execute the method according to claims 1 to 11.

15. Computer-readable storage medium on which the computer program according to claim 14 is stored.

Images