Method for error handling during the navigation of a mobile device

Abstract

The invention relates to a method for error handling during navigation of a mobile device (100), wherein the mobile device moves or is intended to move in an environment, using SLAM-based localisation, comprising: - repeatedly carrying out localisation processes (501, 502, 503), wherein each localisation process comprises: providing (200) device and environment information which have been sensed at least in part by means of sensors of the mobile device from the device and / or the environment, creating a dataset (201') based on the device and environment information, and providing (230) the dataset (201') for use for the localisation and / or navigation of the mobile device (100); - checking (510), during the localisation processes, whether there is an error (F), and determining (512) a dataset, during the localisation process (502) of which there was an error, as a potentially faulty dataset (513), and determining a dataset, during the localisation process (501, 503) of which there was no error, as a correct dataset (514), and - if a proper dataset is obtained after a potentially faulty dataset: checking (520) the potentially faulty dataset to see whether it is used further or not.

Description

[0001] Description

[0002] title

[0003] MOBILE DEVICE NAVIGATION TROUBLE HANDLING PROCEDURES

[0004] The present invention relates to a method for error handling during navigation of a mobile device, in particular of an at least partially automated vehicle or robot, a system for data processing and a computer program for carrying out the method, as well as a mobile device.

[0005] Background of the invention

[0006] Mobile devices such as at least partially automated vehicles or robots typically move in an environment, in particular an environment to be processed or a work area, such as a home, in a garden, in a factory hall or on the street, in the air or in water. One of the fundamental problems of such or other mobile devices is to orient itself, i.e. to know what the environment looks like, in particular where obstacles or other objects are, and where it is (absolutely) located. For this purpose, the mobile device can be equipped with various sensors, such as cameras, lidar sensors, radar sensors or inertial sensors, with the help of which the environment and the movement of the mobile device can be recorded, for example in two or three dimensions.

[0007] This allows the mobile device to move locally, detect obstacles in a timely manner and avoid them.

[0008] Disclosure of the invention

[0009] According to the invention, a method for error handling during navigation of a mobile device, a data processing system and a computer program for implementing the method, as well as a mobile device having the features of the independent patent claims are proposed. Advantageous embodiments are the subject of the dependent claims and the following description.

[0010] The invention generally relates to mobile devices that move or at least can move in an environment or there, e.g. in a work area. Examples of such mobile devices (or mobile work devices) are, for example, robots and / or drones and / or partially or (fully) automated vehicles (on land, water or in the air). Possible robots include, for example, household robots such as cleaning robots (e.g. in the form of vacuum and / or mop robots), floor or street cleaning devices, construction robots or lawnmower robots, but also other so-called service robots, as well as at least partially automated vehicles, e.g. passenger transport vehicles or goods transport vehicles (including so-called industrial trucks, e.g. in warehouses), but also aircraft such as so-called drones or watercraft.

[0011] Such a mobile device comprises, in particular, a control or regulating unit and a drive unit for moving the mobile device, allowing the mobile device to be moved within its environment, e.g., along a movement path. For this purpose, navigation information can be determined, such as specific instructions regarding the direction in which the mobile device should travel in order to follow the movement path. These instructions can then be implemented via the control or regulating unit and the drive unit. In general, this can be referred to as the navigation of the mobile device.

[0012] In addition, a mobile device can have one or more sensors by means of which the environment or information in the environment can be recorded. As mentioned, these can be, for example, cameras, lidar sensors, radar sensors or inertial measuring units (or inertial sensors) as well as radodometry, with the help of which the environment and the movement of the mobile device are recorded, for example, two- or three-dimensionally. Depending on the type of mobile device, other or additional sensors can also be provided. The sensors can be differentiated in particular according to different types of sensors, namely so-called device sensors and so-called environmental sensors. Using device sensors, e.g. inertial measuring units and radodometry, device information, in particular device sensor data, can be recorded and provided, which includes information about a position and / or movement of the mobile device, for examplea speed, acceleration, or yaw rate of the mobile device. Environmental sensors, such as cameras, lidar sensors, or radar sensors, can be used to capture and provide environmental information, particularly environmental sensor data, which includes information about the mobile device's surroundings, including, for example, the distance of the mobile device (or the relevant sensor) to objects in the environment.

[0013] One way to determine the position and orientation, and thus also the navigation, of such a mobile device is to use localization based on SLAM. SLAM ("Simultaneous Localization and Mapping") is a robotics technique in which a mobile device, such as a robot, can or must simultaneously create a map of its surroundings and estimate its spatial position within this map. It thus serves to detect obstacles and thus supports autonomous navigation.

[0014] In SLAM, there are various approaches to representing maps and positions. Conventional SLAM methods generally rely on geometric information such as nodes and edges. Nodes and edges are typically components of the SLAM graph. The nodes and edges in the SLAM graph can take on various forms; traditionally, the nodes correspond, for example, to the pose (position and orientation) of the mobile device or certain environmental features at specific times, while the edges represent relative measurements between the mobile device and the environmental feature. SLAM graphs are described in more detail in “Giorgio Grisetti, Rainer Kümmerle, Cyrill Stachniss, Wolfram Burgard, A Tutorial on Graph-Based SLAM, IEEE Intelligent Transportation Systems Magazine, Vol. 2(4), pp. 31-42, 2010.” Based on such a SLAM graph, a map of the environment (environment map) in which the mobile device moves can be determined or will be determined.With each new data set containing information about the environment and / or the mobile device, which information is obtained from or based on one or more sensors of the mobile device, the map (or the SLAM graph) can be extended or updated.

[0015] Localization processes can be used for localization using SLAM. A localization process generally comprises the provision of device and environmental information that has been at least partially acquired from the device and / or the environment using sensors of the mobile device, e.g. the aforementioned device and / or environmental information. Based on the device and environmental information, a data set is generated, which is then made available for use in localizing and / or navigating the mobile device. For example, the environmental information or environmental sensor data can comprise a so-called point cloud, as provided by typical lidar sensors. The device information can then be used additionally to determine a position and orientation (together also referred to as a so-called pose) of the mobile device for this point cloud.

[0016] Such a data set obtained during a localization process can then be compared with a previous data set obtained during a previous localization process. The attempt is made to align the two data sets, at least within certain tolerances, in order to determine the movement or trajectory of the mobile device. This can also be referred to as a so-called match or scan match (the latter refers in particular to the case of a lidar sensor that captures data during a scan). The comparison of a data set with a (typically immediately preceding) data set is also referred to as incremental localization.

[0017] Often, however, not every new dataset is added to the aforementioned environmental map; rather, a new dataset may be added to the environmental map only at certain temporal and / or spatial intervals. This keeps the data volume smaller.

[0018] While localization using SLAM generally works very well under favorable conditions, it has been shown that a number of problems can arise in practice, for example, with incremental localization. Such problems can include the loss of sensor data, wheel slippage, so-called kidnapping (where the mobile device, usually a household robot, is picked up and then placed somewhere else), and sensor occlusion (this particularly affects environmental sensors).

[0019] Reusing the results of a localization process, especially an incremental localization process in which an error has occurred, can lead to reduced accuracy (in localization) or complete failure of the SLAM algorithm. Against this background, we propose ways to deal with potential errors in mobile device navigation.

[0020] One embodiment relates to a method for determining a position and orientation, and thus in particular also for navigation, of a mobile device, wherein the mobile device moves or is intended to move in an environment using SLAM-based localization.

[0021] Here, data is provided that includes information about the position and / or movement of the mobile device. This can be the device information already mentioned. The data has been obtained from various types of sources. Possible sources include sensors such as inertial measurement units and radodometry, but also other sources such as models or assumptions for a current speed. The term “sources of various types” should be understood in particular to mean that there are several sources, e.g. a sensor and a model or two different, particularly different, sensors. Here, it is provided that one or more partial aspects of the position and orientation (pose) are each assigned a set of sources from the different types of sources. In each respective set, the sources are in turn assigned a predetermined hierarchy of priorities. While position and orientation, for example,can have six degrees of freedom, three translational degrees of freedom (position) and three rotational degrees of freedom (orientation), a partial aspect is understood as a portion of these total degrees of freedom. However, it should be noted that even with position and orientation, not all six degrees of freedom necessarily have to be taken into account; for example, in a mobile device that only moves on one plane, a vertical direction may be ignored. The given hierarchy of priorities then means that there is an order of the sources of the set, according to which the sources are arranged.

[0022] A simple example is that the yaw angle (a rotational degree of freedom), as a sub-aspect of position and orientation, is assigned a set of sources comprising an inertial measurement unit, the wheel odometry, and a velocity model. The predefined hierarchy of priorities can, for example, correspond to the order in which the three sources are listed here.

[0023] The availability of at least some of the sources of various types for the respective sentence (and thus also for the respective sub-aspect) is then checked. Furthermore, it is determined whether the checked sources are available or not. The availability of a source can be understood in particular as whether the source provides its information at all or not, or whether a source, for example, provides inaccurate information that should therefore not be used.

[0024] Based on the data, the position and orientation (in particular, a so-called pose) of the mobile device are determined for localizing the mobile device. One or at least one of the multiple sub-aspects is determined based on the data from a selected source from the respective assigned set of sources. In the example mentioned, the yaw rate is determined, for example, either based on data from the inertial measurement unit, based on data from the wheel odometry, or based on data from the speed model.

[0025] The selected source is the one from the respective set of sources that has the highest priority according to the respective hierarchy among the sources determined as available. If, in the example above, all three sources are available, the data from the inertial measurement unit is used to determine the yaw rate. This also applies if at least the inertial measurement unit is available, but, for example, the wheel odometry is not available. However, if the inertial measurement unit is not available but the wheel odometry is, the yaw rate is determined based on the wheel odometry.

[0026] This hierarchy allows the best possible source to always be used, ensuring sufficient accuracy even in the event of errors that lead to the unavailability of a source. If, however, multiple sources were weighted against each other, minor inaccuracies could be compensated for, but larger deviations from an erroneous source would significantly distort the result. The hierarchy or order of sources in such a set can be determined, for example, depending on the needs and situation or the type of sources.

[0027] It should be noted that this only ever applies to one respective sub-aspect, i.e. different sources and / or a different hierarchy can be provided depending on the sub-aspect. For example, two specific sensors can be provided for two different sub-aspects, but with a different hierarchy, e.g. because one sensor provides more precise values ​​for one sub-aspect than the other sensor, and vice versa for the other sub-aspect. Likewise, determining whether a source is available or not also applies individually for each sub-aspect or the respective set of sources. For example, a specific sensor can be assigned to two different sub-aspects, but only be considered unavailable for one of them. This can be the case, for example, if this sensor provides a detected inaccuracy, and a sensor with more precise data is assigned to one sub-aspect but not to the other sub-aspect.If, on the other hand, the sensor no longer provides any data, it will no longer be considered available for any of the sub-aspects.

[0028] In one embodiment, the sources of different types comprise at least two of the following sources: one or more sensors, one or more position and / or motion models, and one or more position and / or motion specifications. In particular, the sources of different types comprise an inertial measurement unit, a wheel odometry unit, a target speed, a constant speed model, a constant speed, a constant position model, and a constant position.

[0029] While the sensors, e.g. inertial measurement unit and radodometry, provide current measured values, the models can be used as a fallback solution.

[0030] If, for example, no sensor is available, it can be assumed that the mobile device is moving at the last measured or determined speed or is at the last determined position. However, if available, specifications, such as a target speed, can also be used, which does not necessarily have to be constant. This applies not only to positions and translational speeds, but also to orientations and rotational speeds.

[0031] In one embodiment, one or more of the sub-aspects of the position and orientation comprise a position in the horizontal direction of movement, e.g., in the x- and / or y-direction (in the case of a Cartesian coordinate system). This sub-aspect is then assigned the set of sources with the following predefined hierarchy: the wheel odometry, the target speed, the constant speed model, and the constant position model. In one embodiment, one or more of the sub-aspects of the position and orientation comprise a yaw angle. This sub-aspect is then assigned the set of sources with the following predefined hierarchy: the inertial measurement unit, the wheel odometry, the target speed, the constant speed model, and the constant position model.

[0032] In one embodiment, the one or more sub-aspects of the position and orientation comprise a pitch and / or roll angle. This sub-aspect is associated with the set of sources with the following predefined hierarchy: the inertial measurement unit, the constant velocity, and the constant position.

[0033] The three sub-aspects mentioned enable the most accurate determination of the mobile device's position and orientation, and thus its navigation, even in the event of errors or other reasons why a source or sensor is unavailable. It should be noted that even if the proposed approach is applied to only one of the sub-aspects, an improvement is already achieved.

[0034] In one embodiment, if two sources of different types, each comprising a sensor, are assigned to the same partial aspect of the position and orientation, in particular the yaw angle, it is determined that one of the two sources is unavailable if the corresponding data received from the two sources deviate from each other by more than a predetermined value. One of the two sources can comprise the inertial measurement unit, and the other of the two sources the radodometry. In general, however, this can also be the case with two different sensors, by means of which a certain partial aspect can be determined. The two sensors can provide a different type of data, but with which the same partial aspect can be determined. Corresponding data must then be compared. While with the inertial measurement unit such as a gyroscope, for examplea rotation rate is determined, based on which the yaw rate can then be determined (or which can be used as such), the radodometry provides, for example, lengths of traveled paths of two wheels, which can be compared in order to determine the yaw rate based on this.

[0035] Likewise, the two sensors do not have to be different types of sensors; they can also be two similar sensors, e.g. two inertial measuring units, which, for example, have a different quality or are arranged at different positions on the mobile device.

[0036] It is particularly preferred that if the corresponding data from the source comprising the inertial measurement unit and the source comprising the wheel odometry differ by more than the predetermined value, it be determined that the source comprising the wheel odometry is unavailable. In other words, if the corresponding values ​​provided by the inertial measurement unit and the wheel odometry for, for example, the yaw rate differ significantly, preference is given to the inertial measurement unit. It has been found that, at least over a short period of time, the inertial measurement unit provides more accurate measurements than the wheel odometry. If one of the two is faulty, it is therefore more likely that the wheel odometry is at fault.

[0037] As already mentioned, this does not necessarily mean that wheel odometry is no longer used for another aspect. For the horizontal position, the information from wheel odometry may still be more accurate than that from a model or the target speed. However, depending on the degree of discrepancy between the values ​​from the inertial measurement unit and wheel odometry, wheel odometry may also be determined to be unavailable, for example, for the horizontal position.

[0038] In one embodiment, it is determined that a source, in particular a sensor, is unavailable if no data is or has been received from the source for at least a predetermined period of time, and / or if an error is detected for which it is specified that the source should not be available. In this way, further errors can be taken into account. If no data is received for a source or a sensor for at least a predetermined period of time, this source can, as mentioned, be determined to be unavailable, in particular for several sub-aspects (if this source is assigned to several sub-aspects). In this case, it can then be assumed, for example, that the sensor or a communication connection is defective.

[0039] Various possibilities can be considered for errors for which the source is specified as unavailable. For example, a so-called lift-up sensor can be provided on the mobile device, which is triggered when the mobile device is lifted, especially in the case of a household robot. This can be, for example, a contact sensor built into a wheel suspension of the mobile device. If a lifting can be detected, especially as long as the mobile device is lifted, the wheel odometry can be determined to be unavailable - without contact with the ground, the wheel odometry will not provide reasonable values. Such a lifting, as well as a subsequent setting down, can also be detected, for example, by the inertial measurement unit. Another error is, for example, a jump in the wheel odometry or its data. Likewise, a slip in the yaw rate, which can be detected, for example, by comparing the data or values ​​from the inertial measurement unit and the wheel odometry (this can, for example,occur when the mobile device is moved laterally by an external force). All these errors indicate, for example, that wheel odometry should not be used, ie, it should be determined as unavailable.

[0040] Another error is, for example, a robot standstill, which can be detected based on the inertial measurement unit and / or wheel odometry. Standstill can be used, for example, to reduce the computational load by avoiding unnecessary calculations and to eliminate the accumulation of small errors while the mobile device is stationary (e.g., due to a gradual drift of the inertial measurement unit).

[0041] In one embodiment, navigation information for the mobile device is further determined based on the position and orientation of the mobile device. This can then include, in particular, the consideration of additional sensor data, such as the lidar sensor, as already mentioned and as will be explained in more detail below. One embodiment relates to a method for localizing, and thus in particular also for navigating, a mobile device, wherein the mobile device is moving or is intended to move in an environment using SLAM-based localization.

[0042] In a localization process, device information is provided, in particular device sensor data that has been recorded using sensors of the mobile device. The device information includes information about a position and / or movement of the mobile device. In addition, environmental sensor data that has been recorded using sensors of the mobile device is provided. The environmental sensor data includes information about the environment of the mobile device in order to obtain a current data set. The device information and the environmental sensor data can be the information or data already explained in more detail above, for example from an inertial measurement unit and radodometry, as well as information obtained from a lidar sensor, a radar sensor or a camera.

[0043] Environmental sensor data may not all be collected at the same time. This is the case, for example, with a lidar sensor, which typically scans the environment by rotating or panning, which takes a certain amount of time. Meanwhile, the mobile device continues to move. In this case, depending on the desired accuracy, a correction to the position and, if applicable, orientation of the mobile device can be made, or other preprocessing can be performed. This will be explained in more detail later.

[0044] While a localization process typically requires one data set of environmental sensor data, e.g. a point cloud from a lidar sensor from a scan of the lidar sensor, there are often multiple data sets of device information or device sensor data, since the sampling frequency of an inertial measurement unit or radodometry is usually higher than that of a lidar sensor. Based on the device information, a preliminary position and orientation (i.e. a preliminary pose, also referred to as the initial pose) of the mobile device is then determined. For this purpose, the device information can be used, for example, to determine the position and orientation of the mobile device relative to the previous localization process. If there are multiple data sets of device information for a localization process, the last one received or an average of these can be used.

[0045] A first check (or a first check process) then follows. This includes a first comparison of whether the current data set with the provisional position and orientation can be matched according to at least one first matching criterion with one or more previously generated data sets or an existing map of the surrounding area, preferably a data set provided in the previous (i.e., in particular, the immediately preceding) localization process. The first check process then produces a check result; this can be positive if the data sets can be matched, otherwise, for example, negative.

[0046] For example, such a comparison uses an algorithm that takes into account a cost function that compares the two data sets (measurements) to be compared depending on the relative position between the measurements. An optimization algorithm can be used to determine the pose z that minimizes the cost function. The cost function can be, for example, a negative log-likelihood function if a probabilistic model is used or another function that represents a measure of the quality of the alignment between two data sets. Certain tolerances can be taken into account as to how precisely the two data sets must match in order to determine the comparison as positive. This can be taken into account during the initial check using the aforementioned at least one first matching criterion.

[0047] During the first comparison, an optimum for the pose is found, starting from the preliminary or initial pose, i.e. an adapted pose (or generally position and orientation), especially if the first comparison or the first check is positive.

[0048] A second check (or a second check process) then follows. This includes a second comparison to determine whether the current data set can be matched, according to at least one second matching criterion, with one or more previously generated data sets or the existing environmental map, preferably the data set provided in the previous localization process. Both the first and second comparisons can be a match or scan match, as already mentioned.

[0049] During the second comparison, one or more pieces of device information (in particular from the current localization process, but possibly also from a previous localization process) are taken into account, each with a weighting dependent on the test result. This can be done, for example, as a term (e.g. a penalty term) in the cost function (in the aforementioned optimization as part of the comparison), which can be selected depending on the test result. Alternatively or additionally, depending on the test result, the current data set with the provisional position and orientation or with a position and orientation obtained during the first comparison can be used in the second comparison. In particular, if the first comparison is positive, the position and orientation obtained during the first comparison can be used, as this is then particularly accurate.However, if the first comparison was negative, it is useful to use the preliminary position and orientation, since, for example, no better pose was found in the first comparison.

[0050] The first comparison and the second comparison can be carried out in the same way, but if necessary with a different weighting, e.g. as a term of the cost function, or via the pose from which the comparison is carried out. In this way, the current reliability of the device information can always be taken into account as precisely and as well as possible, e.g. more or less. In the first comparison, for example, a standard weighting for the device information can be used, from which the weighting in the second comparison may then deviate. If the second comparison is positive, the data set with the position and orientation obtained in the second comparison is made available for use for the localization and / or navigation of the mobile device. Due to the weighting in the second comparison, the position and orientation, e.g.obtained during the optimization in the second adjustment may be different from that obtained in the first adjustment.

[0051] The two comparisons now make it possible to determine only the parameters for the second comparison in the first comparison, which then becomes the actual comparison (or scan match), and can then be used as before. This allows for more efficient and precise localization.

[0052] In one embodiment, the test result comprises a first test result if the first comparison is negative. In this case, a first weighting is used for the first test result, according to which, in particular, at least some of the device information is given a higher weighting than in the first comparison. This allows a deviation from the position and orientation derived from the device information to be kept as small as possible, and a deviation is thus penalized more severely. This is advantageous, for example, if the environmental sensor data is less accurate; this is indicated by a negative first comparison. The device information can then be trusted more.

[0053] In one embodiment, the test result comprises a second test result if the first comparison is positive and if the first comparison is also underdetermined. An underdetermined comparison is understood in particular to mean that the environment (from which the environmental sensor data, i.e. the data set, originate) is featureless or has few features. For example, it may be a long corridor. In such a case, it may be that not all degrees of freedom can be reliably estimated. In the case of a lidar sensor and a corridor, for example, it is easy to determine the distance to the walls of the corridor and the orientation, but it may be very difficult or impossible to estimate the movement along the direction of the corridor. To detect such cases, the eigenvalues ​​of the submatrix can be considered, for example, which relate to the position of the Hessian matrix of the optimization result from the first comparison.It can be determined whether the uncertainty is much larger in one direction than in the other by comparing the ratio of the two eigenvalues ​​to a threshold. This method can be used to identify cases of underdetermination in general, not just for the specific corridor case. For example, this approach can also detect a case where the mobile device is close to a wall on one side and there is a large empty space on the other. Alternatively, a detector or algorithm can be used that explicitly detects the shape of the corridor in the sensor data.

[0054] In this case, a second weighting is used for the second test result, according to which at least some of the device information is given a higher weighting than in the first comparison. Due to the insufficient accuracy of the environmental sensor data, the device information can be relied upon more. The weighting can also be the same for the first and second test results, for example.

[0055] In one embodiment, the test result comprises a third test result if the first comparison is positive, if the first comparison is not under-determined, and if the device sensor data have been determined to be unreliable. The fact that the device sensor data (or possibly the device information in general) have been determined to be unreliable is to be understood in particular to mean that the sensors in question, for example, provide incorrect values ​​or should not be used for other reasons. For example, due to wheel slip, the value provided by the sensor in question (e.g. wheel odometry) can be regarded as unreliable, and a fallback to constant speed or position or another model can be used as already operated above. For this purpose, the pose determined from the device sensor data (provisional pose orPosition and orientation) can be compared with the pose determined during the initial alignment. This can then be checked to see whether the two poses differ by more than a predefined threshold or whether they agree within specified tolerances. If a corridor was previously detected, for example, it can be assumed that this is due to the problem of insufficient alignment, and it can also be assumed that the device information is more likely to be correct.

[0056] In the third test result, a third weighting is used as a weighting, according to which in particular at least part of the device information is given a lower weighting than in the first comparison.

[0057] Otherwise, it can be assumed that a good alignment has taken place and thus proprioception is more likely the cause of the problem.

[0058] In one embodiment, the test result includes a fourth test result if the first comparison is positive, if the first comparison is not underdetermined, and if the device sensor data has not been determined to be unreliable. For the fourth test result, a weighting used in the first comparison, e.g., a standard weighting, is then used as the weighting. As a standard weighting or for the first comparison, it can be provided, for example, that the position and orientation are given little consideration and, for example, that the preliminary position and orientation are also used (then also in the second comparison).

[0059] In one embodiment, it is also provided that the data set provided in the previous localization process (with the relevant position and orientation) is added to the existing environment map if at least one update criterion is present, which in particular includes a detected error in the (current) localization process. Unlike the already mentioned usual criteria, according to which a data set is added to the environment map, in the event of an error, the data set obtained from it, which is correct, is added to achieve a final update.

[0060] The at least one update criterion preferably comprises at least one of the following update criteria: If the second comparison is negative, in particular if the second comparison in the previous localization process was also positive. If the device sensor data has been determined to be unreliable, in particular if the device sensor data had not been determined to be unreliable in the previous localization process. If the first comparison is under-determined, in particular if the first comparison in the previous localization process was not under-determined. These are therefore cases in which, as explained above, different weightings can be used. It can also be included if the mobile device was placed on a surface during the previous localization process (and then lifted even earlier). This can be detected, for example, using the aforementioned lift-up sensor.In this case, it can be assumed that the device sensor data will be incorrect. It may also be the case that the environmental sensor data was not received or provided, at least temporarily, during the localization process. Here, too, the data is assumed to be unreliable. Furthermore, the previously mentioned update criteria, such as the elapse of a certain period of time, a certain distance traveled, or the number of scans performed (data sets received), can also be used.

[0061] In one embodiment, adding the data set provided in the previous localization process to the existing environment map comprises checking whether a difference between a position and orientation of the mobile device determined based on the device information in the data set being added and the position and orientation of the mobile device in a data set most recently added to the environment map lies within a predetermined tolerance. This can also be referred to as a so-called refinement match. In this way, it can be checked whether there is a drift in the position and orientation from the device sensor data and whether this drift is too small in the incremental localizations to be considered an error. If this difference lies within the tolerance, the data set can be added; otherwise, for example, not.In one embodiment, navigation information for the mobile device is further determined based on the provided data set with the relevant position and orientation of the mobile device.

[0062] One embodiment of the invention relates to a method for error handling during navigation of a mobile device, wherein the mobile device is moving or is intended to move in an environment using SLAM-based localization. Localization processes are performed repeatedly.

[0063] In a localization process, device and environmental information is provided, which has been at least partially acquired from the device and / or the environment using sensors of the mobile device. This can be the aforementioned device sensor data and environmental sensor data, but possibly also data obtained based on models. A dataset is then generated based on the device and environmental information. The dataset is provided for use in localizing and / or navigating the mobile device. In particular, all further processing operations can be included in a localization process, as explained above, for example.

[0064] However, during the localization process, a check is made to determine whether an error (or problem) exists. A record that encountered an error during the localization process is then identified (or, for example, marked) as potentially erroneous, and a record that encountered no error during the localization process is identified as a valid record.

[0065] If a valid data set is received after a potentially erroneous data set, the potentially erroneous data set is checked to determine whether it can be reused. This way, the decision on how to handle the error is postponed to the very end, when the actual processing of the data set in the individual (incremental) localization process has already been completed. If, for example, it is determined that the error was not serious, the data can still be used.

[0066] In one embodiment, checking the potentially erroneous data set comprises checking whether the correct data set can be matched (i.e., matches) with one or more correct data sets obtained prior to the potentially erroneous data set or with an existing environmental map, according to at least one first matching criterion, in order to obtain a first check result. This can be done, for example, as part of a comparison or scan match, as already explained above—with the difference that one of the data sets to be compared here is considered potentially erroneous.

[0067] In one embodiment, if the first check result is negative, ie, if the records cannot be matched, the potentially erroneous record is determined to be the erroneous record. In this case, the record is no longer used.

[0068] In one embodiment, if the first test result is positive and if the potentially erroneous data set matches the data set obtained immediately before the potentially erroneous data set according to at least a second matching criterion, the potentially erroneous data set is determined to be the correct data set. This means that the comparison performed during the incremental localization was positive, meaning the error was at most minor or had little to no impact. The data set can therefore continue to be used; no further measures are required.

[0069] Preferably, if the potentially erroneous data record does not match the data record obtained immediately before the potentially erroneous data record according to the at least one second matching criterion, the potentially erroneous data record is determined to be an erroneous data record. In this case, the error was so large that the comparison during the incremental localization was unsuccessful, even if the comparison between correct data records was successful, so that the data should then no longer be used.

[0070] One embodiment further provides for generating a new environment map based on the correct data set obtained after the data set determined to be faulty. In particular, the faulty data set is not used in this case. This new environment map can then, for example, be merged with the existing environment map. In the event that the initial test result was positive, but the data set ultimately contained errors, these environment maps can also be merged immediately, since a sufficiently accurate comparison was available.

[0071] In one embodiment, it is further provided that correct data records are added to the existing environmental map if at least one update criterion is present. This can be one of the following: The data record received after the correct data record was determined to be faulty. Since the last addition of a correct data record to the existing environmental map, the mobile device has covered an assigned distance. A predetermined period of time has passed since the last addition of a correct data record to the existing environmental map. A predetermined number of correct data records have been received since the last addition of a correct data record to the existing environmental map. In this way, in particular with the first-mentioned criterion, it is achieved that a still correct data record is used to update the map before a new map is generated if necessary.

[0072] In one embodiment, it is determined that an error has occurred during a localization process if at least one of the following error criteria is met: Sensor data from a sensor of the mobile device is determined to be faulty, e.g. because the sensor is delivering data that differs from other sensors or is no longer delivering any data at all. An error can also be assumed if, based on the sensor data, it is determined that the mobile device has been lifted. This can be done, for example, based on the aforementioned lift-up sensor or using the inertial measurement unit. An error can also be assumed if, based on the sensor data, it is determined that the mobile device has been moved by an external force, in particular horizontally. This can be determined, for example, using the inertial measurement unit, and if necessary also using wheel odometry.Likewise, an error can be detected if the difference between the position and / or orientation of the mobile device determined based on the device sensor data in the correct data set being added and a correct data set recently added to the environment map is not within a specified tolerance. This means that the aforementioned refinement match was unsuccessful.

[0073] In one embodiment, navigation information for the mobile device is further determined based on the provided data sets.

[0074] A system for data processing according to the invention or a computing unit, e.g. a control device or a control unit of a mobile device, or a server or other computer, is set up, in particular in terms of programming, to carry out a method according to the invention, e.g. in one of the described embodiments.

[0075] The invention also relates to a mobile device having such a data processing system or configured to receive navigation information determined as described above. The mobile device preferably also has a drive system and a control or regulating unit for moving the mobile device according to the navigation information.

[0076] The mobile device is preferably also configured to carry out processing; in particular, the mobile device may be one as described above, e.g. a cleaning robot or a lawnmower robot.

[0077] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0078] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0079] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

[0080] Short description of the drawings

[0081] Figure 1 shows schematically a mobile device in an environment for explaining the invention.

[0082] Figure 2 shows schematically a sequence of a method for explaining the invention.

[0083] Figure 3 shows schematically a further sequence of a method for explaining the invention.

[0084] Figure 4 schematically shows another sequence of a method for explaining the invention. Figure 5 schematically shows another sequence of a method for explaining the invention.

[0085] Embodiment(s) of the invention

[0086] Figure 1 schematically and by way of example shows a mobile device 100 in an environment 120, in particular a work area, to explain the invention. The mobile device 100 is, for example, a robot vacuum cleaner with a control or regulating unit 102 and a drive unit 104 (with wheels) for moving the robot vacuum cleaner 100, e.g., along a movement path 130. Furthermore, the robot vacuum cleaner 100 has, for example, a sensor 106 designed as a lidar sensor with a detection range. For better illustration, the detection range is selected to be relatively small here; in practice, however, the detection range can also be up to 360° (e.g., but at least at least 180° or at least 270°). The environment 120 can be detected by means of the sensor 106, i.e., environmental sensor data or general environmental information can be generated.

[0087] Furthermore, the robot vacuum cleaner 100 has, for example, a sensor 107 designed as an inertial measurement unit and a sensor 108 designed as a wheel odometry unit. The wheel odometry unit can be distributed, in particular, between the two wheels of the drive unit 104, although this is not shown here for the sake of simplicity. Using the sensors 107 and 108, information about a position and / or movement of the robot vacuum cleaner 100 can be acquired, i.e., device sensor data or general device information can be generated.

[0088] Furthermore, the robot vacuum cleaner 100 has a computing unit 103, e.g., a control unit, by means of which data can be exchanged with a higher-level system 110, e.g., via an indicated radio connection. In the system 110, for example, movement paths (or general navigation information) can be determined, which are then transmitted to the system 103 in the robot vacuum cleaner 100, which the latter is then to follow. However, it can also be provided that a movement path (or general navigation information) is determined in the system 103 itself or is received there in some other way. Instead of a movement path or the navigation information, the system 103 can, for example, also receive control information that has been determined based on a movement path or the navigation information, and according to which the control or regulating unit 102 can move the robot vacuum cleaner 100 via the drive unit 104 to follow a movement path.The movement path 130 is only indicated here as an example.

[0089] The vacuum robot 100 is intended, for example, to move or navigate independently within the environment 120 and, for example, to clean a floor. Furthermore, several different objects or obstacles are depicted in the environment as examples, namely a wall 140 and a cabinet 142.

[0090] Although the invention is explained here and below using the example of a robot vacuum cleaner, this also applies to other mobile devices such as robotic lawnmowers or other self-driving vehicles. Instead of the lidar sensor, a camera or a radar sensor, for example, can also be provided. Furthermore, the robot vacuum cleaner or the mobile device in general can also have other sensors, such as the aforementioned lift-up sensor.

[0091] Figure 2 schematically illustrates the flow of a method for explaining various embodiments. The method generally serves to navigate a mobile device, such as the robot vacuum cleaner shown in Figure 1, which may also include determining the position and orientation of the mobile device and its localization. Error handling may also be performed. The method described below encompasses various individual aspects and embodiments, which can also be used individually.

[0092] In a step 200, device and environmental information is first provided. This can include, in particular, environmental sensor data 201 from, for example, the lidar sensor, as well as device sensor data 202 from the inertial measurement unit and device sensor data 203 from the radodometry. This can further include device information 204, such as a target speed of the robot vacuum cleaner. Likewise, data or information 205 from, for example, the lift-up sensor can be provided (possibly also as part of the device information). In general, all relevant information and sensor data can be provided in step 200.

[0093] This shows that device information comes from different sources, in the example shown, namely from the inertial measurement unit, from the radodometry and a target speed.

[0094] When providing this information, and if necessary collecting sensor data and / or other information, it can also be checked whether an error F is present, e.g. a sensor is not providing any data.

[0095] Based on this device information or the corresponding data, the position and orientation of the mobile device are determined in a step 212 for localizing the mobile device. The position and orientation 213 (hereinafter also referred to as pose) can be determined, for example, relative to a last determined pose.

[0096] For example, several sub-aspects of the pose are each assigned a set of sources from the sources of various types. For example, a sub-aspect 213.1 comprises a position in the horizontal direction of movement, to which the set 214.1 of sources is assigned with the following predefined hierarchy: the radometry, the target velocity, a constant velocity model, and a constant position model.

[0097] For example, a sub-aspect 213.2 includes a yaw angle to which the set 214.2 of sources is assigned with the following, predefined hierarchy: the inertial measurement unit, the wheel odometry, the desired speed, the constant speed model, and the constant position model.

[0098] For example, a sub-aspect 213.3 includes a pitch and / or roll angle to which the set 214.3 of sources is assigned with the following, predefined hierarchy: the inertial measurement unit, a constant speed, and a constant position.

[0099] However, before step 212, a check is performed in step 210 to determine whether the various types of sources are available for the respective set of sources. A source is unavailable, for example, if no data is or has been received from the source for at least a predetermined period of time, and / or if an error is detected for which the source is specified to be unavailable. The latter may include, for example, the lift-up sensor indicating that the mobile device is raised, which means that the wheel odometry can be considered unavailable.

[0100] Likewise, for example, the case may arise where the corresponding data obtained from the inertial measurement unit by the wheel odometry differ from each other by more than a predetermined value. In this case, it can be determined that the wheel odometry or the source comprising the wheel odometry is unavailable. Generally, in a step 211, it can be determined for each source whether the source is available or not.

[0101] In steps 210, 211, and 212, it can also be checked whether an error F is present, for example, if a sensor is not providing data. The case where a sensor or another source is determined to be unavailable can also be considered an error.

[0102] In step 212, the sub-aspects are determined individually, each based on the source that has the highest priority in the respective set, whereby a position and orientation (pose) are then obtained. It should be noted that six different degrees of freedom do not necessarily have to be determined here; for example, depending on the situation, a vertical position may not be necessary. Based on the position and orientation of the mobile device, navigation information for the mobile device can then be determined in the further course. It should be mentioned that, for example, not only position and orientation can be treated separately, but a distinction can also be made between the yaw angle (which can be determined using radodometry and target speed, but can usually only be measured relative to the starting position) and the roll / pitch angles (which, for example,can only be measured with the inertial measurement unit, but are measured in an absolute coordinate system with respect to the gravity vector).

[0103] As a fallback solution, as mentioned, the constant velocity model or the constant position model can be used, which assumes that the mobile device is moving at a constant speed and rotating at a constant angular rate, or is not moving and not rotating at all. The constant speed may not be available for the first journal, for example, because no speed can yet be estimated. Furthermore, the output of the constant velocity model should be compared with the maximum speed achievable by the respective mobile device and discarded if this speed is significantly exceeded, as this would indicate an incorrect speed estimate. The constant position model, on the other hand, is always available as a last resort.

[0104] To calculate the pose or trajectory (pose relative to the last pose), the start and end timestamps of the time interval of interest, as well as all timestamps within the interval associated with a sensor measurement, can be considered. It is not necessary to assume that all sensors have the same sampling rate or that their measurements are performed synchronously. Therefore, the data from all sensors for all timestamps can be interpolated based on neighboring measurements, e.g., using linear or spline-based interpolation. In this way, a trajectory with high temporal resolution can be obtained.

[0105] If a lidar sensor is used to collect the environmental sensor data, this environmental sensor data (or the lidar scan) can first be preprocessed. One of the problems with rotating lidar sensors can be that the points of a lidar scan are not all acquired at the same time, leading to scan distortion. Therefore, the trajectory from the previous step can be used to calculate the position of all scan points as if they were acquired at the same time, e.g., at the beginning or end of the scan interval (dewarped).

[0106] If the supplied trajectory cannot be used for the rectified calculation (e.g. because a fallback to a constant position was performed), a modified scan matcher can be used instead, which performs rectification during the scan matching.

[0107] In addition to the dewarped representation step, dynamic objects can also be detected in the measurement or processing of the environmental sensor data at this stage. If a lidar sensor is used, a change detection algorithm can be employed. To distinguish between static and dynamic objects, the current measurement can be compared with previous measurements or a local map (the environmental map). By removing dynamic objects such as animals, people, moving doors, etc., their impact on the tracking algorithm can be eliminated or at least reduced. If the pose from the previous step is not sufficiently accurate for this due to previously identified problems, detection of dynamic objects can be omitted to avoid incorrectly removing static parts of the measurement.

[0108] When using a two-dimensional environmental sensor such as a 2D lidar, it can also be beneficial to perform leveling. During leveling, the roll / pitch angle of the mobile device can be used to rotate the measurement (e.g., the point cloud) so that it is parallel to a global, gravity-aligned coordinate system. This enables the use of 2D matching algorithms even when the mobile device is tilted, e.g., due to uneven ground. Furthermore, it can also be beneficial to filter the observed points by their elevation. In 2D SLAM systems, observing the floor or ceiling can be problematic, as this leads to unstable measurements that move depending on the angle of the sensor and therefore should not be used in the matching algorithm. To avoid such problems, points that are on or below the floor can be filtered based on their elevation after leveling.It's possible to be even more restrictive and only allow points within a vertical range of a few centimeters around the sensor's mounting height. This avoids accidentally detecting structures at a different height that are not normally visible to the sensor.

[0109] Subsequently, based on the determined position and orientation 213 and the environmental sensor data 201, a data set is to be determined that is used for further localization and / or navigation of the mobile device, in particular, if necessary, also for expanding an environmental map. For this purpose, in a step 220, the position and orientation 213 are to be provided as a preliminary position and orientation 213', and the environmental sensor data 201 are to be provided as a current data set 20T.

[0110] In a step 221, a first check is performed, comprising a first comparison of whether the current data set 201' with the preliminary position and orientation 213' (i.e., if the environmental sensor data are linked to the preliminary pose) can be matched according to at least one first matching criterion with one or more previously generated data sets or an existing environmental map, e.g., a data set provided in the previous localization process. A localization process is understood to be a process in which a data set with pose is determined, which is then used for further processing, e.g., also added to an environmental map. A test result 225 is obtained.

[0111] The first check or comparison, for example, is an optimization process in which, starting from the current data set with the preliminary pose, an attempt is made to match the current data set with the aforementioned previous data set. This can, for example, result in an adjusted pose 213. This is a so-called scan match.

[0112] Subsequently, in a step 222, a second check is carried out, taking into account the test result 225. In this case, a second comparison is made as to whether the current data record 201 ' can be matched according to at least one second matching criterion with one or more previously generated data records or the existing environmental map, preferably the data record provided in the previous localization process.

[0113] During the second comparison, however, one or more pieces of device information are now taken into account, each with a weighting 226 dependent on the test result 225. Alternatively or additionally, depending on the test result 225, the current data set 20T with the preliminary position and orientation (pose) 213' or with the position and orientation 213" obtained during the first comparison is used. For the purposes of the example, four possible test results 225.1, 225.2, 225.3, 225.4 will be considered, which will be explained in more detail below.

[0114] The second check or second comparison is also an optimization process in which, starting from the current data set with the preliminary pose or the pose obtained in the first comparison, an attempt is made to match the current data set with the aforementioned previous data set. This can then, for example, result in a further adjusted pose 213'. This is also a so-called scan match, in particular a second scan match.

[0115] If the second comparison is positive, in a step 230 the current data set 20T with the pose 213' (obtained in the second comparison) is provided for use for localizing and / or navigating the mobile device.

[0116] An error F can also occur during these two check or comparison processes (here, for example, indicated at step 200). In a step 340, for example, navigation information for the mobile device can be determined in order to be able to navigate or move the mobile device.

[0117] To track the movement of the mobile device, the aforementioned scan matches can be used. A matching algorithm, such as Iterative Closest Point (ICP), Normal Distribution Transform (NDT), Correlative Scan Matching, or a visual feature matching algorithm, can be used. These matching algorithms work by considering a cost function that compares the two measurements (data sets) to be matched depending on the relative position between the measurements. Depending on the application, these measurements can be point clouds (in the case of a lidar sensor), camera images, radar measurements, etc.

[0118] For example, an optimization algorithm is used to determine the position that minimizes the cost function. The cost function can be a negative log-likelihood function if a probabilistic model is used, or any other function that represents a measure of the quality of the alignment between two measurements. Typically, these optimization algorithms start with an initial estimate, which is then refined over time, e.g., using a Newton iteration. Since the cost function cannot be convex, the resulting pose after convergence depends on a good choice of the initial pose. For this purpose, the pose determined based on the device sensor data (preliminary pose) could in principle be used to increase the probability that the algorithm converges to the global optimum rather than to a local optimum that is far from the true value.

[0119] To introduce prior knowledge about the relative transformation (i.e., between two poses), it is possible to extend the cost function by adding a term that describes the deviation from the previous pose. Introducing an additional term changes the shape of the cost function, and its minimum is shifted toward the previous position, since deviations from the previous position are penalized with higher costs, even if the match itself is better. In this way, weights can be taken into account during a scan match, for example, to allow more or less weight to be given to device sensor data.

[0120] The main problem with such an approach is the choice of weighting for the term for the device sensor data or the pose determined with it. If the weighting is too large, the result depends only on the pose and ignores information from the measurements (environmental sensor data, e.g., from the lidar sensor); if the weighting is too small, the pose is ignored. It has now been shown that a good weighting for the pose depends on the specific circumstances and can be adjusted accordingly. If the pose comes from very reliable sensors that are not currently affected by problems, their weighting should be chosen large. If, on the other hand, the pose comes from inaccurate sensors, or these sensors are currently affected by problems, the weighting should be chosen significantly lower.

[0121] To determine a good choice of parameters for the matching algorithm, it is proposed to perform two scan matches. The first match serves to gain insight into the current situation, and its results are used to adjust the parameters of the second match. Subsequently, the second match is performed with the adjusted parameters to obtain the actual result. A particularly relevant case that can be detected based on the first match is the case of a featureless environment, such as a corridor, where the degree of freedom is underdetermined, i.e., not all degrees of freedom can be reliably estimated. For example, in the case of a lidar sensor and a corridor, it is easy to estimate the distance to the walls of the corridor and the orientation, but it is very difficult or impossible to estimate movement along the direction of the corridor.

[0122] To detect such cases, one can, for example, consider the eigenvalues ​​of the sub-matrix related to the position of the Hessian matrix of the optimization result from the first alignment. It is determined whether the uncertainty is much larger in one direction than in the other by comparing the ratio of the two eigenvalues ​​with a threshold. This method not only detects specific corridor cases, but also cases where other under-determination exists. This approach also detects, for example, the case where the mobile device is close to a wall on one side and there is a large empty space on the other. Alternatively, a detector that explicitly detects the shape of the corridor in the sensor data could be considered.

[0123] Another problem that can be identified from the first alignment is that the pose determination, i.e. the device sensor data, is unreliable, e.g., due to wheel slip that was not detected by the wheel slip yaw detector (e.g., via wheel odometry), a reversion to constant speed / position, or other reasons. For this purpose, the pose determined based on the device sensor data is compared with the result of the first alignment, and it is checked whether the two poses deviate from each other by more than a predefined threshold. If a corridor was previously detected, it can be assumed that this is due to the problem of insufficient alignment, and it can be assumed that the pose is more likely to be correct. Otherwise, it can be assumed that a good alignment has taken place and thus the device sensor data is more likely the cause of the problem.In the latter case, the device sensor data may be considered unreliable.

[0124] This will be explained in more detail below with reference to Figure 3, i.e., when which test result can be used. For this purpose, step 220 from Figure 2 is first repeated, in which the current data set 201' with the preliminary pose 213' is provided.

[0125] This is followed by the first check 221 or the first comparison. A standard weight can be used here, for example. In step 301, a check is made to determine whether the first comparison is positive. The test result should, for example, include the first test result 225.1 if the first comparison is negative (N). The first weighting is then used as the weighting, after which, for example, at least some of the device information is weighted higher than in the first comparison.

[0126] If the first comparison is positive (Y), a check can be carried out in step 302, for example, to determine whether the first comparison was under-determined, i.e., whether a corridor was detected. If this is the case (Y), the test result should include, for example, the second test result 225.2. The second weighting is then used as the weighting, according to which, for example, at least some of the device information is weighted more than in the first comparison. For example, the first and second weightings can also be identical.

[0127] If it is determined in step 302 that the first comparison is not underdetermined (N), a check can be carried out in step 303 to determine whether the device sensor data has been determined as unreliable. If so (Y), the test result can include, for example, the third test result 225.3. The third weighting is then used as the weighting, according to which, for example, at least some of the device information is given a lower weighting than in the first comparison. The device sensor data is determined as unreliable, for example, if the preliminary position and orientation do not match a position and orientation obtained during the first comparison within specified tolerances.

[0128] If this is not the case according to step 303 (N), the test result should include, for example, the fourth test result 225.4. The weighting can be the same as the one used in the first comparison, i.e., the default weighting.

[0129] This is followed by the second check, step 222, with the second comparison, using the first, second, third or fourth weighting depending on the situation.

[0130] The data set 201 provided in step 230 with the pose 213'" can then be added to an existing environment map 235 (step 232); in this case, the data set 201 with the pose 213'" can also be referred to as a so-called keyframe. This can only occur, for example, if an update criterion 231 is present. The update criterion can include, for example, that no data set has been added to the environment map for a certain period of time, or that the mobile device has moved a certain distance since the last addition.

[0131] If an error F occurred in the previously described process, the resulting data set may be faulty. In this case, this can be used as update criterion 231 to add the data set obtained in the previous process (localization process), in which, in particular, no error occurred, to the environment map 235.

[0132] Other update criteria for adding the data set obtained in the previous process (localization process) to the environment map 235 may also be that the second comparison is negative, in particular if the second comparison in the previous localization process was also positive; that the device sensor data has been determined to be unreliable, in particular if the device sensor data in the previous localization process had not been determined to be unreliable; that the first comparison is under-determined, in particular if the first comparison in the previous localization process was not under-determined; that the mobile device was placed on a surface during the previous localization process; or that the environment sensor data was not received or provided, at least temporarily, during the localization process.

[0133] Adding the data set provided in the previous localization process to the existing environmental map—or generally adding a data set to the environmental map—can, in particular, involve a special check. This check determines whether a difference between a position and orientation (pose) of the mobile device determined based on the device information in the added data set and the position and orientation (pose) of the mobile device in a data set most recently added to the environmental map lies within a specified tolerance. This is illustrated in Figure 4. For this purpose, several localization processes n to n+4 are shown as examples, each of which is based on environmental sensor data and the data sets obtained thereby. Furthermore, a pose can be determined based on the associated device information or device sensor data, as explained above. This typically always occurs with reference to the most recently determined pose.For example, in localization process n, a pose may have been determined according to the above-mentioned procedure with two scan matches. For localization process n+1, a pose relative to the pose of localization process n is then determined based on the current device information. This is indicated by arrows 401 (dashed line). With the pose thus determined, localization process n+1 can be performed with, for example, the two scan matches. However, the scan match itself, as mentioned above, is performed with the current data set relative to the data set last added to the environment map. This is indicated by arrows 402 (solid line).

[0134] For example, if the data set of localization process n is the last added data set, the scan match is also performed for localization processes n+1, n+2, n+3 relative to the data set of localization process n, but the pose is always determined relative to the previous localization process, e.g. for localization process n+3 relative to localization process n+2.

[0135] In this way, small deviations in the pose can add up. To this end, it is now proposed that when a data set is to be added to the environment map, the pose is also determined relative to the pose of the most recently added data set. If, for example, the data set from localization process n+4 is to be added, the pose is also determined based on the device information for localization process n+4 relative to localization process n. If the deviation is too large here, this can be regarded as an error, so that the current data set from localization process n+4 is not used, for example. As already mentioned, Figure 2 describes an example of a localization process which in particular comprises the provision of the necessary data in step 200 up to the provision of a data set 20T in step 230, which is added to the environment map if necessary.As also described, it is possible to check at various steps during the localization process whether an error F is present. This can be done not only for a single localization process, but for each localization process.

[0136] This is shown again in Figure 5 in a highly simplified manner for a localization process 506. It is schematically shown that there are several localization processes, which are designated 501, 502, 503 for example.

[0137] In step 510, a check is made to determine whether an error F occurs. If an error F occurs, the data record whose localization process contained an error is identified as a potentially faulty data record 513 in step 512. For example, this may be the case for the data record during localization process 502. A data record whose localization process did not contain an error is identified as a correct data record 514 in step 514. For example, this may be the case for the data record during localization processes 501 and 503.

[0138] Various error criteria can be provided, the presence of which assumes or determines that an error has occurred during a localization process. For example, sensor data from a sensor on the mobile device can be determined to be faulty. Based on the sensor data, it can be determined that the mobile device has been lifted (cf. the lift-up sensor already mentioned). Based on the sensor data, it can be determined that the mobile device has been moved by an external force, in particular horizontally. The difference between the position and / or orientation of the mobile device determined based on the device sensor data in the correct data set being added and a correct data set last added to the environment map may not be within a specified tolerance (as explained in more detail with reference to Figure 4).If a valid data record is received after a potentially erroneous data record—in the aforementioned case, during localization process 503 or at its end—the potentially erroneous data record is checked in step 520 to determine whether it will be reused. In this way, it is noted that an error is occurring or has occurred, but further processing of the error is postponed to the end.

[0139] Checking the potentially erroneous data set in step 520 includes, for example, checking whether the correct data set 515 (i.e., the one from the current localization process 503) can be matched, according to at least a first matching criterion, with one or more correct data sets obtained before the potentially erroneous data set or with an existing environmental map in order to obtain a first check result. Preferably, the check is performed against the data set from the localization process 501 that preceded the localization process 502 with the potentially erroneous data set. Thus, a scan match, for example, can be performed here.

[0140] A first test result 521 is obtained, which indicates whether the two data sets can be matched. If the first test result 521 is negative (N), the potentially erroneous data set 513 can be determined as the (finally) erroneous data set, step 522.

[0141] If the first test result 521 is positive (Y), it can be checked whether the potentially erroneous data record matches the data record obtained immediately before the potentially erroneous data record according to at least one second matching criterion. If so (Y), the potentially erroneous data record can be determined as a correct data record, step 524. If, however, this is not the case (N), the potentially erroneous data record can be determined as an erroneous data record, step 525.

[0142] In the case of step 524, the dataset can be used as usual and, for example, added to the existing environment map as described above (see Figure 2, step 232), here designated step 530. In other words, the comparison was successful, and there was at most a small error in the pose estimation. This means that ultimately no (significant) error occurred and error prevention during incremental localization was successful. It can be assumed that the pose estimates and the data associated with the potentially erroneous dataset are valid.

[0143] For the cases according to step 522 or 525, however, it can be provided that the then erroneous data set is not reused; instead, a new environment map can be created. This new environment map can then, for example, be merged with the existing environment map. For the case according to step 525, i.e. that the first test result was positive, it can be assumed that the error was relatively small, so that these environment maps can also be merged immediately, since a sufficiently accurate comparison was available. The comparison was therefore also successful here, but with a larger error in the pose estimation: An error occurred during the incremental localization, but the comparison for error detection was able to compare the last data set with the environment map. This means that the error in the incremental localization most likely caused a relatively small error.

[0144] In the case of step 522, i.e., if the first test result was negative, these environment maps can be merged later, for example, when more information is available to check whether or how these environment maps can also be merged. Therefore, the comparison failed in this case. An error occurred during incremental localization, and the error detection comparison could not compare the most recent data with the environment map. This happens, for example, if the error is serious or if the mobile device has explored a new part of the environment. The latter means that the map does not contain correct data at the location where the mobile device is located.

[0145] A recovery procedure can then be provided to ensure that the SLAM state returns to a valid state after an error. The following steps can be performed for this purpose. Problematic data records from the previous localization process are deleted, and the last (correct) data record is moved to a new map. This results in two disjoint maps, both of which are consistent. The SLAM system reports the position of the mobile system with respect to this new map. This means that other map data is (temporarily) no longer available, ensuring that other modules of the system behave accordingly. Optionally, it can be provided that if the error detection comparison was successful, the two maps are immediately merged (see step 525), using the pose estimate from the error detection comparison.The advantage of an explicit merge is that it can be undone if it later turns out to be incorrect. A new environment map is created because the exact trajectory from the last correct record to the current record is uncertain. After the recovery procedure, the two maps can be merged again.

[0146] As already mentioned above, a correct data set does not always have to be added to the existing environmental map; rather, this can also only occur if at least one update criterion is met. This can include, for example, that the data set received after the correct data set was determined to be faulty. Then, the last data set received as correct is added. This can also include that since the last addition of a correct data set to the existing environmental map, the mobile device has covered an assigned distance, or a predetermined period of time has passed during which a predetermined number of correct data sets were received.

[0147] The proposed approach thus enables several innovations. For example, a holistic approach is possible, meaning that many different types of problems or errors can be addressed that may arise at different stages of the overall algorithm and involve different sensor settings (in contrast to specialized approaches that can only address a single problem).

[0148] The problems or errors can be forwarded to later processing stages and then considered there. This can avoid premature and suboptimal decisions, especially in cases where the correct response to a problem only becomes clear later (in contrast to conventional approaches, where a processing stage can only either silently output a potentially incorrect result or stop the process with a fatal error). Nevertheless, it can also be intended, for example, to attempt to continue the regular localization process or navigation despite problems or errors.

[0149] In addition, global map data can be used. The final decision as to whether the incremental localization (localization process) has failed is based on global map data (environmental map). This decision is postponed until non-problematic data (correct data sets) are available for incremental localization, and erroneous data is removed from the map or not included at all. This mechanism allows SLAM to continue in extreme situations and enables higher-risk error avoidance schemes where conventional approaches abort with a fatal error or risk corrupting the environment map.

Claims

Claims 1 . A method for error handling during navigation of a mobile device (100), in particular of an at least partially automated vehicle or robot, in particular a cleaning robot or a lawnmower robot, wherein the mobile device moves or is intended to move in an environment using SLAM-based localization, comprising: repeatedly performing localization processes (501, 502, 503), wherein a localization process each comprises: - Providing (200) device and environmental information which has been at least partially acquired from the device and / or the environment by means of sensors of the mobile device, - Generating a data set (20T) based on the device and environment information, and - Providing (230) the data set (201') for use in locating and / or navigating the mobile device (100); Checking (510) during the localization processes whether an error (F) is present, and determining (512) a data record in whose localization process (502) an error was present as a potentially faulty data record (513), and determining a data record in whose localization process (501, 503) no error was present as a correct data record (514); and if a correct data record is obtained after a potentially faulty data record: checking (520) the potentially faulty data record to determine whether it will be used further or not.

2. The method according to claim 1, wherein checking (520) the potentially erroneous data set comprises: checking whether the correct data set matches one or more of the following criteria according to at least one first matching criterion: several correct data sets obtained before the potentially faulty data set or an existing environmental map can be matched in order to obtain an initial test result.

3. The method of claim 2, wherein if the first test result (521) is negative: determining the potentially erroneous data set as an erroneous data set.

4. The method according to one of claims 2 to 5, wherein the device and environmental information comprises device information, in particular device sensor data that has been acquired by means of sensors of the mobile device, wherein the device information comprises information about a position and / or movement of the mobile device, wherein the device and environmental information comprises environmental sensor data that has been acquired from the environment by means of sensors of the mobile device, further comprising, if the first test result (521) is positive: if the potentially erroneous data set matches the data set obtained immediately before the potentially erroneous data set according to at least one second matching criterion: determining (524) the potentially erroneous data set as a correct data set.

5. The method according to claim 6, further comprising, if the potentially erroneous data set does not match the data set obtained immediately before the potentially erroneous data set according to the at least one second matching criterion: Determining (525) the potentially faulty data record as a faulty data record.

6. The method according to claim 3 or 5, further comprising: Generating (522, 525) a new environment map based on the correct data set obtained after the data set determined to be faulty.

7. The method of claim 6, further comprising: Merging the new environmental map with the existing environmental map, whereby in particular the erroneous data set is not used.

8. Method according to one of the preceding claims, further comprising: Adding proper data records to the existing environmental map if at least one update criterion is present, wherein the at least one update criterion comprises at least one of the following update criteria: the data record received after the proper data record was determined to be faulty, the mobile device has traveled an assigned distance since the last addition of a proper data record to the existing environmental map, a predetermined period of time has passed since the last addition of a proper data record to the existing environmental map, a predetermined number of proper data records have been received since the last addition of a proper data record to the existing environmental map.

9. Method according to one of the preceding claims, wherein it is determined that an error (F) exists during a localization process if at least one of the following error criteria is met: Sensor data of a sensor of the mobile device are determined to be faulty, based on the sensor data it is determined that the mobile device has been lifted, based on the sensor data it is determined that the mobile device has been moved by an external force, in particular horizontally, and with reference to claim 4, if the difference between the position and / or orientation of the mobile device determined based on the device sensor data in the adding proper data set and a correct data set last added to the environment map is not within a specified tolerance.

10. The method according to any one of the preceding claims, further comprising: Determining (240) navigation information for the mobile device based on the provided data sets, in particular the correct data sets. 11 . A data processing system comprising means for carrying out the method according to any one of the preceding claims.

12. A mobile device comprising a system according to claim 11 and / or configured to receive navigation information determined according to a method according to claim 10, and configured to navigate based on the navigation information, preferably with a control or regulating unit and a drive unit for moving the mobile device according to the navigation information.

13. Mobile device according to claim 12, which is designed as an at least partially automated vehicle, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. a cleaning robot, a floor or street cleaning device or a lawnmower robot, and / or as a drone.

14. A computer program comprising instructions which, when executed by a computer, cause the program to carry out the method steps of a method according to any one of claims 1 to 10 when executed on the computer.

15. A computer-readable storage medium on which the computer program according to claim 14 is stored.

Images