Method for determining navigation information for mobile device

Abstract

A method for determining navigation information for a mobile device. The invention relates to a method for determining navigation information for a mobile device, in particular an at least partially automatically moving vehicle or robot, further in particular a mowing robot, comprising: providing an image of an environment in which the mobile device moves or should move; assigning one of a plurality of categories to at least some segments of the image, in particular each of the image points, based on the image and in particular in the scope of semantic segmentation, where the categories (C1, C2) characterize a measure of the dynamics of the objects (140, 142, 144, 146) assignable to the relevant segment; determining navigation information on the basis of the image taking into account the assigned category (C1, C2); and providing navigation information.

Description

Technical Field

[0001] The invention relates to a method for determining navigation information for a mobile device, in particular a vehicle or a robot that moves at least partially autonomously, such as a lawnmower robot, a method for training a machine learning algorithm, a mobile device for executing the method, and a computing unit and a computer program. Background Art

[0002] Mobile devices or processing devices such as mowing robots, vacuum or mopping robots or other household robots typically move in the environment to be processed, such as in a garden or in a house. In this case, the task is especially the navigation of the mobile device, which should move in the environment, for example on a specific path within certain boundaries (working area). Summary of the invention

[0003] According to the invention, a method for determining navigation information for a mobile device, a method for training a machine learning algorithm, a mobile device for carrying out the method, as well as a computing unit and a computer program are proposed with the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The present invention discusses mobile devices that move in particular automatically, such as preferably robots, for example mowing robots. Although the present invention should be explained below mainly with respect to mowing robots, other mobile devices or robots may also be considered. Examples of such mobile devices (or mobile working devices) are generally, for example, robots and / or drones and / or vehicles that move partially or (completely) automatically (on land, in water or in the air). As robots, for example, household robots such as suction and / or wiping robots, ground or road cleaning equipment or mowing robots may be considered, as well as other so-called service robots, as vehicles that move at least partially automatically, such as passenger vehicles or freight vehicles (also so-called land transport vehicles, for example in warehouses) and aircraft, such as so-called drones or ships. Automatic driving or autonomous driving passenger or trucks may also be considered as mobile devices.

[0005] For such a mobile device, navigation information can be determined, based on which the mobile device can be moved. Such navigation information can, for example, include a specific movement path or trajectory along or according to which the mobile device should move. In the case of a lawnmower robot, for example, a movement path can be determined, which allows the lawn (i.e. the working area in the environment) to be driven through and mowed as quickly and efficiently as possible. For this purpose, control information for moving the mobile device can be determined based on the navigation information, based on which the mobile device can be moved. Such a mobile device typically has a drive system and a control or regulation unit for control according to such control information.

[0006] For example, it is usually desirable for a lawnmower robot to stay within the boundaries of a garden or lawn, since there may be areas that are not suitable for optimal operation, such as ponds, mud, gravel roads, flower beds, etc. For this purpose, wires or cables that can be detected or detected by means of sensors in the robot can be laid in the garden or in the lawn, especially at the edge of the lawn or where the lawnmower robot is to mow. In this context, we can also talk about lawn edges, where such wires are laid. Although this allows the movement of the robot to be limited as desired, it requires a high level of effort due to the laying of wires or cables.

[0007] Another possibility for the (mobile) device to be able to recognize specific features in the environment, for example, the edge of a lawn in the case of a lawn mower robot, is to use an environment detection device or one or more sensors, such as a camera or video camera, a thermal imager, a radar sensor, a lidar sensor, a laser rangefinder, an ultrasonic sensor, an inertial sensor and / or an odometer. In particular, within the scope of the present invention, detection devices or sensors that enable an image representation of the environment should be considered, i.e. preferably a camera or video camera. However, it is also possible to obtain depth information, which can be represented in an image or as a (digital) image, for example, by means of other sensors, such as a lidar or ultrasonic sensor. Such an image of the environment can then be analyzed in order to recognize or identify specific things or objects in the environment, such as the lawn edge just mentioned, for example, in the case of a lawn mower robot.

[0008] In particular, in this case, many different objects or object types in the environment can be recognized or identified, which can be taken into account when determining navigation information. Such objects can be, for example, houses, walls, trees, stumps, stones, rocks, roads, lawns, flowers, flower beds, as well as, for example, people, animals, leaves or the sky or clouds in the sky.

[0009] Here, in particular, so-called semantic segmentation can now be used. In this case, each image point (pixel) in the image is assigned one of several classes, i.e. the image point is labeled or classified. For this purpose, firstly features of the image point are typically determined or extracted, based on which the class is then assigned. Such features can be, for example, shape, color, context, pattern, light changes and / or image context. Thus, for example, a picture point can be assigned the color green, and the surrounding picture points can also all be green. This would indicate a lawn.

[0010] Different features can be considered as categories here. In the example of a lawnmower robot, for example, two categories can be used in a simple case, namely "lawn" and "non-lawn", i.e. for each image point in the image it is determined whether the image point indicates a lawn. It goes without saying that more than two categories can also be used, in order to, for example, also recognize roads, paths, buildings, vehicles or people or other objects mentioned, and assign them to image points if necessary. Based on the category "non-lawn" - the category "background" is also conceivable, to which everything that is not assigned to another category is assigned. Compared to pure object recognition, which (only) recognizes objects in the image, more information is added by all these labelings.

[0011] In principle, the classification does not have to be performed for each individual pixel, but can also be performed for segments or parts of the image, each of which includes a plurality of pixels. Such segments can be fixedly predetermined or can also be formed within the scope of semantic segmentation, possibly also with different sizes within the image. For this purpose, one or more artificial neural networks or generally artificial intelligence-based pattern recognition methods are preferably used, which are also referred to below as machine learning algorithms, and will be discussed in more detail below.

[0012] In this way, a classified result image (output image), i.e. a kind of map, can be generated from an image (input image), which indicates where specific objects or general features are located. In a simple example with two categories "lawn" and "non-lawn", the edge of the lawn can be identified or determined by the boundary between the areas of the image assigned either to "lawn" or "non-lawn" (the associated result image can be, for example, pure black / white). The result image can then be used in particular for controlling the device, i.e., for example, the device can be stopped or turned before or when the edge of the lawn is reached. It is also conceivable, for example, to distinguish between uncut lawns and already cut lawns as categories.

[0013] At this point it should be mentioned that, although a camera or video camera at or in such a mobile device may preferably be considered for detecting such (digital) images, within the scope of the present invention, as already mentioned, representations of the environment detected using other sensors such as the mentioned lidar or ultrasonic sensors or thermal imagers or, if necessary, obtained by further processing should also be understood as images; semantic segmentation can then also be applied there.

[0014] By carrying out this procedure repeatedly or continuously, in particular for the respectively current image of the environment, i.e. in particular when the (mobile) device is moving, the mobile device can be operated or controlled based on the respectively current result image; thus, for example, a mowing robot can be controlled so that it only drives as far as the identified lawn edge (and mows the lawn there), but then, for example, continues to drive along the lawn edge but cannot go beyond it.

[0015] As has now been shown, although it is advantageous in principle that as many different objects or object types as possible are identified or classified in order to determine navigation information based thereon. However, not all types of objects are equally well suited for determining navigation information. Here, in particular, the dynamics of the object may have an influence. If the object considered in this case is always in the same position, it is particularly advantageous for, for example, in the case of a mowing robot, such as, in particular, for the necessary safe and repeatable navigation of an environment that is repeatedly driven by a mobile device. Since basic navigation methods such as SLAM ("simultaneous localization and mapping") construct an internal map of the environment, the map is oriented to the identified object, and therefore a map based on an object that constantly changes position will not achieve reasonable navigation. Such an object that constantly changes position is, for example, a person or an animal, but also, for example, the cloud mentioned or a leaf located on the ground. Objects such as houses, walls or even trees (at least usually) do not change or at least do not change position in a short period of time.

[0016] In order to take this into account, it is proposed to assign one of a plurality of classes to at least some of the segments of the image, in particular to each of the image points, based on the image. As explained above, this can be done within the scope of semantic segmentation. However, the class used in this case should now characterize a measure of the dynamics of the object assigned to the relevant segment. Based on the image, navigation information can be determined taking into account the assigned classes, which can then be provided and used for navigation as already mentioned. Together with the assigned classes, the image can also be referred to as a classified image or a classified result image; in this sense, the navigation information is then determined based on the classified result image.

[0017] In this way, navigation can be performed better and in a more targeted manner, since objects for which a certain dynamicity or too high a dynamicity (and therefore unsuitable for navigation) are expected can be taken into account accordingly, i.e. can be taken into account less intensively or not at all when determining the navigation information, for example.

[0018] In one embodiment, determining the navigation information includes determining long-term navigation information, which is used in particular over a plurality of movement cycles of the mobile device. For example, a movement cycle can be understood as mowing the (entire) lawn once or vacuuming the house once. The long-term navigation information is then determined based on segments in the image or objects assigned to the segments, to which the segments are assigned a category that meets a predetermined criterion; that is, for example, to static objects or objects that are assumed to be static.

[0019] Navigation information that can be used over a longer period of time can thus be determined very easily by segmentation or assignment of categories, since it can be assumed that the basic objects remain unchanged over a longer period of time.

[0020] Particularly preferably, the plurality of classes comprises exactly two classes, one of which comprises static objects or objects that are assumed to be static, and the other class comprises the remaining objects (i.e. also dynamic objects). As has already been shown, the assignment can be made particularly quickly, efficiently and accurately, wherein only a distinction must be made between two classes. However, it is also possible in principle to use more than two classes, for example in order to map different "intensities" of dynamics, i.e., to view leaves on the ground differently from (very fast-moving) animals.

[0021] Generally speaking, the output of semantic segmentation of an image can be a kind of "heat map", that is, each segment is first assigned only a certain probability, and the segment can be assigned to a specific object with this probability; accordingly, there can also be multiple probabilities for different objects for the same segment. Then the object with the highest probability can be selected.

[0022] In this case, it is particularly expedient to assign the categories based on a predetermined assignment of a large number of different objects to one of the multiple classes, i.e., for example, one of the two mentioned categories. Such a predetermined assignment can, for example, include assignment criteria in the form of a list. The list can include a large number of different objects, such as houses, walls, trees, stumps, stones, rocks, roads, lawns, flowers, flower beds, people, animals, leaves, sky, clouds. Each of these objects is then assigned one of the categories; thus, for example, the object house can be assigned the category static object, while the object animal can be assigned the other category (if there are two categories). This allows a particularly fast assignment and therefore also allows a fast determination of navigation information. It goes without saying that the list or objects are only exemplary here, and other and / or additional objects can actually be considered, for example depending on the field of application.

[0023] In one embodiment, determining the navigation information comprises determining short-term navigation information, which is used in particular for only one movement cycle of the mobile device. The short-term navigation information is determined based on segments in the image or objects assigned to the segments, to which segments are assigned a category that does not meet a predetermined criterion (which is used for the long-term navigation data); that is, for example, not assigned to static objects or objects that are assumed to be static. Here - as also when determining the long-term navigation data - in particular the two mentioned categories can then be used; if the object can be assigned to the category static object or object that is assumed to be static, the predetermined criterion is met. It goes without saying that here - as also when determining the long-term navigation data - other criteria can also be used.

[0024] In this way, navigation can also be optimized in a targeted manner over short periods of time, so that, for example, within the scope of the current movement cycle, a location where such an object was detected can be approached again later; that is, if, for example, an animal is sitting on the lawn, this may no longer be there later, so that mowing can then be carried out there.

[0025] As mentioned, this process, i.e. the acquisition of images and their use for navigation, can be carried out in particular repeatedly or continuously, i.e. in particular when the (mobile) device is moved. If images of the environment are therefore provided repeatedly, respectively, it is preferred that a class is assigned only with each n-th image - or that a class is newly determined only with each n-th image. In this case, n is an integer greater than 1, preferably greater than or equal to 10. This allows computing power to be saved, but navigation itself is not affected or hardly affected, since at typical movement speeds of mobile devices, the objects in the images acquired here do not change particularly quickly.

[0026] Since semantic segmentation adds more or new information to an image, it can be used in particular in self-driving cars in order to (better) identify multiple objects and their respective categories on an image, such as cars, people, traffic signs, road obstacles, etc. Other autonomous robots can also use semantic segmentation in order to obtain more information in their environment; in the case of a robotic arm, for example, in order to identify which objects the semantic segmentation should select; in the case of a drone, for example, in order to identify sky boundaries and obstacles.

[0027] However, such semantic segmentation typically requires high computational power. Therefore, it is challenging to use semantic segmentation in the context of small (mobile and especially autonomous) devices or vehicles due to the high computational requirements and complex architecture of deep neural networks, which are mostly required in order to make semantic segmentation fast and accurate.

[0028] In one embodiment, the categories are respectively assigned to the segmentation only for a selected region of the image, or the assignment is performed for the selected region of the image with a higher priority than for the rest of the image, i.e., for example, a higher share of the (available) computing power of the execution computing unit is used for the selected region than for the rest, i.e., at least with respect to the share of the image that the selected region occupies. In other words, the available computing power is therefore used or divided (at least to the extent that it is used or available for semantic segmentation) so that a higher computing power is used per image share for the selected region than for the rest. If the selected region comprises, for example, half of the image (i.e., half of the image points present in the image), more than half of the computing power is used for the selected region, for example three quarters or more.

[0029] The region selected or to be selected does not necessarily have to be a coherent region; it can also be an individual subregion of the image.

[0030] This allows for a selected area, i.e. an area of ​​the image (and thus of the environment) which is classified as particularly relevant, for example on account of a specific feature, for example at the height of a house or a wall, to ensure a fast and nevertheless precise determination or recognition of features, while nevertheless also recognizing or determining features in the remainder of the image. Compared to using computing power uniformly for the entire image, the available computing power can thus be used more efficiently. Likewise, the required computing power can be reduced when recognizing features, at the same speed and accuracy for the selected area. This allows use even in the case of smaller (mobile) devices, in which typically less computing power is available.

[0031] Therefore, in particular artificial intelligence or "deep learning" (preferably artificial neural networks) is used in order to perform a semantic segmentation of the image and identify relevant segments, for example at the height of the horizon or above the horizon, for example where houses or trees or walls (or generally static objects) can be expected. This is done with low computing resources and high precision by focusing on the relevant areas, without losing the context of the entire image. In particular, the solution uses a data set specific to the environment in which such a device or robot will work, in order to train a deep learning network that focuses on the segmentation and definition of relevant static objects, for example. It is thus possible to select a region of interest in an image particularly simply and quickly.

[0032] This approach is different from similar approaches used in conjunction with semantic segmentation in fields such as self-driving cars. Semantic segmentation requires high processing overhead so that the semantic segmentation runs not only accurately but also quickly. This also applies in the case of small devices or vehicles that work especially near the ground, but the computing power there is usually more limited than the hardware used in cars. The image that should be semantically segmented (i.e., the input image for semantic segmentation) also cannot (or at least should not) be cropped to the region of interest, because the context (i.e., also the rest of the image, outside the selected area) is (also) very important for smaller and more accurate networks (artificial neural networks). Therefore, the image is used as input, focusing (selecting) a specific area, but without losing the context of the complete image, while enabling difficult computational operations to be focused on such an area, so as to improve not only throughput but also accuracy.

[0033] The proposed method is furthermore (more) robust with respect to different environmental changes in the image, for example when the viewing angle of the camera is displaced. The scheme is also robust with respect to light changes in the environment, weather conditions, seasons, possible similar objects in the garden, etc. A big problem in the case of artificial neural networks for semantic segmentation is that they are complex and time-consuming if they should be accurate; however, if high speed is a requirement, the accuracy must be reduced when the network is small. The proposed scheme now makes it possible to achieve not only accuracy by keeping the entire image, but also high speed, by using a small or smaller network that focuses primarily on the relevant areas of the image (the selected areas).

[0034] The proposed solution is therefore also more robust, since more computing power is applied to the focused or selected area, without losing contextual information of the entire image. The area of ​​interest or selected will typically be, for example, an area at the height of the horizon, but the network needs, for example, information about weather conditions (e.g. sunny or cloudy) in order to adapt to the current lighting conditions. If it is, for example, sunny, the windows in a house may, for example, reflect and look different than when it is cloudy. If the network would be distorted in the direction of or with respect to the colors, errors would occur in one of these two scenarios. With the contextual information, a better generalization (adaptation) is possible.

[0035] In this regard, the proposed approach is particularly special also with respect to other deep learning solutions, as it targets designed environments (e.g. gardens and outside areas) and devices or robots that work close to (or closer than autonomous vehicles) to the ground.

[0036] Most methods for semantic segmentation function in such a way that the entire image is analyzed and the same set of computational operations is performed for each region, which results in the use of the same or evenly distributed computing power. This is useful in other fields due to the unpredictability of the appearance of objects on the image (for example, in the case of self-driving cars). However, especially in the case of small autonomous robots, the use field and perspective are well defined based on the robot size, camera configuration, placement and orientation. For example, by analyzing multiple internal data sets (for example, through previously performed training), the areas in the image that usually show relevant objects, such as houses or walls, can be averaged, and after being projected onto the image, the region of interest can be determined, that is, selected. However, the proposed approach is also applicable to other devices in which features in the image of the environment should be recognized.

[0037] The region in the image for which the higher computing power is used is selected in particular as a function of the position of an environment detection device (e.g. a camera) in the device, in particular relative to a level or surface (e.g. underground) on which the device moves. Advantageously, the region in the image is also selected as a function of the current position of the device in the environment, preferably with respect to the operating area of ​​the device in the environment. The procedure for selecting or obtaining the region of interest is carried out in particular mathematically, i.e., for example, by analyzing the camera orientation relative to the ground level and selecting the boundaries of the robot's operating area; for example, the robot should be able to recognize all its objects within three or ten meters.

[0038] In the scope of the present invention, after the image (or image pair) is preprocessed by the positioning engine, the artificial neural network (or deep learning network) preferably receives the image (or image pair) and typically performs dimensionality reduction by, for example, a small or small number of convolutional layers of the artificial neural network, in addition to batch normalization, pooling mechanisms and a custom layout of activation functions. Features are then extracted. This can be done first for the entire image.

[0039] Image pairs can be detected in order to generate depth information or to identify the depth information in the network. This increases the information content. For example, identifying a wall is a challenge because it usually has one color and little texture; when depth information is present, the wall can be more easily identified and correctly segmented.

[0040] In particular, the positioning engine is used to correct the recorded images, ie in particular distortions of the lens, and if necessary also to carry out a color correction.

[0041] For example, dimensionality reduction helps reduce the number of inputs or input values ​​(for neural networks) in order to make processing easier by performing convolutions and focusing on relevant features. An image with, for example, only 720x480 pixels and three color bits already results in 1,036,800 input values, which is usually too much for processing. Pooling achieves "downsampling" by selecting specific values ​​to pass to subsequent layers (e.g. so-called "max pooling" will select and pass the maximum value in a 2x2 matrix, thereby reducing the size of 2x2 pixels to 1x1).

[0042] For example, activation functions act like logic gates in an integrated circuit. They determine which neurons are activated at each stage of the neural network and determine the final output (e.g., for a section of grass or lawn, neurons that focus on specific grass patterns, textures, and colors will be activated). "Batch Normalization" alleviates the problem of so-called internal covariant shift and smoothes the target function by normalizing the inputs in each batch.

[0043] Furthermore, a region of interest of the image is selected or extracted in order to apply - in the selected region of the image - a larger convolution or a further set of a larger number of convolutions, if necessary in addition to other operations. This enables the artificial neural network to take the entire image into account and focus heavier convolution kernels (i.e. such convolution kernels with higher weights) on the region of interest or selected.

[0044] In other words, more computing operations are therefore performed in the selected area than in the case of the rest of the image. This can be done, for example, by first performing feature recognition for the entire image and then additionally performing further computing operations for the selected area, whereby the features are recognized more precisely or better there. For example, an additional neural network can then be used in the selected area. However, it is also conceivable that feature recognition is performed separately (from the outset) for the selected area and the rest of the image. In this case, for example, different artificial neural networks can then be used for the selected area and the rest of the image, which differ, for example, in terms of depth and / or the number of layers.

[0045] The calculations (convolutions, pooling, batch normalization, activations) for the entire image are simpler than those performed for the (selected) region of interest. Since the entire image is evaluated, no contextual information is lost, but more is calculated in the region of interest so that the segmentation can be focused. For example, in the upper region of the image there is usually the (less interesting) sky, which does not contain relevant classes that would help when a lawn edge should be identified; but it can be relevant to identify: whether it is cloudy, clear, etc. For example, in the lower region of the image there is usually grass and closer objects (close to the robot); therefore, more calculations are performed there in particular in order to obtain a more accurate segmentation.

[0046] Because the result of feature recognition (due to the above-mentioned dimensionality reduction) is extended downward, some values ​​(determine or extract features according to the values) can also be stored in advance, so that the upward extension of the result becomes more accurate. When extending downward, values ​​can be stored, and then the values ​​are used again when extending upward, so as to obtain more accurate output. If, for example, a 10x10 matrix is ​​reduced to a 2x2 matrix, and then enlarged to a 10x10 matrix again at the end, the output (i.e. the image obtained) will be fuzzy and inaccurate. However, if, for example, intermediate values ​​are stored with a 5x5 matrix, these stored intermediate values ​​can be used when extending upward. Then, for a specific category (i.e., grass, road, soil, people, house, etc.), the extracted features can be selected, linked, and threshold values ​​can be equipped to the extracted features, and then extended upward to a feature map, i.e., the probability of each of these categories is stored in the image. This then represents the result image.

[0047] The invention also relates to a method for training a machine learning algorithm for a mobile device, such as the one described above. For this purpose, training data are provided; the training data include images of the environment and / or one or more other environments and information about the categories assigned to segments of the images. Based on the training data, the machine learning algorithm is then adapted so that one of a plurality of categories is assigned to each of at least some of the segments of the images. A trained machine learning algorithm is then provided. The trained machine learning algorithm can then be used in the case of the method described above or its variants.

[0048] A computing unit according to the invention, for example a control unit or a control unit of a mobile device or a server or another computer, is configured, in particular in terms of programming, to carry out the method according to the invention.

[0049] The invention also relates to a mobile device which is designed to receive control information which has been determined as described above or which has a computing unit according to the invention. The mobile device also has a drive system and a control or regulating unit for controlling according to the control information. The mobile device can in particular be a mobile device as described above, i.e., for example, a lawnmower robot.

[0050] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this causes particularly low costs, especially if the execution control device is still used for other tasks and therefore originally exists. Finally, a machine-readable storage medium is provided, with a computer program as described above stored thereon. Suitable storage media or data carriers for providing computer programs are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download programs via computer networks (the Internet, intranets, etc.). Such downloading can be done wired or cabled or wirelessly (e.g., via WLAN networks, 3G, 4G, 5G or 6G connections, etc.).

[0051] Further advantages and embodiments of the invention are apparent from the description and the accompanying drawings.

[0052] The invention is schematically illustrated in the drawings on the basis of exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A mobile device is schematically shown in the context of explaining the present invention.

[0054] Figure 2 An image detected by an environment detection device of a mobile device is shown.

[0055] Figure 3 In the context of the method according to the invention, in a preferred embodiment, the Figure 2 The resulting image is generated from the image.

[0056] Figure 4 The sequence of the method according to the invention is schematically shown in a preferred embodiment.

[0057] Figure 5 The sequence of the method according to the invention in a further preferred embodiment is schematically shown. DETAILED DESCRIPTION

[0058] exist Figure 1 1 shows schematically and exemplarily a mobile device 100 in an environment 120, in particular in a work area, for presenting the present invention. The mobile device 100 is exemplarily a lawnmower robot having a control or regulating unit 102 and a drive unit 104 (with wheels) for moving the lawnmower robot 100, for example, along a trajectory 130. In addition, the lawnmower robot 100 exemplarily has a sensor 106 designed as a camera with a field of view (indicated by dashed lines). For better illustration, the field of view is selected to be relatively small here; in practice, however, the field of view can also be up to 360° (for example, but at least 180° or at least 270°).

[0059] The mowing robot 100 also has a computing unit 108, for example a control device, by means of which data can be exchanged with a superordinate system 110, for example via the indicated radio connection. For example, a trajectory (or navigation information in general) can be determined in the system 110 and then transmitted to the system 108 in the mowing robot 100, which is then to follow the trajectory. However, it can also be provided that the trajectory (or navigation information in general) is determined in the system 108 itself or is obtained there in another way. Instead of the trajectory, however, the system 108 can also obtain control information, for example, which is determined based on the trajectory and based on which the control or regulating unit 102 can move the mowing robot 100 via the drive unit 104 in order to follow the trajectory. The trajectory 130 is indicated here only as an example.

[0060] The mowing robot 100 is intended to move independently on the ground or in the environment 120 or to navigate there, for example, and in this case to mow the grass (not explicitly shown here). In addition, a plurality of different objects are shown in the environment by way of example, namely a house 140, a person 142 and a tree 144. As already mentioned, these objects can be detected by the mobile device 100 or its camera 106 (ie, they are visible in the images recorded by means of the camera) and then taken into account during navigation, i.e., when determining navigation information.

[0061] As has also been mentioned, different objects can be of different importance or relevance for navigation, ie, in particular for long-term navigation, ie, for example, in order to know how the surroundings appear in principle.

[0062] exist Figure 2 The figure shows the environment detection device, such as Figure 1 250 detected by the camera 106 of the mowing robot 100 in FIG. Various objects, houses 140, people 142, trees 144 and additionally clouds 146 (from the perspective of the mowing robot or its camera) can be seen therein.

[0063] It can also be seen here that the majority of the objects in image 250 are located approximately at or slightly above the center or horizon level, which is particularly due to the close-to-ground position of mowing robot 100 or its camera 106. This region at or slightly above the center or horizon level of image 250 is denoted by 256, while the region close to the ground is denoted by 252 and the upper region (here essentially the sky) is denoted by 254.

[0064] exist Figure 3 For example, in the context of the method according to the invention, in a preferred embodiment, Figure 2 The resulting image 350 is generated from the image 250 .

[0065] In this case, features are extracted from the image 250, for example, for each image point (pixel), and then, as a function of these features, one of a plurality of classes is assigned to each image point, for example. By way of example, only the classes "static objects" C1 and "other objects" C2 should be used, which on the one hand may be completely sufficient for a mowing robot and, moreover—as mentioned—can be implemented particularly efficiently. In the result image, the two classes are shown lightly hatched (C1) and heavily hatched (C2); in practice, this can be black / white or, in the case of more than two classes, in different colors; various other features that can be processed digitally can also be used. The result image 350 shown here can also be understood, in particular, as a combination of the image 250 with the assigned classes.

[0066] At this point it should be mentioned that, for the sake of clarity, only the relevant (and shown) objects themselves are shown according to the classification. However, depending on the implementation, other areas of the image 250 can also be classified accordingly.

[0067] For example, the objects can be classified or assigned classes by first determining with what probability a particular object can be seen in a segment (or pixel) of the image 250. For example, a (completely comprehensive) list of different objects can be used here, as already mentioned above. Then, for each segment, the object that can be seen there with the highest probability can be assumed to be seen there. Then, the segment can be assigned—for example, according to the list—a class corresponding to the object, i.e., for example, "static object" C1 or "other objects" C2.

[0068] As mentioned, in the case of semantic segmentation, the entire image 250 may be processed in the same manner, in particular when extracting features. However, it may be preferred to select a region of the image 250, wherein the selected region accounts for a smaller portion of the (entire) image 250 than for the rest of the image 250 using a performing calculation unit, such as Figure 1 A higher share of the computing power of control unit 108 in . By way of example, such a selected region is already mentioned region 256 ; in this case, this is the center region of image 250 .

[0069] The selection of this area 256 can be made, in particular, based on the current position of the mowing robot 100 in the environment 120 and due to the fact that its camera is relatively close to the ground. Thus, for example, it can be assumed that in the lower area a large part of the image will be lawn, due to the special viewing direction of the camera of the mowing robot. Based on the robot configuration and the camera placement and orientation, the horizon ( Figure 2 Or the horizontal boundary line between region 252 and 256 of region 3) where in the image would extend assuming a flat ground. Typically, the middle portion of the image will therefore contain more relevant features (static objects).

[0070] It can therefore be assumed that relevant objects are located in selected region 256 , so that semantic segmentation with increased computing power compared to the remainder is sensible there; thus, despite the overall lower computing power, relevant objects, for example static objects, can still be precisely determined.

[0071] exist Figure 4 The process of the method according to the invention in a preferred embodiment is shown schematically, that is to say in the form of a flow chart. An exemplary sequence of steps that can be performed is shown here.

[0072] In step 400, an image, such as from Figure 2An image 250 is detected and obtained, for example, by a computing unit. There, the image, as an input image, can be preprocessed by means of an image processing system, if necessary, in step 402 in order to correct for lens distortion, lighting conditions, perspective, etc. The (if necessary preprocessed) image is then forwarded to an artificial neural network 404 (i.e., a machine learning algorithm) for processing.

[0073] For example, in step 406, dimensionality reduction is performed. Because calculations cannot (or should not) be performed for each pixel of the image due to the size (amount of data), the image dimension is reduced through convolution, pooling, batch normalization, and activation functions as explained above.

[0074] Then, in step 408, features 410 of the image are extracted, for example based on a number of factors such as shape, color, context (kontext), pattern, light changes, image context, etc. By definition, a feature is, for example, a characteristic of a property or phenomenon that can be measured individually. In the context of neural networks, this would, for example, mean training specific neurons to focus on specific characteristics of the image; some neurons may focus on color, pattern, texture, etc. In particular, the extraction of features should be understood as the activation of specific neurons in specific areas in the image, and the network forwarding their values ​​for analysis, threshold determination and finally segmentation into specific categories.

[0075] In step 412, a portion of these features are stored to later ensure simple and fast up-scaling, which means that the features help to scale the result to the size of the original input image without losing resolution. The network also learns the stored values; this is more efficient than interpolation or nearest neighbor interpolation of values ​​because contextual information is added. To e.g. up-scale a square, typically only four points must be used; to up-scale a circle, at least in image processing - significantly more points will be needed.

[0076] Additionally, in step 414, context-aware pooling, batch normalization, activation functions, and others may be performed. "Context-aware" may be understood here as the network extracting context information, such as: "This image region looks like a door, a window, and a wall. Therefore the entire region is most likely a house." If something is then found in this "house" that looks most likely to be the sky, this information may be discarded because it does not match the context. This may contribute to or result in "grass" not being recognized in the middle of the sky because this is not possible in a context-aware manner.

[0077] In step 416, a region of the image is selected (eg Figure 3area 256 in the image); the area can be determined or calculated, for example using a method that depends on the physical properties of the lawn mowing robot, the placement of the camera and its orientation relative to the base surface.

[0078] Then, in step 418, with the aid of a neural network, a heavier convolution or a larger set of operations is focused on the selected area, while (only) a smaller set of operations is applied to the rest of the image.

[0079] Then, in step 420, the obtained feature map is classified into its corresponding category, such as but not limited to: "static object" and "other objects", such as Figure 3 424 - the classes are thus assigned here, i.e. in particular according to the already mentioned assignment or list 450. The results of the calculations for the rest of the image and the focused calculations in the selected area on the image are mapped and linked to the resulting classified matrix 422. Then, in step 424, the classified matrix (i.e. the intermediate result) is expanded upward in the decoding phase using the residuals (stored features or values) of the encoding phase, so that in step 426 a result image is obtained as output, which is similar in size to the input image.

[0080] The resulting image can then, for example, be mapped and, if necessary, post-processed (eg, class indexing, output class separation, thresholding) and forwarded to a possible application.

[0081] In step 428, navigation information 430 can then be determined based on the result image or based on the image and the assigned category and provided in step 432. Determining navigation information 430 can in this case in particular include determining long-term navigation information.

[0082] A different arrangement or a different order of the method or steps may be chosen; as has been shown, it is possible to perform the selection of the region of interest with a small impact on the accuracy, for example, after the dimensionality reduction, and even better inference times may be achieved. Context-aware feature selection may be performed not only in the entire image but also (only) in selected regions, for example depending on the application and how strongly the focus should be placed on the selected regions.

[0083] exist Figure 5 The process of the method according to the invention in a preferred embodiment is shown schematically, i.e. in the form of a flow chart. The training of a machine learning algorithm (or neural network) is shown here, such as in Figure 4 The machine learning algorithm used in the case of the method.

[0084] To this end, training data 502 are provided in step 500. These training data include images of the environment and / or one or more other environments, and information about the categories assigned to the segments of the images. This is then followed by adapting a machine learning algorithm 506 based on the training data in step 504, so that one of a plurality of categories is assigned to each of at least some of the segments of the images. The trained machine learning algorithm is then provided in step 508.

Claims

1. A method for determining navigation information for a mobile device (100), in particular a vehicle or a robot that moves at least partially automatically, further in particular a lawn mowing robot, the method comprising: providing (400) an image (250) of an environment (120) in which the mobile device (100) is moving or should be moving; Based on the image (250) and in particular within the scope of semantic segmentation, one of a plurality of classes (C1, C2) is assigned (420) to each of at least some segments, in particular image points, of the image (250), wherein the class (C1, C2) characterizes a measure of the dynamics of an object (140, 142, 144, 146) that can be assigned to the relevant segment; determining (428) navigation information (430) based on the image (250) taking into account the assigned categories (C1, C2); and The navigation information is provided (432).

2. The method according to claim 1, wherein determining (428) the navigation information (430) comprises determining long-term navigation information, the long-term navigation information being used in particular over a plurality of movement cycles of the mobile device, Long-term navigation information is determined based on segments in an image (250) or objects (140, 142, 144, 146) assigned to the segments, which are assigned a class (C1) that meets a predetermined criterion, in particular to static objects or objects that are assumed to be static.

3. The method according to claim 1 or 2, wherein the plurality of classes (C1, C2) comprises exactly two classes, wherein one class (C1) comprises static objects or objects assumed to be static and the other class (C2) comprises the remaining objects.

4. The method according to claim 2 or 3, wherein determining (428) the navigation information comprises determining short-term navigation information, the short-term navigation information being used in particular for only one movement cycle of the mobile device, In this case, short-term navigation information is determined based on segments in the image (250) or objects assigned to the segments, to which a category (C2) not meeting a predetermined criterion is assigned. 5 . The method according to claim 1 , wherein the categories are assigned ( 420 ) based on a predetermined assignment ( 450 ) of a plurality of different objects to each of a plurality of categories.

6. A method according to any of the preceding claims, wherein the categories are respectively assigned (420) to the segments only for a selected area (256) of the image or for a selected area (256) of the image with a higher priority than for the rest of the image.

7. The method according to claim 6, wherein the selected area (256) in the image is selected based on a position of an environment detection device (106) of the mobile device (100), in particular relative to a level on which the device (100) is moved, with which the image (250) is detected, and / or The selected area in the image is selected based on a current position of the device (100) within the environment (120).

8. A method according to any of the preceding claims, wherein images (250) of the environment are repeatedly provided, wherein the category is assigned only in every nth image, wherein n is an integer greater than 1, preferably greater than or equal to 10.

9. The method according to any of the preceding claims, wherein the category is also assigned (420) and / or the navigation information is determined (428) using a machine learning algorithm (404).

10. The method according to any one of the preceding claims, further comprising: determining control information for moving the mobile device (100) based on the navigation information (430), and Control information is provided and / or the mobile device (100) is moved based on the control information.

11. A method for training a machine learning algorithm for a mobile device (100), in particular a vehicle or a robot that moves at least partially autonomously, said vehicle or robot being intended to move in an environment, wherein the machine learning algorithm is arranged to assign, based on an image (250) of an environment, one of a plurality of categories (C1, C2) to each of at least some segments of the image, wherein the category characterizes a measure of the dynamics of an object assignable to the relevant segment, the method comprising: providing (500) training data (502) comprising images of the environment and / or one or more other environments and information about categories assigned to segments of the images, adapting (504) the machine learning algorithm (506) based on the training data such that one of a plurality of classes is assigned to each of at least some of the segments of the image, and A trained machine learning algorithm is provided (508).

12. A computing unit (108) comprising a processor configured to perform the method according to any one of the preceding claims.

13. A mobile device (100) which is configured to receive control information determined according to the method of claim 10 or has a computing unit (108) according to claim 12, a drive system and a control or regulating unit for controlling according to the control information, The mobile device (100) is in particular constructed as an at least partially automatically moving vehicle, in particular a passenger vehicle or a freight vehicle, or a robot, in particular a household robot, such as a suction and / or wiping robot, a floor or road cleaning device or a lawn mowing robot, or a drone.

14. A computer program comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to claims 1 to 10.

15. A computer-readable data carrier having stored thereon a computer program according to claim 14.

Images