Method for determining the type of subsurface

Abstract

The invention relates to a method for determining the type of substrate in an environment in which a mobile device, in particular a robotic lawnmower, moves autonomously, using at least one machine learning algorithm, comprising performing a substrate determination, including: providing an image of an environment in which the mobile device moves; assigning, in particular within the framework of semantic segmentation, at least one of several object classes to segments, in particular pixels, of the image in order to obtain an object class assignment; determining, based on the image of the environment and the object class assignment, a substrate image (450) that at least partially depicts a substrate on which the mobile device moves, in particular by a perspective projection onto a plane;Assigning segments, in particular pixels, of the background image to one of two background classes (GC1, GC2) in order to obtain a background class assignment that indicates a type of background; and providing the background image and the background class assignment.

Description

[0001] The present invention relates to a method for determining a type of subsurface in an environment in which a mobile device, in particular a robotic lawnmower, moves autonomously, a mobile device as well as a computing unit and a computer program for carrying out the method. Background of the invention

[0002] Mobile devices or processing equipment, such as robotic lawnmowers, vacuum or floor mops, or other household robots, typically move within an environment that requires maintenance, such as a garden or a home. One of the main tasks here is the navigation of the mobile device, which is required to move within the environment, e.g., along a specific path within certain boundaries (the work area). Various types of surfaces can occur during this process. Disclosure of the invention

[0003] According to the invention, a method for determining a subsurface type, a mobile device, a computing unit, and a computer program for implementing the method are proposed, each having the features of the independent patent claims. Advantageous embodiments are the subject of the dependent claims and the following description.

[0004] The invention relates to mobile devices which move in particular in an automated or autonomous manner, such as preferably a robot, e.g. a robotic lawnmower. Even though the invention will be explained below primarily with reference to a robotic lawnmower, other mobile devices or robots are also conceivable. Examples of such mobile devices (or mobile work devices) are, in general, robots and / or drones and / or vehicles which move in a partially automated or (fully) automated manner (on land, water or in the air). Robots which come into consideration include, for example, household robots such as vacuum and / or floor cleaning robots, floor or street cleaning devices or robotic lawnmowers, but also other so-called service robots, as well as vehicles which move at least partially in an automated manner, e.g. passenger transport vehicles or goods transport vehicles (also so-called industrial trucks, e.g. in warehouses), but also aircraft such as so-calledDrones or watercraft. Self-driving or autonomous vehicles or trucks can also be considered mobile devices.

[0005] Navigation information can be determined for such a mobile device, based on which the mobile device can move. Such navigation information can, for example, comprise a specific movement path or a trajectory along or according to which the mobile device should move. In the case of a robotic lawnmower, for example, a movement path can be determined which is such that the lawn (i.e. a work area in the environment) is driven over and mowed as quickly and efficiently as possible. For this purpose, control information for moving the mobile device can be determined based on the navigation information, based on which in turn the mobile device can be moved. Such a mobile device typically has a drive system and a control or regulating unit for actuating according to such control information.

[0006] For example, a robotic lawnmower generally requires it to stay within the boundaries of the garden or lawn, as there may be areas that are not suitable for optimal operation, such as ponds, mud, gravel roads, flowerbeds, etc. To achieve this, a wire or cable can be laid in the garden or lawn, particularly along the edges of the lawn or where the robotic lawnmower is intended to mow. This wire can be detected by a sensor in the robot. In this context, the wire can also be referred to as a lawn edge.

[0007] Although this allows the robot's movement to be limited as desired, it requires a lot of effort to lay the wire or cable.

[0008] Another possibility for a (mobile) device to recognize certain features in the environment, such as a lawn edge in the case of a robotic lawnmower, is the use of environment detection means or one or more sensors, such as cameras or video cameras, thermal cameras, radar sensors, lidar sensors, laser rangefinders, ultrasonic sensors, inertial sensors and / or odometers. In particular, within the scope of the present invention, detection means or sensors are to be considered that enable a visual representation of the environment, i.e. preferably cameras or video cameras. However, depth information can also be obtained, for example using other sensors such as lidar or ultrasonic sensors, which can be represented visually or as (digital) images. Such an image of the environment can then be analyzed in order to identify certain things or objects in the environment, such as, for example,to detect or identify the mentioned lawn edge in the case of a robotic lawnmower.

[0009] Not only can various objects or types of objects in the environment be detected or identified, which can be taken into account when determining navigation information, but it has also been shown that the ground itself and various types of ground can be detected, for example, whether the ground is grass or not, or whether there is mown or unmown grass in a particular area.

[0010] This is where so-called semantic segmentation comes into play. This involves assigning each pixel in an image to one of several classes, meaning the pixels are labeled or classified. For this purpose, features of the pixels are typically first determined or extracted, based on which the class is then assigned. Such features can include shape, color, context, pattern, light variance, and / or image context. For example, a pixel could be assigned the color green, while the surrounding pixels could also all be green. This would indicate lawn (or grass).

[0011] Various features can be considered as classes. In the example of the robot lawnmower, for example, two classes can be used in a simple case, namely "lawn" and "not lawn", i.e. for each pixel in the image it is determined whether this shows grass or not. It goes without saying that more than two classes can also be used, for example to recognize paths, roads, buildings, vehicles, trees, leaves, people or other objects and to assign them to a pixel if necessary. It is also conceivable - based on the class "not lawn" - to have a class "background" to which everything that is not assigned to another class is assigned. All of these labels add more information than with pure object recognition, in which (only) objects in an image are recognized.

[0012] In principle, classification does not have to be performed for each individual pixel; it can also be performed for segments or parts of the image, each comprising multiple pixels. Such segments can be predefined or created as part of the semantic segmentation process, possibly with different sizes within the image. For this purpose, one or more artificial neural networks or pattern recognition methods based on artificial intelligence—hereinafter also referred to as machine learning algorithms—are preferably used, which will be discussed in more detail below.

[0013] In this way, a classified result image (output image) can be created from the image (input image), i.e. a type of map, which indicates where certain objects or general features are located. In the simple example with the two classes "lawn" and "not lawn" (or "grass" and "not grass"), it is possible to determine, for example, the condition of the entire area, i.e. whether there are many places where there should actually be grass but there isn't. In the example with the two classes "mown grass" and "unmown grass", it is possible, for example, to recognize or determine where the robotic lawnmower still needs to mow. The result image can then be used in particular to control the mobile device or robotic lawnmower, i.e. it can be used to drive or navigate specifically to grass that has not yet been mown.

[0014] At this point, it should be mentioned that although cameras or video cameras on or in such a mobile device are preferably considered for capturing such (digital) images, within the scope of the present invention, as already mentioned, environmental representations captured with other sensors, such as the aforementioned lidar or ultrasonic sensors or even thermal cameras, or possibly obtained through further processing, are also to be understood as images; semantic segmentation can then also be applied there.

[0015] By performing this procedure repeatedly or continuously for a current image of the environment, i.e., particularly when the (mobile) device is moving, the mobile device can be operated or controlled based on the current resulting image; for example, a robotic lawnmower can be controlled in such a way that it only drives to a detected area with unmown grass (and then mows there), but not beyond.

[0016] Within the scope of the present invention, based on the obtained image, at least one (i.e., possibly more than one) of several object classes is assigned to segments, in particular pixels, of the image. This results in an object class assignment. The image, together with the object class assignment, can also be referred to as a resultant image, as already mentioned above.

[0017] This can be done, for example, by assigning one or more probabilities to a segment, where a probability indicates the probability with which the segment can be assigned to a certain object class. The classes mentioned above can be considered as object classes here. For example, a segment can be assigned the class “lawn” or “grass” with a probability of 56%, the class “plant” with a probability of 30%, and the class “leaf” or “leaves” with a probability of 10%. In this way, more than one object class can generally be assigned to a segment. However, for this procedure it is preferred if exactly one object class is assigned to each segment. For this purpose, for example, the object class with the highest probability can be selected. This can be done within the framework of semantic segmentation, as explained above.

[0018] Based on the image and the object class assignment (or based on the image and the resulting image), a background image is then determined. The background at least partially depicts a surface on which the mobile device is moving. Preferably, based on the image of the environment and the object class assignment, an area of ​​the image of the environment that corresponds to one of one or more selected classes of the plurality of object classes can be determined. The one or more selected classes can in particular be assigned to a background, i.e. they are classes that can be assumed to be part of the background or can be assigned to a plane on which the mobile device is moving. These can be, for example, the object classes “lawn” or “grass” or “road” or “path” or “weeds”.

[0019] The background image is then generated from the area of ​​the image, particularly through perspective projection onto a plane. Since the mobile device, particularly in the case of a robotic lawnmower, moves very close to or on the ground (subsurface), its camera is also very close to the ground. In the image captured, the ground or subsurface is thus represented in perspective and is partially distorted. By knowing the position and / or orientation of the camera (e.g., how far it is above the ground and its detection angle), a 2D image of the subsurface—the background image—can be determined.

[0020] One of two (and in particular exactly two) background classes can then be assigned to segments, in particular pixels, of the background image. This results in a background class assignment. This can be done, for example, in the same way or in a similar way to the assignment of object classes above, but here only two classes are assigned. Since only two classes need to be distinguished here, this process can be faster and more efficient, and in particular can also be automated. Manual labeling of background images (or so-called "ground truth images") is not necessary here - even when training a machine learning algorithm used. This can therefore also be done unsupervised. Such assignment is then particularly robust against changes in perspective, lighting conditions and any weeds in the lawn.

[0021] This ground determination can preferably be performed repeatedly for different images of the surrounding area, for example, during a control drive or other movement of the mobile device in the surrounding area. It is useful if images are obtained that together cover the entire (desired) ground in the surrounding area, for example, the entire lawn or the entire working area of ​​the mobile device.

[0022] A map of the environment can then be provided, which includes, in particular, the boundaries within which the mobile device is intended to move. In other words, the environment map can contain, for example, the working area. The subsurface images and the subsurface class assignments can then be assigned to the environment map to obtain a subsurface environment map. In such a subsurface environment map, the types of subsurface are then specified, in particular for a specific area (e.g., within the boundaries or working area of ​​the mobile device), and are specifically divided into two types of subsurface.

[0023] Particularly when the autonomous, mobile device is a robotic lawnmower, the ground classes can be one of the following pairs of ground classes: "grass" and "not grass", or "mown grass" and "unmown grass". This information can then be included in the ground environment map. It goes without saying that one of the two pairs can be used, but both can also be used - in which case the described procedure should be carried out separately if necessary, at least once the ground image is available. In principle, such a ground environment map can then be displayed, e.g. on a display - if necessary once it has been put into a form that is understandable for a user, e.g. with a suitable graphic representation.

[0024] If the ground classes are the pair of ground classes "mown grass" and "unmown grass," in one embodiment, based on the ground environment map, a coverage value can be determined with respect to a proportion of mown or unmown grass in the environment, particularly within the boundaries. Information about the coverage value can then be provided. This can, for example, inform a user how much of their lawn has already been mown and / or how effective the robotic lawnmower was.

[0025] Particularly preferably, based on the information about the coverage value, control information for moving the mobile device can also be determined, specifically such that the coverage value is increased or decreased, for example, so that the areas not yet mowed are also mowed. This control information can be provided and / or the mobile device can be moved based on the control information. This allows, for example, automated re-mowing of areas that have not been mowed or have not been mowed correctly.

[0026] If the subsoil classes are the pair of subsoil classes grass and non-grass, in one embodiment, a health value can be determined based on the subsoil environment map, based on the proportion of grass or non-grass in the environment, particularly within the boundaries. Information about the health value can then be provided. This can be used to inform a user, for example, about the condition of their lawn and whether grass needs to be reseeded in certain areas.

[0027] Because semantic segmentation adds more or new information to an image, it can be used especially in the field of self-driving cars to (better) recognize multiple objects and their respective classes, e.g., cars, people, traffic signs, road obstacles, etc., in an image. Other autonomous robots can also use semantic segmentation to obtain more information in their environment; for example, in robot arms, it can be used to determine which objects to select, and in drones, it can be used to detect sky boundaries and obstacles.

[0028] However, this semantic segmentation typically requires high computational power. Therefore, the use of semantic segmentation in small (mobile and especially autonomous) devices or vehicles remains challenging due to the high computational demands and the complex architectures of deep neural networks that are usually required to make semantic segmentation fast and accurate.

[0029] In one embodiment, the object class is assigned to a segment only for a selected area of ​​the image, or the assignment is made for the selected area of ​​the image with a higher priority than for the rest of the image, i.e., for example, a higher proportion of the (available) computing power of an executing processing unit is used for the selected area than for the rest, at least in relation to a portion of the selected area in the image. In other words, the available computing power (at least insofar as it is or can be used for the semantic segmentation) is used or divided in such a way that a higher computing power is used for each image portion of the selected area than for the rest. If the selected area, for example,covers half of the image (i.e. half of the pixels present in the image), more than half of the computing power is used for the selected area, e.g. three quarters or more.

[0030] The selected area or area to be selected does not necessarily have to be a contiguous area; it can also be separate sub-areas of the image.

[0031] This allows for a fast yet accurate determination or recognition of features for the selected area – i.e., an area of ​​the image (and thus the surrounding area) that is classified as particularly relevant with regard to certain features, for example – while still recognizing or identifying the features in the rest of the image. Compared to a uniform use of computing power for the entire image, the available computing power can be used more efficiently. Likewise, while maintaining the same speed and accuracy for feature recognition for the selected area, the required computing power can be reduced. This also allows for use with smaller (mobile) devices, which typically have less computing power available.

[0032] In particular, artificial intelligence or "deep learning" (preferably artificial neural networks) is used to perform semantic segmentation of an image and identify relevant segments, e.g., in the grass area or in the ground area. This is achieved with minimal computing resources and high accuracy by focusing on relevant regions without losing the context of the entire image. This approach uses a dataset specific to the environment in which such a device or robot would operate to train a deep learning network that focuses on segmenting and defining, for example, relevant static objects. This allows the region of interest in the image to be selected particularly quickly and easily.

[0033] This approach differs from similar approaches used in the field of self-driving cars or similar applications related to semantic segmentation. Semantic segmentation requires a lot of processing to be both accurate and fast. This also applies to small devices or vehicles, especially those operating at ground level, but computing power is generally more limited than in hardware used in cars. The image for which semantic segmentation is to be performed (i.e., the input image for semantic segmentation) cannot (or at least should not) be cropped to the region of interest, since the context (i.e., the rest of the image outside the selected region) is (also) very important for a smaller and more accurate network (artificial neural network).The image is thus used as input, a specific region is focused (selected) without losing the context of the full image, while allowing difficult computational operations to be focused on such a region to improve both throughput and accuracy.

[0034] The proposed approach is also more robust to various environmental changes in the image, e.g., when the camera's viewpoint is shifted. The approach is also robust to changes in ambient light, weather conditions, seasons, possible similar objects in the garden, etc. A major problem with artificial neural networks for semantic segmentation is that they are complex and time-consuming if they are to be accurate; however, if high speed is a requirement, accuracy must decrease if the network is small. The proposed approach now makes it possible to achieve both accuracy by retaining the entire image and high speed by using a small or smaller network that focuses primarily on relevant areas (the selected region) of the image.

[0035] The proposed approach is also more robust because more computational power is applied to the focused or selected region without losing contextual information of the entire image. The region of interest or selected region would normally be, for example, an area below the horizon, but the network needs information about the weather conditions (e.g., sunny or cloudy) to adapt to current lighting conditions. For example, when it is sunny, the grass or lawn usually looks greener to the camera than when it is cloudy. If the network were biased towards or with respect to color, errors would occur in one of these two scenarios. With contextual information, however, it can generalize (adapt) better.

[0036] The proposed approach is also unique compared to other deep learning solutions in that it is focused on the designed environment (e.g. garden and outdoor area) and on devices or robots that work close to the ground (or closer than, for example, self-driving cars).

[0037] Most semantic segmentation methods work by analyzing the entire image and performing the same set of computational operations on each region, resulting in the same or evenly distributed computing power being used. This is useful in other areas due to the unpredictability of the appearance of objects in the image (e.g., self-driving cars). However, especially for small autonomous robots, the area of ​​operation and perspective are well defined due to the robot dimensions, camera configuration, placement, and orientation. For example, by analyzing multiple internal datasets (e.g., through pre-training), the area in which relevant objects such as houses or walls normally appear in the image can be averaged, and after projection onto the image, a region of interest can be determined, i.e. selected.Nevertheless, the proposed approach is also suitable for other devices in which features in images of the environment are to be recognized.

[0038] The area in the image for which the higher computing power is used is selected in particular as a function of a position of the environment detection means (e.g. the camera) in the device, in particular with respect to a plane or surface (e.g. the ground) on which the device is moving. Advantageously, the area in the image is also selected as a function of a current position of the device within the environment, expediently with respect to an operating range of the device in the environment. The procedure for selecting or obtaining the area of ​​interest is carried out in particular mathematically, for example by analyzing the camera orientation with respect to the ground plane and selecting a boundary of the robot's operating range; for example, the robot should be able to detect all of its objects within three or ten meters.

[0039] With such a focus on the background, the determination of the background image described above can be carried out even more accurately and quickly.

[0040] In the context of the present invention, an artificial neural network (or deep learning network) preferably receives an image (or an image pair) after it has been preprocessed by a localization engine and performs dimensionality reduction via, for example, small or few convolutional layers of an artificial neural network, typically alongside a user-defined layout of batch normalization, pooling mechanisms, and activation functions. Features are then extracted. This can initially be done for the entire image.

[0041] Image pairs can be captured to generate depth information or to detect it in the network. This increases the information content. For example, detecting walls is challenging because they typically have color and less texture; if depth information is present, they can be more easily detected and correctly segmented.

[0042] A localization engine is used in particular to correct the distortion of the captured image, i.e. in particular the distortion of the lens, and if necessary also to perform color correction.

[0043] Dimensionality reduction, for example, helps reduce the number of inputs (for the neural network) to facilitate processing by performing convolutions and focusing on relevant features. For example, an image with only 720 x 480 pixels and three color bits already results in 1,036,800 input values, which is usually too many to process. Pooling achieves downsampling by selecting specific values ​​that are passed to the subsequent layers (e.g., so-called "max pooling" would select and pass the maximum value in a 2x2 matrix, reducing a size of 2x2 pixels to 1x1).

[0044] Activation functions, for example, are like logic gates in an integrated circuit. They decide which neurons are activated at each stage of the neural network and determine the final output (for example, for the grass or lawn part, neurons focused on the specific grass patterns, textures, and colors would be activated). Batch normalization mitigates the problem of internal covariant drift and smooths the objective function by standardizing the inputs in each batch.

[0045] In addition, a region of interest from the image is selected or extracted, for example, in order to apply another set of larger convolutions or a higher number of convolutions, possibly in addition to other operations, to the selected region of the image. This allows the artificial neural network to consider the entire image and focus heavier convolution kernels (i.e., those with higher weight) on the region of interest or selected region.

[0046] In other words, more computing operations are carried out in the selected area than in the rest of the image. This can be achieved, for example, by first performing feature recognition for the entire image and then additional computing operations for the selected area, whereby the features are recognized more precisely or better there. Additional neural networks can then be used, for example, in the selected area. It is also conceivable, however, that feature recognition is carried out (from the outset) separately for the selected area and the rest of the image. In this case, different artificial neural networks can be used for the selected area and the rest of the image, for example, which differ in terms of depth and / or number of layers.

[0047] Calculations (convolution, pooling, batch normalization, activations) for the entire image are simpler than those performed on the (selected) region of interest. Since the entire image is evaluated, no contextual information is lost; instead, more calculations are performed in the region of interest, allowing the focus to be on segmentation. For example, the upper region of the image usually contains the (less interesting) sky, which does not contain any relevant classes that are useful when detecting the edge of the lawn; but it is relevant to detecting whether it is cloudy, sunny, etc. The lower region of the image usually contains grass and the closer objects (close to the robot); therefore, more calculations are performed there in particular to obtain more accurate segmentations.

[0048] Since the result of feature detection is downscaled (due to the aforementioned dimensionality reduction), some values ​​(based on which the features are determined or extracted) can be saved beforehand to make upscaling the result more accurate. During downscaling, values ​​can be saved and then reused during upscaling to obtain a more accurate output. For example, if a 10x10 matrix is ​​reduced to a 2x2 matrix and then enlarged again to a 10x10 matrix at the end, the output (i.e., the resulting image) would be blurry and inaccurate. However, if intermediate values ​​are stored in a 5x5 matrix, these stored intermediate values ​​can be used during upscaling. The extracted features can then be classified for specific classes (e.g., grass, road, dirt, people, house, etc.).) are selected, concatenated and thresholded, and then upscaled into a feature map, i.e. the probabilities of each of these classes are stored in the image.

[0049] This then represents the resulting image, specifically the image with the object class assignment described above. Instead of the semantic segmentation procedure described here to obtain the object class assignment, a different procedure can also be chosen. In this case, the background image can also be determined based on the image and the object class assignment.

[0050] A computing unit according to the invention, e.g. a control device or a control unit of a mobile device, or a server or other computer, is configured, in particular in terms of programming, to carry out a method according to the invention.

[0051] The invention also relates to a mobile device configured to receive control information determined as described above, or having a computing unit according to the invention. The mobile device also has a drive system and a control or regulating unit for controlling according to the control information. The mobile device can, in particular, be one as described above, for example, a robotic lawnmower.

[0052] The implementation of a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, since this entails particularly low costs, in particular if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon, as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, and others. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).

[0053] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

[0054] The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

[0055] Short description of the drawings Fig. 1 schematically shows a mobile device in an environment for explaining the invention. Fig. 2 shows an image captured by an environment sensing means of a mobile device. Fig. 3 shows a preferred embodiment of a method according to the invention from the image in Fig. 2 generated result image. Fig. 4 shows a background image generated in a preferred embodiment of a method according to the invention. Fig. 5 shows schematically a sequence of a method according to the invention in a preferred embodiment. Embodiment(s) of the invention

[0056] In Fig. 1 schematically and by way of example, a mobile device 100 is shown in an environment 120, in particular a work area, to explain the invention. The mobile device 100 is, for example, a robotic lawnmower with a control or regulating unit 102 and a drive unit 104 (with wheels) for moving the robotic lawnmower 100, e.g., along a trajectory 130. Furthermore, the robotic lawnmower 100 has, for example, a sensor 106 embodied as a camera with a field of view (indicated by dashed lines). For better illustration, the field of view has been chosen to be relatively small here; in practice, however, the field of view can also be up to 360° (e.g., but at least at least 180° or at least 270°).

[0057] Furthermore, the robotic lawnmower 100 has a computing unit 108, e.g., a control unit, by means of which data can be exchanged with a higher-level system 110, e.g., via an indicated radio connection. In the system 110, e.g., trajectories (or general navigation information) can be determined, which are then transmitted to the system 108 in the robotic lawnmower 100, which the latter is then to follow. However, it can also be provided that a trajectory (or general navigation information) is determined in the system 108 itself or is received there in some other way. Instead of a trajectory, the system 108 can, for example, also receive control information that has been determined based on a trajectory and according to which the control or regulating unit 102 can move the robotic lawnmower 100 via the drive unit 104 to follow a trajectory. The trajectory 130 is indicated here only as an example.

[0058] The robotic lawnmower 100 is intended, for example, to move independently across the property or in the surrounding area 120 or to navigate there, for example, mowing a lawn 142. Furthermore, several different objects are shown in the surrounding area by way of example, namely a house 140, a tree 144, several weed patches 146, and a path 148. As already mentioned, these objects - as well as other features such as, in particular, the lawn 142 - can be detected by the mobile device 100 or its camera 106 (i.e., they are visible in images taken by the camera) and then taken into account during navigation, i.e., when determining navigation information.

[0059] In Fig. 2 shows an image 250 which is taken by an environment detection means such as the camera 106 of the robotic lawnmower 100 Fig. 1. It shows the various features or objects, the house 140, the lawn 142, the tree 144, the weeds 146 and the path 148 (from the perspective of the robotic lawnmower or its camera).

[0060] Here, it can also be seen that a large proportion of the objects in image 250, which can be assigned to the ground on which the robotic lawnmower is moving, are located approximately in the lower half or the lower third, or below a horizon, which is particularly due to the near-ground position of the robotic lawnmower 100 or its camera 106. This lower or near-ground area is designated by 252, while a middle area is designated by 256 and an upper area (here essentially the sky) is designated by 254.

[0061] In Fig. 3 shows a result image 350 which, for example, in the context of a method according to the invention in a preferred embodiment is generated from the image 250 of Fig. 2 has been generated.

[0062] Here, features are extracted from the image 250, e.g. for each pixel, on the basis of which features are then assigned to at least one of several object classes, for example. The object classes "houses" OC1, "trees" OC2, "paths" OC3, "lawn" OC4 and "weeds" OC5 are to be used as examples. In the resulting image, these object classes are shown with different shaded areas. As already mentioned, instead of directly assigning an object class, one or more probability values ​​can be determined and / or assigned for the segments, which indicate the probability with which a segment can be assigned to a certain object class. Based on this, the object class with the highest probability value can then be selected for each segment.

[0063] The result image 350 shown here can in particular also be understood as a combination of the image 250 with the assigned object classes, ie the object class assignment.

[0064] It should be noted here that, for illustrative purposes, only the relevant (and shown) objects themselves are represented according to the classification. Depending on the implementation, other areas of image 250 may also be classified accordingly.

[0065] As mentioned, during semantic segmentation, the entire image 250 can be processed uniformly, in particular during the extraction of features. Preferably, however, a region of the image 250 can be selected in which, relative to a portion of the selected region in the (entire) image 250, a higher proportion of the computing power of the executing processing unit, e.g., the control unit 108, is available. Fig. 1, than for a remainder of the image 250. An example of such a selected area is the aforementioned area 252; this is a lower, ground-level area of ​​the image 250.

[0066] The selection of this area 252 can be made, in particular, depending on a current position of the robotic lawnmower 100 within the environment 120, as well as due to the fact that its camera is located relatively close to the ground. For example, it can be taken into account that a large part of the image in the lower area will be grass, which is due to the particular viewing direction of the camera of a robotic lawnmower. Based on the robot configuration and the camera placement and orientation, it can be determined at least approximately where the horizon line (the horizontal boundary line between the areas 252 and 256) is. Fig. 2 or Fig. 3) in the image, assuming a flat floor. Therefore, the central part of the image will typically contain the more relevant features (static objects).

[0067] It can therefore be assumed that the objects relevant for the sub-area are located in the selected area 252, so that semantic segmentation with increased computing power compared to the rest is useful there; this allows the relevant objects to be precisely determined despite the overall low computing power.

[0068] In Fig. 4 shows a background image 450 which, in a preferred embodiment, can be determined or generated within the scope of a method according to the invention based on the image 250 and the object class assignment (or the result image 350).

[0069] Here, the object class assignment can be used as Fig. 3. For example, it can be assumed that the object classes "lawn", "weeds", and "paths" are to be assigned to the underground. Knowing the position and / or orientation of the camera (e.g., how far it is above the ground and its detection angle), a 2D image of the underground - the underground image - can then be determined. A underground class assignment can then be made there, namely by assigning one of two underground classes to the segments of the underground image. Examples of the two underground classes in Fig. 4 “Lawn” (or “grass”) GC1 and “not lawn” GC2 (or “not grass”).

[0070] In Fig. Figure 5 schematically shows a sequence of a method according to the invention in a preferred embodiment, in the form of a flow chart.

[0071] In a step 500, an image, such as image 250 from Fig. 2, and received, for example, by the computing unit. There, the image—as an input image—can be preprocessed, if necessary, using an image processing system to correct lens distortions, lighting conditions, perspective, etc. The (possibly preprocessed) image is then forwarded to an artificial neural network 502 (i.e., a machine learning algorithm) for processing.

[0072] In a step 504, segments in the image 250 are then assigned at least one of several object classes, as in Fig. 3 to obtain an object class mapping 506 - or a result image combining the image and the object class mapping.

[0073] In a step 508, based on the image 250 and the object class assignment 506, a background image such as the image 450 according to Fig. 4, which at least partially depicts a surface on which the mobile device is moving.

[0074] This background image 450 is then fed to a differentiation module 510; the differentiation module can be, in particular, a machine learning algorithm based, for example, on a convolutional neural network. There, in a step 512, segments, in particular pixels, of the background image 450 are each assigned one of two background classes to obtain a background class assignment 514, which indicates a type of background, such as, for example, also with respect to Fig. 4 explained.

[0075] Furthermore, in step 516, an environment map 518 can be provided, to which one (or more) underground images and the one (or more) associated underground class assignments 514 are then assigned in step 520. A subsurface environment map 522 is obtained.

[0076] Based on the underground environment map 522, a coverage value 526 can then be determined in step 524 with respect to a proportion of mowed or unmowed grass in the environment, for which information can then be provided. For example, based on the information about the coverage value, control information for moving the mobile device can also be determined in such a way that the coverage value is increased or decreased. The mobile device or the robotic lawnmower can then be moved accordingly.

[0077] The determination of the result image or the object class assignment 506 is described above with reference to Fig. 5 has only been briefly explained. This will be explained in more detail below using a possible embodiment. An exemplary sequence of steps that can be performed is shown.

[0078] In step 500, the image, such as image 250 from Fig. 2, and received, for example, by the computing unit. There, the image—as an input image—can be preprocessed, if necessary, using an image processing system to correct lens distortions, lighting conditions, perspective, etc. The (possibly preprocessed) image is then forwarded to the artificial neural network 502 (i.e., a machine learning algorithm) for processing.

[0079] In a further step, for example, dimensionality reduction is performed. Since the calculation cannot (or should not) be performed for every pixel of the image due to the size (amount of data), the image dimension is reduced using convolution, pooling, batch normalization, and activation functions, as explained above.

[0080] Then, in a further step, features of the image are extracted, e.g. based on several factors such as shape, color, context, pattern, lighting variance, image context. By definition, a feature is, e.g., an individually measurable property or a characteristic of a phenomenon. In the context of a neural network, this would mean, e.g., training specific neurons to focus on certain properties of the image; some neurons could focus on color, pattern, texture, etc. Feature extraction should be understood in particular as meaning that in an image, certain regions activate certain neurons, and their value is passed on to the network for analysis, thresholding, and ultimately segmentation into the specific class.

[0081] In a subsequent step, some of these features are stored for easy and fast upscaling later. This means they help scale the result to the size of the original input image without losing resolution. The network also learns the stored values; this is more efficient than interpolating values ​​or nearest-neighbor interpolation because context information is added. For example, to upscale a square, typically only four points need to be used; to upscale a circle, at least in image processing, significantly more points would be needed.

[0082] In a further step, context-aware pooling, batch normalization, activation functions, and other techniques can be performed. "Context-aware" means that the network extracts contextual information, e.g., "This image area looks like a door, a window, and a wall. This entire area is therefore most likely a house." If something is then found in this "house" that most likely appears to be sky, the information can be discarded because it doesn't fit the context. This facilitates or leads to the failure to detect "grass" in the center of the sky, since this is impossible with context-awareness.

[0083] In a further step, an area of ​​the image (e.g. area 252 from Fig. 3) selected; this area can be determined or calculated using, for example, a method that depends on the physical characteristics of the robotic lawnmower, the placement of the camera and its orientation with respect to the ground surface.

[0084] In a further step, the heavier convolutions or a larger set of operations using the neural network are then concentrated on the selected area, while (only) a smaller set of operations is applied to the rest of the image.

[0085] The resulting feature map is then classified, in step 504, into its corresponding object classes, as in Fig.3 – this is where the object classes are assigned. The results of the calculation for the rest of the image and the focused calculation in the selected area of ​​the image are mapped and concatenated to form a resulting classified matrix. The classified matrix (i.e., an intermediate result) is then upscaled in a decoding phase using the residuals (the stored features or values) from the encoding phase to obtain the resulting image as output, which is similar in size to the input image.

[0086] The resulting image can then be mapped and, if necessary, further processed (e.g., indexing of classes, separation of output classes, threshold values) and passed on to potential applications.

[0087] Different arrangements or a different order of this approach or steps can be chosen; for example, it has been shown that performing region of interest selection after dimensionality reduction can be performed with little impact on accuracy and can even achieve better inference times. Context-aware feature selection can also be performed both on the entire image or (only) on the selected region, depending, for example, on the application and how much focus should be placed on the selected region.

Claims

[1] Method for determining a type of subsurface in an environment in which a mobile device (100), in particular a robotic lawnmower, moves autonomously, using at least one machine learning algorithm, comprising performing a subsurface determination, comprising: Providing (500) an image (250) of an environment (120) in which the mobile device moves, Assigning (504), in particular within the framework of a semantic segmentation, at least one of a plurality of object classes (OC1, OC2, OC3, OC4, OC5) to segments, in particular pixels, of the image (250) in order to obtain an object class assignment (506); Determining (508), based on the image (250) of the environment and the object class assignment (506), a background image (450) that at least partially depicts a background on which the mobile device is moving, in particular by a perspective projection into a plane; Assigning (512) to segments, in particular pixels, of the background image, one of two background classes (GC1, GC2) in each case in order to obtain a background class assignment (514) which indicates a type of background; and Providing the subsurface image and subsurface class assignment. [2] The method according to claim 1, wherein the background determination is performed repeatedly for different images of the environment, further comprising: Providing (516) an environment map (518) of the environment, which includes in particular boundaries within which the mobile device is to move; and Assigning (520) the subsurface images and the subsurface class assignments to the environment map to obtain a subsurface environment map. [3] The method of claim 1 or 2, wherein the autonomous mobile device is a robotic lawnmower, and wherein the ground classes are one of the following pairs of ground classes: grass and non-grass, or mown grass and unmown grass. [4] The method of claim 2 and 3, further comprising, when the ground classes are the pair of ground classes mown grass and unmown grass: Determining, based on the subsurface environment map, a coverage value in relation to a proportion of mown or unmown grass in the environment, in particular within the boundaries, and Providing information about the coverage value. [5] The method of claim 4, further comprising: Determining, based on the information about the coverage value, control information for moving the mobile device such that the coverage value is increased or decreased; and Providing the control information and / or moving the mobile device based on the control information [6] The method of claim 2 and 3, further comprising, when the background classes are the pair of background classes grass and non-grass: Determining, based on the subsurface environment map, a health value in relation to a proportion of grass or non-grass in the environment, particularly within the boundaries, and Providing information about health value. [7] A method according to any one of the preceding claims, wherein determining the background image comprises: Determining, based on the image of the environment and the object class assignment, a region of the image of the environment that corresponds to one of one or more selected classes of the plurality of object classes, wherein the one or more selected classes are in particular assignable to a background; and Creating the background image from the area of the image, particularly by perspective projection into a plane. [8] Method according to one of the preceding claims, wherein the assignment of the object class to the segments is carried out only for a selected area of the image or for the selected area of the image with higher priority than for a remainder of the image. [9] Method according to claim 8, wherein the selected area in the image is selected depending on a position of an environment detection means of the mobile device (100) with which the image was captured, in particular with respect to a plane on which the device (100) is moving, and / or wherein the selected area in the image is selected depending on a current position of the device (100) within the environment. [10] Method according to one of the preceding claims, wherein a first machine learning algorithm is used for assigning the respective at least one of a plurality of object classes, and wherein a second machine learning algorithm is used for assigning the respective one of two underground classes. [11] Method according to one of the preceding claims, wherein the images are captured by means of an environment detection means, in particular an image sensor, of the mobile device. [12] Computing unit (108) comprising a processor configured to carry out the method according to any one of the preceding claims. [13] Mobile device (100) designed to move autonomously, in particular a robotic lawnmower, comprising an image sensor for capturing images of an environment in which the mobile device is to move, which is arranged to receive control information which has been determined by means of a method according to claim 5, and / or with a computing unit (108) according to claim 10, and with a drive system and a control or regulating unit for controlling the drive system, in particular the control information. [14] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to claims 1 to 11. [15] A computer-readable data carrier on which the computer program according to claim 14 is stored.

Images