Method for determining boundary of working area of mobile device

Abstract

The invention relates to a method for determining a boundary (223, 233) which at least partially delimits a working area (222) of a mobile device (200), in particular an at least partially automatically moving vehicle or robot, in particular a mowing robot, the method comprises: providing, in a first step, environment information of an environment (220) in which the mobile device moves or should move, obtained at or at an externally controlled movement of the mobile device; determining a boundary of a working area within the environment for the mobile device based on the environment information from the first step (223); in a second step, environmental information, in particular an image, of the environment at least in the region of the boundary (223) is provided, which is obtained when or when the mobile device is moved again along and / or within the boundary, and the boundary (223) is confirmed and / or adapted on the basis of the environmental information from the second step.

Description

Technical Field

[0001] The invention relates to a method for determining a boundary, a computing unit and a computer program for executing the method, and a mobile device, wherein the boundary at least partially delimits a working area of ​​the mobile device, in particular an at least partially autonomous vehicle or a robot, in particular a lawnmower robot. Background Art

[0002] Mobile or working devices, such as at least partially autonomous vehicles or robots, usually move in an environment, in particular in an environment to be processed or a work area, such as an apartment, a garden, a factory hall or a street, in the air or in water. Summary of the invention

[0003] According to the invention, a method for determining the boundaries of an operating area of ​​a mobile device, a computing unit and a computer program for carrying out the method, and a mobile device are proposed with the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The present invention generally relates to a mobile device that moves in an environment or moves there, for example, in a work area or at least can move. Thus, it can also be referred to as a mobile working device. Examples of such mobile devices (or mobile working devices) are, for example, robots and / or drones and / or partially automated or (fully) automated (on land, in water or in the air) moving vehicles. As robots, for example, household robots, such as cleaning robots (for example in the form of vacuuming and / or mopping robots), floor or street cleaning equipment, construction robots or mowing robots are considered, but there are also other so-called service robots, as at least partially automated moving vehicles, for example, passenger transport vehicles or cargo transport vehicles (also called ground transport vehicles, for example in warehouses), but also flying vehicles, such as so-called drones or water vehicles are considered.

[0005] Such a mobile device has in particular a control or regulation unit and a drive unit for moving the mobile device so that the mobile device can be moved in an environment, in particular along a movement path. To this end, navigation information, i.e. specific instructions, such as in which direction the mobile device should travel in order to follow the movement path, can be determined based on the movement path. These specific instructions can then be implemented via the control or regulation unit and the drive unit.

[0006] For example, a map of the environment can be used for navigation, which can be obtained or determined in particular by means of SLAM. The following method in robotics is called SLAM (“Simultaneous Localization and Mapping”, i.e., simultaneous determination of position and mapping): in this method a mobile device, such as a robot, can or must simultaneously create a map of its environment and estimate its spatial position within the map. The method is thus used to detect obstacles and thus support autonomous navigation.

[0007] Furthermore, the mobile device or working device may have one or more sensors, by means of which the environment or information in the environment can be detected. This may be, for example, a camera, a lidar sensor or also an inertial sensor, by means of which the movement of the environment and / or the mobile device can be detected, for example, in two or three dimensions.

[0008] Furthermore, such a mobile device can be designed to receive data and / or send data by means of a communication connection, i.e. generally for communication or exchange of data. In this way, communication with the mobile device can, for example, give instructions to the mobile device, transmit other data to the mobile device or receive data or information from the mobile device. In this context, wireless communication connections are considered in particular. For this purpose, the mobile device can, for example, have corresponding (possibly different) modules for wireless communication, but the modules can also be integrated into the computing unit, for example.

[0009] In the case of such mobile devices, it is also common that there are functions that the mobile device can perform, ie the mobile device is designed to perform one or more, preferably different functions. Instead of functions, applications can also be mentioned. Such functions can be, for example, working functions or teaching functions.

[0010] The working function includes, for example, the mobile device automatically moving in the environment and performing a working process in the environment, performing a working process during the movement in the environment at least temporarily. Thus, in the case of a lawn mower robot, the working function may include, for example, mowing the lawn (during movement). To this end, a start command may be issued to the mobile device to perform the working function, for example, by sending corresponding data via a wireless communication connection.

[0011] The teach-in function comprises, for example, manually controlled movement of the mobile device in the environment, in particular when no working process is being performed in the environment. Thus, in the case of a lawnmower robot, the teach-in function can comprise, for example, (manual) movement or control along the boundaries of the working area in order to teach the lawnmower robot the boundaries. For this purpose, control commands can be issued, for example (continuously), to the mobile device to navigate or move in the environment, for example by sending corresponding data via a wireless communication connection. In this context, we also speak of teaching or so-called "Teach-In".

[0012] One aspect of the teaching is that a boundary (or limit) around a working area, such as a garden, should be found or preset. Then, in the working function, the boundary can be used to limit the automatic movement of the mobile device.

[0013] For the mobile device, the possibility is provided of defining boundaries which at least partially delimit the working range of the mobile device.

[0014] For this purpose, in a first step, environmental information of the environment in which the mobile device moves or should move is provided. The environmental information can be, for example, a position, an orientation, etc. that can be used for SLAM. However, in particular, the environmental information can also include, for example, an image or an image of the environment detected or already detected by a camera of the mobile device. Here, when the mobile device is moved in a controlled manner from the outside, more precisely, in particular when the mobile device is moved manually, as mentioned above with respect to the teaching function, the environmental information (in the first step) is obtained. Here, the mobile device can be controlled, for example, via a smartphone and a Bluetooth connection. Here, for example, the user can control the mowing robot at least approximately or as well as possible along the edge of the desired working area and follow the mowing robot here. This movement of the mobile device for the first step can be referred to as the first movement.

[0015] Then, based on the environment information from the first step, the boundaries of the working area of ​​the mobile device within the environment are determined. Here, this can be a temporary or rough boundary, but in principle the boundary can be used for the working function. Since the boundary is based on manual control of the mobile device, it happens that the boundary is not very accurate.

[0016] Here, the environmental information from the first step can be obtained in particular by means of SLAM, in particular visual SLAM. For this purpose, for example, the detected image can be processed accordingly. Preferably, loop closure is used to determine and / or refine the boundary based on the environmental information from the first step.

[0017] The so-called loop closure is an aspect related to SLAM. Here, it is based on the following: the mobile device reaches a specific position in the environment where it has appeared once before during its movement in the environment or work area. Based on environmental information such as images or, for example, spacing information observed from the corresponding position of the mobile device, and by comparing this environmental information with the previous environmental information, it can be determined whether the mobile device has been in a position once before or the mobile device has been in a position once before. If this is confirmed, the map created within the scope of SLAM or the route of the mobile device marked as passed must also indicate that the mobile device is in the same position again. However, due to possible deviations, this is often not the case. More precisely, in the created map, the current position will be different from the previous position. Now, the map and then the boundary can be adapted here so that two positions separated from each other are consistent again. Here, it is called the so-called loop closure.

[0018] In this context, it is also particularly expedient if the externally controlled movement of the mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a predefined tolerance. For example, the starting point and the end point can be located at a docking station. This allows loop closure to be achieved and the boundaries to be particularly well determined in a first step.

[0019] Then, in a second step, environmental information, in particular an image of the environment, is provided at least in the border area. In this case, the environmental information is obtained when the mobile device is re-moved along the border and / or within the border. The re-moval is in particular an automatic movement, i.e., for example within the scope of the mentioned working functions, i.e., processing can be performed here. For example, this can include a first mowing of the (possibly entire) working area. Such a (re-)movement of the mobile device for the second step can also be referred to as a further movement.

[0020] Then, based on the environmental information from the second step, the boundary (obtained in the first step) is confirmed and / or adapted. If the boundary is confirmed, the boundary obtained in the first step can be used further, which can be called verification. If the boundary is adapted, the boundary obtained in the first step can be adapted, for example, based on the environmental information obtained in the second step. Thus, for example, information can be obtained based on the environmental information (preferably an image) from the second step that a lawn is still present at a specific location outside the boundary from the first step, so that the boundary can be adapted accordingly there. It goes without saying that part of the boundary can be confirmed and other parts of the boundary can be adapted.

[0021] As mentioned, the environmental information from the second step may include an image of the environment. In one embodiment, confirming and / or adapting the boundary then includes determining a map of at least a portion of the environment including the boundary based on the image from the second step. For example, this can be done using a positive projection.

[0022] In this way, in the first step, no or at most a small number of images must be detected and stored in order to subsequently create a map. More precisely, the boundaries are determined only via (visual) SLAM. This saves computing resources. The environmental information, in particular the images, from the second step can then be used for verification or adaptation. Since the second step can be carried out within the scope of the working process, the mobile device with the actual working function can be used more quickly overall than, for example, an additional step would be used for verification (before the working function).

[0023] However, as an alternative to these two steps, it is also possible to provide environmental information and images of the environment in which the mobile device moves or is to move, which are obtained or acquired when the mobile device moves in an externally controlled manner. Based on the environmental information and the images, the boundaries of the working area in the environment are then determined for the mobile device. In this case, the image is therefore also detected and, if necessary, stored and processed. This allows particularly precise boundaries to be determined in only one movement of the mobile device, although more computing resources are required in this case. Compared to the above-described two-step process, only the first step is required here, but the image is also detected there.

[0024] In one embodiment, in this case, environmental information of the environment at least in the region of the border, in particular an image of the environment, can also be provided, which environmental information is obtained when the mobile device is moved further (in particular automatically) along and / or within the border or is obtained during this movement. During the further movement, the mobile device in particular performs a processing process. This can itself correspond to the second step of the previously described variant. With the help of the environmental information or image, it is then not necessary to confirm and / or adapt the border (although this may be the case). More precisely, a map of the environment can be determined thereby, which map includes at least a part of the border; as mentioned above, this can be done, for example, using an orthographic projection.

[0025] As has already been shown, a particularly precise boundary can also be obtained by targeted movement of the mobile device, specifically depending on whether the boundary is determined using two steps or only one step, for example.

[0026] Here, environmental information and / or images of the environment in which the mobile device moves or is to move are provided. Here, the environmental information and / or images are obtained when the mobile device moves: the movement can be external, manual or automatic.

[0027] However, the movement of the mobile device here includes at least one movement section in which the mobile device is moved at least partially outside the working area on a ground surface that is different from the ground surface of the working area (or the expected working area). This can be, for example, a path or a sidewalk, while the (expected) working area has a lawn as the ground surface. In other words, the mobile device is thus moved in a targeted manner outside the (actual or expected) working area, whereas this usually only takes place within the (actual or expected) working area.

[0028] Based on the environment information and / or the image, the boundaries of the working area within the environment are then determined for the mobile device. By means of this targeted movement of the mobile device outside the (actual or desired) working area, it is possible to detect the edges of the (desired) working area more accurately, so that the boundaries (e.g. preliminary or final boundaries) can also be determined significantly more accurately. It should be mentioned here that the boundaries do not have to be directly based on the movement of the mobile device, e.g. an externally controlled movement, but can be adapted accordingly if necessary in order to limit the boundaries to the (desired) working area.

[0029] It is expedient if the ground plane different from the working area ground plane is adjacent to the working area ground plane and / or has the same height within a predetermined tolerance. The height is in particular related to the direction of gravity. In this way, the mobile device can be moved without problems from one ground plane to another, or even simultaneously on two ground planes.

[0030] It should be mentioned that usually only part of the boundary can be determined in this way, since for example parts of the ground outside the (desired) working area may be inaccessible; for example, fences, walls or water may partially limit the (desired) working area.

[0031] As mentioned, the environment map can be determined using an orthographic projection, or more precisely a visual map. In this case, a visual map is to be understood in particular as a graphical representation of the environment, without a SLAM map with nodes and edges, etc. This can also be referred to as a structure map, which thus shows the structure of the environment, for example areas with lawn and areas without lawn.

[0032] For this purpose, an image of the environment is provided, wherein an image of the movement of the mobile device in the environment is obtained. The movement can be from the outside, manual or automatic. Then, an orthographic view of the image of the environment is determined. Based on the orthographic view of the image of the environment, a visual map of the environment is then provided; the visual map can then be used as explained above.

[0033] In one embodiment, a visual map of the environment is determined using a computer vision algorithm, such as in particular segmentation, for example semantic segmentation. By using a computer vision algorithm, the image can be preprocessed. In this case, in particular, continuous areas in the image can be determined, which is the case in particular when segmenting. In this case, the image of the environment can be segmented before determining the orthographic view, or generally processed with the aid of a computer vision algorithm to obtain a segmented image of the environment, or generally a preprocessed image. The orthographic view of the image of the environment is then determined based on the segmented image of the environment or the preprocessed image.

[0034] However, the orthographic view of the environment image can also be determined before segmentation or application of a computer vision algorithm. The orthographic view of the environment image is then segmented or preprocessed.

[0035] In one embodiment, the segmentation is performed with the aid of a machine learning algorithm, such as an artificial neural network. However, computer vision algorithms may also be used in a classical manner.

[0036] If the mobile device is a lawn mowing robot and the environment image includes an image of grass (or lawn), the boundaries of the grass can be determined when segmenting or using a computer vision algorithm.

[0037] In this way, the limits of the (desired) working area, i.e. for example the lawn, can therefore be determined particularly accurately, so that the visual map can represent the working area particularly accurately. In particular, however, boundaries can be determined particularly accurately hereby, i.e. for example via the boundary between grass and other ground surfaces. As described above, this allows a particularly accurate determination or adaptation of boundaries.

[0038] In this context, an orthographic projection of an image (or a preprocessed or segmented image) can be understood in particular as meaning that the image is projected onto the ground, i.e. from top to bottom. It should be borne in mind that the camera of a mobile device, such as a lawn mower, is usually aligned at least approximately horizontally. As a result, in an image captured or recorded with the aid of such a camera, the ground is represented distorted only at the lower edge. Taking into account camera parameters, such as the specific arrangement of the camera on the mobile device and the field of view, etc., the part of the image showing the ground can be corrected and represented in a top view. In this representation, i.e. in the orthographic projection, the specific and precise limits of the working area can be recognized significantly better.

[0039] In segmentation, in particular so-called semantic segmentation, one of a plurality of classes is associated with each image point (pixel) in the image, i.e. the image point is labeled or classified. For this purpose, firstly features of the image point are usually determined or extracted, and then the class is associated based on these. Such features can be, for example, shape, color, context, pattern, light changes and / or image context. Thus, for example, the color green is associated with an image point, and the surrounding image points can also all be green. This indicates a lawn.

[0040] In this case, different features can be considered as categories. Taking a lawnmower robot as an example, in a simple case, two categories can be used, namely "lawn" and "non-lawn", i.e. for each image point in the image, it is determined whether the lawn is shown. It goes without saying that more than two categories can also be used, for example in order to recognize paths, streets, buildings, vehicles or people and, if necessary, associate them with image points. It is also possible to consider - based on the category "non-lawn" - the category "background", to which everything that is not associated with other categories is associated with the category "non-lawn". Compared to pure object recognition, in which (only) objects in the image are recognized, more information is added by all these labels.

[0041] In principle, the classification does not have to be performed for each individual pixel, but can also be performed for segments or parts of the image, each of which includes a plurality of pixels. Such segments can be fixedly predefined, but can also be formed only within the scope of semantic segmentation, possibly also in different sizes within the image. For this purpose, one or more artificial neural networks or generally pattern recognition methods based on artificial intelligence are preferably used.

[0042] The segmented image thereby allows a simpler and faster determination of the position of the boundaries between different ground surfaces.

[0043] In one embodiment, a changed, in particular extended, boundary is also determined, more precisely based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. In this case, the changed boundary is determined in particular automatically. The changed boundary is then provided to the user, in particular as a map. For example, a suggestion of the changed boundary can thus be provided. For this purpose, the changed boundary can be displayed, for example, on a smartphone or other input device.

[0044] In one embodiment, it is provided that a modified, in particular extended, boundary is determined based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. In this case, the modified boundary is determined in particular automatically. In this case, the modified boundary is determined as the adapted boundary. Thus, in this case, the boundary can be improved automatically, in other words without the need for a user to make a change.

[0045] In particular, the changed boundaries can be determined regardless of how accurately the existing boundaries have been determined. For this purpose, existing boundaries of the working area within the environment in which the mobile device is moving or is to move can be provided. In this case, the existing boundaries have been determined at least based on environmental information of the environment that was obtained when the mobile device was moved for the first time.

[0046] Then, environmental information, in particular an image, of the environment at least in the area of ​​the existing border is provided. The environmental information is then obtained when the mobile device is moved again (in particular automatically) along the existing border and / or within the existing border. Then, the changed border is automatically determined, more precisely based on the existing border and the environmental information, in particular the image, obtained during the re-movement of the mobile device. In particular, the changed border is provided to the user, in particular as a map (for example, the mentioned visual map). For example, this can be done on a smartphone or other input or display device. Thus, the user can receive suggestions for improved borders.

[0047] The changed boundary can be, in particular, an expanded boundary or an improved boundary. It is thereby possible to obtain a better boundary particularly easily and quickly (because automatically), which allows a more precise coverage of the desired working area.

[0048] It can also be determined whether the deviation between the changed border and the existing border meets a preset criterion; for example, the average or cumulative distance between the two borders can be determined and compared with a threshold value. The changed border is only offered if the deviation meets the preset criterion. Thus, for example, a suggestion is only made to the user if this makes sense.

[0049] Furthermore, it can be provided that correction information is obtained from the outside. For this purpose, the user can, for example, make and transmit a change to the suggestion based on the suggestion. The adapted boundary is then determined based on the boundary (depending on what boundary exists, either the existing boundary and the proposed changed boundary) and based on the correction information.

[0050] In one embodiment, it is provided that the environment information from the first step includes information about reference points in the environment, in particular images. The information about the reference points can then be used in particular when determining the boundary and / or when determining the visual map. Such reference points can be, for example, checkpoints, in particular also only a few checkpoints, which are not along the entire route, but, for example, at specific intervals and / or in specific maneuvering situations (90° bends, etc.). This allows a more precise verification or adaptation of the boundary or also the map, while only a small number of images must be stored in the first step.

[0051] In addition to the above-mentioned possibilities for determining a boundary which at least partially delimits the working area (this relates in particular to an outer boundary), it is also possible to determine, starting from an existing boundary (which has then been determined, for example, as before), an additional boundary, in particular an inner boundary, i.e. a boundary enclosed by the existing boundary, for example. Such an additional boundary can, for example, exclude trees or other obstacles present in the lawn from the working area to be processed.

[0052] For this purpose, environmental information (e.g. position, orientation) and / or images of the environment in which the mobile device moves or is to move are provided. The environmental information or images are obtained when the mobile device moves. The movement of the mobile device here includes a movement starting from a starting point, wherein the starting point corresponds to a point in the environment at which the mobile device is stopped or has stopped at or along an existing boundary (this can be a boundary, as described above, whether it has been adapted or changed). The movement of the mobile device is particularly automatic here. The mobile device is stopped actively and / or manually, i.e. consciously.

[0053] Based on the environment information and / or the image, and if necessary also on the basis of existing boundaries, additional boundaries of the working area within the environment are then determined for the mobile device. In other words, the mobile device is stopped during its (automatic) movement along already existing boundaries or at least (temporarily) at such existing boundaries, and from there it is controlled, for example, around obstacles. In this case, the mobile device is at least partially controlled from the outside (for example manually), for example as described above for the first step. In this way, it is known for the additional boundaries how the additional boundaries are located relative to the existing boundaries.

[0054] Preferably, the movement of the mobile device also includes a movement to an end point, wherein the start point and the end point are identical at least within a predefined tolerance. This makes it possible to optimize the additional boundary, for example by means of loop closure.

[0055] Preferably, the movement from the starting point to the intermediate point comprises an externally controlled (e.g. manual) movement of the mobile device, while the movement from the intermediate point to the end point comprises, for example, an automatic movement of the mobile device. In particular, the intermediate point corresponds, at least within a predefined tolerance, to a point that the mobile device has passed during the movement from the starting point to the intermediate point. In this way, it is ensured that the mobile device completely avoids obstacles.

[0056] Furthermore, what has already been stated for the other boundaries, ie what can be determined, for example, by means of SLAM, then applies to the additional boundaries.

[0057] Navigation information for the mobile device can then be determined based on the boundaries (if necessary also additional and / or changed or adapted boundaries) and / or the (possibly visual) map of the environment. Based on the navigation information, control information for moving the mobile device can in turn be determined. The control information can then be provided and / or the mobile device can be moved based on the control information.

[0058] Particularly in terms of programming technology, a computing unit according to the invention (i.e. a system generally for data processing), for example a control device or control unit of a mobile device, or a central server or other computing system is designed to carry out the method according to the invention.

[0059] The invention also relates to a mobile device, which is designed to obtain control information that has been determined as explained above. Additionally or alternatively, the mobile device has a computing unit according to the invention. Furthermore, the mobile device has a control or regulation unit and a drive unit for moving the mobile device. Furthermore, the mobile device can have a detection device for detecting environmental information of the environment in which the mobile device moves or is to move, for example the mentioned camera or also a plurality of cameras.

[0060] Preferably, the mobile device is designed as an at least partially autonomous vehicle, in particular as a passenger or cargo transport vehicle, and / or as a robot, in particular as a household robot, such as a cleaning robot, a floor or street cleaning device or a lawn mowing robot, and / or as a drone.

[0061] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this results in particularly low costs, in particular if the execution control device is also used for other tasks and therefore already exists. Finally, a machine-readable storage medium is provided, which has a computer program as described above stored thereon. Suitable storage media or data carriers for providing the computer program are in particular magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. The program can also be downloaded via a computer network (Internet, intranet, etc.). This download can be done in a wired or cabled manner or wirelessly (for example, via a WLAN network, 3G, 4G, 5G or 6G connection, etc.).

[0062] Further advantages and embodiments of the invention are apparent from the description and the drawings.

[0063] The invention is schematically illustrated in the drawings based on exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1aA mobile device is schematically shown to explain the invention.

[0065] Figure 1b Schematically shows Figure 1a Another view of a mobile device in .

[0066] Figure 2a , 2b 2c schematically shows an environment with a mobile device to explain the present invention.

[0067] Figure 3a , 3b 3c and 3d schematically show schematic diagrams for explaining embodiments of the present invention.

[0068] Figure 4a , 4b A map on an input device is schematically shown.

[0069] Figure 5 The flow of a method for explaining an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0070] exist Figure 1a In the figure, a mobile device, in particular a working device, is shown schematically and exemplarily to explain the present invention. Figure 1b In the figure, the mobile device 100 is shown in different views and in different aspects. Figure 1a and 1b .

[0071] The mobile working device 100 is, for example, a mowing robot having a control or regulating unit 102 and a drive unit 104 (with wheels) for navigating or moving the mowing robot 100 in an environment 120 and there, in particular in or on a working area 122, such as a lawn or a garden. The mowing robot 100 can move or be moved, for example, along a movement path or trajectory 130. In addition, the mowing robot 100 has, for example, a sensor 106 designed as a camera. With the aid of the camera 106, images of the environment can be captured, which in turn can be used for navigation. In addition, a docking station 110 is provided, for example, at which the mowing robot can be charged, for example.

[0072] exist Figure 1b As examples, buildings 124, trees 126, and people 128 are also indicated in the environment, which may be considered as objects or obstacles when navigating the lawn mower robot.

[0073] Furthermore, the mowing robot 100 has a computing unit 108, for example a control device, by means of which data can be received and / or sent. As already mentioned, this can be done, for example, via various types of wireless communication connections. These types of wireless communication connections are Figure 1a 170, 172, 174. For example, they are a mobile radio connection 170, a Bluetooth connection 172 and a WLAN connection or WiFi connection 174. For this purpose, the computing unit 108 can, for example, have a corresponding radio module or be connected to such a radio module, which is then part of the mowing robot 100.

[0074] In addition, Figure 1a In the embodiment of the present invention, the mobile input device 140 is, for example, a smartphone, a central computing system 150 (or a server, which can represent a so-called cloud) and a WLAN router 160 of a building 162 (which can be a different building from the building 124 or can also be the same building). Wireless communication connections 170, 172, 174 of the type already mentioned are provided between the mower robot 100 or its computing unit 108, the mobile input device 140, the computing system 150 and the WLAN router 160. It should be noted that the mobile radio connection 170 is established via a mobile radio transmitter 152, which in turn is connected to the computing system 150.

[0075] The Bluetooth connection 172 is provided between the mowing robot 100 or its computing unit 108 and the mobile input device 140, ie a Bluetooth connection can be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140. It goes without saying that the mobile input device 140 has a suitable radio module for this purpose.

[0076] The WLAN connection 174 is provided between the mowing robot 100 or its computing unit 108 and a WLAN router, i.e. a WLAN connection can be established between the mowing robot 100 or its computing unit 108 and a WLAN router 160. It goes without saying that the WLAN router 160 has a suitable radio module for this purpose. The WLAN router 160 can in turn be connected to the Internet via, for example, a wired connection.

[0077] A mobile radio connection 170 is provided between the mowing robot 100 or its computing unit 108 and the computing system 150, i.e. a mobile radio connection can be established between the mowing robot 100 or its computing unit 108 and the computing system 150. In this connection, it should be mentioned that such a mobile radio connection 170 is established by the mowing robot 100 or its computing unit 108 or a radio module therein, for example, with a mobile radio transmitter 152 or a corresponding mobile radio station, to which the computing system 150 is in turn connected, for example, via mobile radio and / or by wire.

[0078] Furthermore, a mobile radio connection 170 is provided between the mobile input device 140 and the computing system 150, i.e. a mobile radio connection can be established between the mobile input device 140 and the computing system 150. It also applies here that such a mobile radio connection 170 is established by the mobile input device 140 or a radio module therein, for example, with a mobile radio transmitter or a corresponding mobile radio station, to which the computing system 150 is in turn connected. It should be mentioned that the mobile input device 140 includes a radio module for WLAN for this purpose, which is usually the case in the case of typical smartphones.

[0079] Likewise, a WLAN connection 174 may be provided between the mobile input device 140 and the computing system 150, i.e., a WLAN connection may be established between the mobile input device 140 and the computing system 150. For this purpose, a WLAN connection 174 may be provided between the mobile input device 140 and a WLAN router. As already mentioned, the WLAN router 160 may in turn be connected to the Internet, for example, via a wired connection, and via this to the computing system 150.

[0080] Thus, the mower robot 100 or its computing unit 108 can thereby send and receive data, i.e. exchange data, via each of the wireless communication connections 170, 172, 174 mentioned. In this case, data can be exchanged not only via a Bluetooth connection but also via a mobile radio connection 170 or a WLAN connection 174, i.e. indirectly via the computing system 150, by means of the mobile input device 140. It should be mentioned that in principle a WLAN connection can also be established between the mower robot 100 or its computing unit 108 and the mobile input device 140.

[0081] The mobile working device or lawn mower robot 100 may be designed to perform one or more functions. In one embodiment, the functions include a working function, such as a lawn mowing function and a teaching function.

[0082] The working functions include, for example, the automatic movement of the lawn mower robot 100 in the environment 120 and the execution of working processes (or processing processes) in the environment, such as mowing the grass at least temporarily during the movement in the environment (depending on the situation, it is also possible that the lawn mower robot first travels to a specific location without mowing the grass in order to start mowing or continue mowing the grass there).

[0083] The mobile radio connection 170 can be associated with the working function. This means that the mower robot 100 or its computing unit 108 obtains and / or sends data required for performing the working function at least partially via the mobile radio connection. This can include, for example, a start command for starting a mowing process, which is sent to the mower robot 100 from the mobile input device (optionally via the computing system 150).

[0084] This can also include, for example, obtaining a trajectory (or navigation information in general) from the computing system 150, which trajectory (or navigation information in general) is determined in the computing system 150 and which the mowing robot should then follow for the mowing process. However, it can also be provided that the trajectory (or navigation information in general) is determined in the mowing robot 100 or its computing unit 108 itself or is obtained there in another way. However, instead of the trajectory, the computing unit 108 can also (only) obtain control information that has been determined based on the trajectory (which in turn can already be determined in the computer system), and the control or regulating unit 102 can move the mowing robot 100 via the drive unit 104 based on this control information in order to follow a trajectory, such as the trajectory 130 indicated here only as an example.

[0085] The mowing robot 100 can then move independently on the working area 122 or in the environment 120 or navigate there and mow the grass in this process. In this case, as already mentioned, various objects can be detected by the mowing robot 100 or its camera 106 (i.e., the objects are visible in the images recorded by means of the camera) and then be taken into account during navigation, i.e., when determining navigation information. For this purpose, the images detected in this case or the corresponding data can be transmitted to the computing system 150, for example, via the mobile radio connection 170.

[0086] The teach-in function includes, for example, manually controlled movements of the mowing robot in the environment, in particular movements along the boundaries of the working area 122, in order to teach the mowing robot the boundaries. In this case, the working function, i.e. mowing, does not have to be used (or should not be used). For this purpose, control commands (or steering commands) for navigation or movement can be (continuously) transmitted to the mowing robot 100, for example. In this case, such control commands or corresponding data are sent from the mobile input device 140 to the mowing robot via the Bluetooth connection 172. Likewise, the mowing robot 100 or its computing unit 108 can send data back to the mobile input device 140 via the Bluetooth connection. In this case, it is also called teaching or so-called "Teach-In".

[0087] For the purpose of explaining the present invention, Figure 2a 2 shows an environment 220 with a building 224 and a tree 226, similar to Figure 1a , 1b In addition, the desired working area, for example a lawn to be mowed, is shown by way of example with 221. By way of example, the mobile device or lawnmower robot 200 and the docking station 210 are shown again.

[0088] It is therefore generally desirable that the desired working area 221 is covered completely and as accurately as possible by the mobile device during processing, so that the entire lawn is mowed as completely as possible.

[0089] To this end, now need to determine the (actual) working area of ​​the boundary, then can be processed by the mobile device to the working area.In other words, must determine the boundary or limit, the mobile device should move within the boundary or limit and should be processed within the boundary or limit.

[0090] Within the scope of the aforementioned teach-in function, mobile device 200 can now be controlled by the user, for example, by means of a smartphone from docking station 210 along the edge of desired working area 221 until mobile device 200 reaches docking station 210 again.

[0091] Based on the movement and the environmental information (eg images) obtained in this process, a boundary 223 of the (actual) working area 222 can be determined in a first step, as shown in FIG2 . It can be seen here that the boundary 223 differs from the actual edge of the desired working area 221 .

[0092] This can be caused in particular by the user moving the mobile device 200 too far from the actual edge of the desired working area 221. This is shown, for example, at the bottom left and top right in FIG. 2, where the boundary 223 is particularly far from the actual edge of the desired working area 221.

[0093] At this point it should be mentioned that when the mobile device 200 is moved, the position of the mobile device may be determined relative to a center point, wherein the geometric dimensions of the mobile device or its handling tool must be taken into account for the boundaries.

[0094] However, now, in a second step, environmental information of the environment, in particular an image of the environment, can be obtained again, more precisely when the mobile device 200 is moved again along the boundary 223 and / or within the boundary 223. This can be the case, for example, within the scope of the mentioned working function, in which the mobile device is moved according to the trajectory 330.

[0095] Based on the environmental information or images obtained here, the boundary 223 can then be determined and / or adapted. This is for example Figure 2a The lower left and upper right center areas show that, as mentioned, the boundary 223 is particularly far from the actual edge of the desired working area 221. The adapted boundary is designated as 233 there.

[0096] If the mobile device 200 moves, for example, along the trajectory 330 , an image of the lower left region is also detected. By analyzing the image, it can be recognized that the actual edge of the desired working area 221 deviates from the boundary 223 .

[0097] Based on the image, a map of the environment including at least a portion of the boundary may also be determined.

[0098] As already mentioned, instead of these two steps, only the step of externally controlling the mobile device can also be used, but in this case also an image of the surroundings is subsequently acquired.

[0099] exist Figure 2b In the Figure 2a 231, along which the mobile device 200 moves, for example, in a manual movement controlled from the outside. It can be seen here that the mobile device comprises a movement section or movement path section 232, in which the mobile device moves at least partially outside the (desired) working area 221, more precisely on a ground surface 221b that is different from the ground surface 221a of the working area 221. For example, the ground surface 221a can be a lawn, while the ground surface 221b can be a path or a sidewalk.

[0100] By means of this movement, the boundary denoted here by 223' can be determined more precisely, i.e. without the movement outside the working area 221. It should be mentioned at this point that the boundary 223' is shown here slightly spaced apart from the edge of the (desired) working area 221, although this can be significantly better matched in practice.

[0101] exist Figure 2c In the Figure 2a The part of the environment 220 in the embodiment. The determination of the additional boundary should now be explained here. Here, the boundary 223 determined as explained above, for example, can be assumed as an existing boundary.

[0102] When the mobile device 200 moves along the boundary 223, more precisely, in particular when moving automatically, it can be stopped, more precisely, for example, at point P1. From there, that is, from point P1 as the starting point, the mobile device 200 can then be manually controlled around the tree 226, the obstacle, for example via the mentioned Bluetooth connection to the smartphone; this is shown with the help of routes 235 and 236. Expediently, the manual movement is carried out to point P3, that is, the intermediate point, which is located at least within a preset tolerance on the point that has been traveled before (here point P2) (i.e. on the path 235, 236). From there, the mobile device can then, for example, automatically travel again to the end point, here point P4, which coincides with the starting point at least within a preset tolerance. This is shown with the help of route 237.

[0103] In this way, an additional boundary can be determined, more precisely in particular as the portion of the route between points P2 and P3; in this case, it is an at least approximately closed loop; Figure 2c , the additional boundary corresponds to route 236 .

[0104] Therefore, the boundary 223 or 233 is in particular an outer boundary, while the boundary 236 can be referred to as an inner boundary.

[0105] exist Figure 3a , 3b 3c and 3d schematically show examples for explaining the application of the mentioned forward projection and segmented images as computer vision algorithms, as they can be used in embodiments of the present invention.

[0106] exist Figure 3a 300a is shown in FIG. 300a , which may be detected by a mobile device or its camera. For example, image 300a may be approximately Figure 2a 2b. Here, in particular, a building 324 is visible in the front view, and a piece of the desired working area 322 is visible at the lower edge, which has, for example, a lawn as ground surface 321a. A path with a different ground surface 321b runs beside the working area 322.

[0107] As can be seen from image 300 a , it would be difficult to determine directly therefrom a boundary which should correspond as precisely as possible to the desired working area 322 or its edge.

[0108] exist Figure 3b, which is an orthographic projection of image 300a. Here, it can be seen that a plan view (or a top-down view) of a desired working area 322 with a lawn as ground surface 321a and a path with a different ground surface 321b is shown. There, the geometrical relationships and the size or course of the edge of the desired working area 322 (i.e., the line denoted here by 322) at least approximately correspond to the actual conditions.

[0109] exist Figure 3c 300c is shown, which is a semantic segmentation of image 300a. Here, only two classes are distinguished, namely, for example, lawn 321a and background or the rest, which is not represented here. For example, buildings, paths and all other objects except lawns fall into this. It can be seen here:

[0110] exist Figure 3d 300d corresponding to image 300a with semantic segmentation and orthographic projection is shown. This can be obtained, for example, from image 300b by semantic segmentation or from image 300c by orthographic projection. The results are at least approximately the same. It goes without saying: different nuances will occur.

[0111] In any case, the orthographic projection of the image, in particular the orthographic projection of the image with the (semantic) segmentation, allows: particularly accurate determination of the edge of the (desired) working area, whereby the boundary can be determined particularly accurately. Likewise, an accurate (visual) map of the environment can be determined thereby, in which at least a part of the boundary is shown, for example. The map can then be displayed to the user on a smartphone, for example.

[0112] exist Figure 4a An input device 440, such as a smart phone, is shown by way of example in FIG. Figure 1a In addition to controlling the mobile device during teaching, the visual map 420 can also be displayed on the input device 440, as shown in the example. Figure 4a For example, a grassy or lawn area 421a and an outside area 421b are then visible there. Furthermore, a line 423 is shown in the map, which line can correspond, for example, to the boundary created in the first step mentioned above; this can correspond, for example, to the boundary according to Figure 2a The boundary 223.

[0113] Furthermore, a line 433 is shown, which can represent a changed, in particular extended and / or improved boundary; this can correspond, for example, to Figure 2a As mentioned, such a changed or extended boundary can in particular be determined automatically and then displayed as a suggestion to the user on the map, as in this case in Figure 4aShown in.

[0114] The user may then confirm, for example via a suitable input command at input device 440, that the user agrees with the suggestion of boundary 433. Likewise, the user may reject the suggestion of boundary 433. Boundary 423 may then, for example, be retained.

[0115] For example, users can also propose changes. Figure 4b , more precisely by means of a line 434 of a changed boundary. To this end, the user can move line 433 (for example at a suitable point of the line) by means of a touch display of input device 440, for example, so that this produces line 424.

[0116] exist Figure 5 The flow of the method for determining a boundary for a mobile device, for example according to Figure 1a , Figure 1b or Figure 2a , Figure 2b , Figure 2c The working area of ​​the lawn mower robot 100 is at least partially delimited. Below, the process should also be explained in more detail with reference to the figures explained above.

[0117] To this end, in step 500, i.e. the first step, environmental information 502 of the environment in which the mobile device moves or should move is provided. The environmental information may include, for example, position, orientation, and may, for example, have been determined based on images detected by means of a camera of the mobile device. For example, this may be done by means of SLAM, in particular visual SLAM. In this case, the mobile device is already controlled from the outside, as described above, for example with reference to Figure 2a In this case, the mobile device can in particular also be moved outside the desired working area on other surfaces, such as for example with reference to Figure 2b As explained.

[0118] Then, in step 504, based on the environment information 502 from the first step, the boundaries of the working area within the environment are determined for the mobile device, such as for example referring to Figure 2a This can be done, for example, using loop closure, in particular if the start and end point are identical or close to each other in a movement controlled from the outside, for example at a docking station. This also provides a boundary.

[0119] In step 506, i.e. in a second step, environmental information 508, such as an image, of the environment at least in the region of the border is then provided. The environmental information or image 508 is obtained when the mobile device moves along the border and / or within the border, as described above, for example with reference to Figure 2a As explained.

[0120] In step 510, a map of the environment, in particular a visual map, may be determined based on or from the image 508. This may be done using forward projection and segmentation. An example of such an image is Figure 3a An example of such a map is Figure 4a Map 420 shown in .

[0121] In this case, in step 512, an orthographic view 514 of an image of the environment can be determined from the image 508, as shown, for example, with the aid of the image 300b in FIG. 3. In step 516, the orthographic image 514 or a view thereof can then be segmented, as shown, for example, with the aid of Figure 3d As shown in the image 300d, a map can be created and provided.

[0122] Then, in step 518, based on the map or also only on the segmented orthographic projection image, the existing boundary (from the first step) can be confirmed (or verified). However, if necessary, the existing boundary can be adapted, i.e. step 520. Then, in step 522, the boundary obtained here (i.e., for example confirmed or adapted) can be provided, for example, for further use.

[0123] However, it can also be provided that, in step 500', the environment information 502 and the image 508 obtained from the externally controlled movement are provided. In step 510', a map, in particular a visual map, can be determined based on or according to the image 508, as also described above. In particular, steps 512 to 516 can also be provided here.

[0124] Then, in step 518', a boundary may be determined based on the environment information and the image. Then, in step 522', the boundary may be provided, for example, for further use.

[0125] Starting from the boundaries already provided in step 522 or 522', or also starting from the boundaries already determined or provided in step 504, in step 524, a changed, in particular extended, boundary can be determined, more precisely based on the boundaries from the first step and / or the environmental information, in particular the image, from the second step, or generally from the existing boundaries, such as, for example, with reference to Figure 2a Likewise, environmental information, in particular images, can then be provided for further movement (eg during a processing procedure).

[0126] In this case, the determination of the changed boundary is particularly carried out automatically. Then, in step 526, the changed boundary is particularly provided to the user as a map, such as for example with reference to Figure 4a As explained.

[0127] The user can then, for example, perform changes or corrections, such as reference Figure 4b Then, in step 528, correction information 530 is received from the outside (for example, from a smartphone). Then, in step 532, the boundaries confirmed in step 518 or obtained in step 518' can be adapted. However, it is also possible to re-adapt the map adapted according to step 520.

[0128] Likewise, in this way, a map adapted according to step 520 can be obtained, i.e., an adapted map is obtained only after a user correction. Thus, the adapted boundary is then determined based on the boundary and based on the correction information. Alternatively, however, the changed boundary can also be determined without correction information.

[0129] Based on the boundaries (e.g., first obtained, adapted, or changed boundaries), navigation information 536 for the mobile device can then be determined in step 534. Based thereon, control information 540 for moving the mobile device can then be determined in step 538 and can then be provided.

[0130] After obtaining a boundary, whether an initial boundary (see step 522 or 522 ′) or an adapted or changed boundary (see step 532 ), additional, for example inner boundaries can be determined starting from the boundary as existing boundaries.

[0131] To this end, in step 550, environmental information 552 (similar to environmental information 502, such as position, orientation) and / or an image 554 (similar to image 508) of an environment in which the mobile device moves or is to move is provided. The environmental information or image is obtained when the mobile device moves, wherein the movement of the mobile device includes movement from a starting point, wherein the starting point corresponds to the following point in the environment, at which the mobile device stops at an existing boundary or stops at an existing boundary when moving (especially automatically). This reference Figure 2c Explain in more detail.

[0132] Then, in step 556, additional boundaries of the working area within the environment are determined for the mobile device based on the environment information and / or the image, such as, for example, also referring to Figure 2c The additional boundary can then be provided in step 558 and can be used again for moving the mobile device, for example, from step 534. The changed boundary can also be determined in the additional boundary, as described above with reference to the boundary 223 with the changed boundary 233.

Claims

1. A method for determining a boundary (223, 233) which at least partially delimits a working area (122, 222) of a mobile device (100), in particular a vehicle or a robot which moves at least partially automatically, in particular a lawnmower robot, the method comprising: In a first step, environmental information (502) of an environment (220) in which the mobile device moves or is to move is provided (500), the environmental information being obtained during an externally controlled movement of the mobile device or being obtained during the movement; determining (504) the boundaries (223) of the working area within the environment for the mobile device based on the environment information (502) from the first step; In a second step, environmental information (508), in particular an image, of the environment at least in the region of the border (223) is provided (506), the environmental information (508) being obtained when the mobile device is moved along the border and / or within the border or when the mobile device is moved along the border and / or within the border, and The boundary (223) is confirmed (518) and / or adapted (520) based on the environmental information from the second step.

2. The method according to claim 1, wherein the mobile device (100) performs a processing procedure when the mobile device moves along the boundary and / or re-moves within the boundary.

3. The method according to claim 1 or 2, wherein the environmental information from the first step is obtained by means of SLAM, in particular visual SLAM. The method of claim 3 , wherein the boundary is determined and / or refined using loop closure based on the environmental information from the first step.

5. The method according to any of the preceding claims, wherein the externally controlled movement of the mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a preset tolerance.

6. A method according to any of the preceding claims, wherein the environmental information from the first step comprises information about reference points in the environment, in particular images, wherein the information about the reference points is used in particular when determining the boundary and / or when determining a visual map.

7. The method according to any of the preceding claims, wherein the environment information (508) from the second step comprises an image of the environment, and wherein confirming and / or adapting the boundary comprises: A map (420) of the environment is determined (510) from the image from the second step, the map including at least a portion of the boundary.

8. The method according to any one of the preceding claims, further comprising: determining (524) a changed, in particular extended, boundary (233) based on the boundary (223) from the first step and / or the environmental information, in particular an image, from the second step, wherein the changed boundary is in particular determined automatically, and In particular, the changed boundaries are provided ( 526 ) to the user, in particular as a map.

9. The method according to claim 8, further comprising: obtaining (528) correction information (530) from an external source, and An adapted boundary is determined (532) based on the boundary and based on the correction information.

10. The method according to any one of claims 1 to 7, further comprising: determining a changed, in particular extended, boundary based on the boundary from the first step and / or environmental information, in particular an image, from the second step, wherein the changed boundary is in particular determined automatically, and The changed boundary is determined as the adapted boundary.

11. The method according to any one of the preceding claims, further comprising: determining (534) navigation information (536) for the mobile device based on the map of the boundary and / or the environment, determining (538) control information (540) for moving the mobile device based on the navigation information, and The control information is provided and / or the mobile device is moved based on the control information.

12. A computing unit (108) comprising means for executing the method according to any one of the preceding claims.

13. A mobile device (100) designed to: obtain control information determined by means of the method according to claim 11, and / or comprising: a calculation unit according to claim 12, and A mobile device having a drive system (104) and a control or regulating unit (102) for controlling the drive system, and in particular having a detection device (106) for detecting environmental information of an environment in which the mobile device is moving or is to move, The mobile device is in particular designed as an at least partially automatically movable vehicle, in particular as a passenger transport vehicle or a cargo transport vehicle, and / or as a robot, in particular as a household robot, such as a cleaning robot, a floor or street cleaning device or a lawn mowing robot, and / or as a drone.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to execute the method according to claims 1 to 11.

15. A computer-readable storage medium having stored thereon the computer program according to claim 14.

Images