Method for determining changed boundary of working area of mobile device

Abstract

The invention relates to a method for determining a changed boundary which at least partially delimits a working area of a mobile device, in particular an at least partially automated mobile vehicle or robot, in particular a mowing robot, comprising: providing an existing boundary of the working area within an environment, the mobile device moves in the environment or should move in the environment, an existing boundary has been determined based on environment information of the environment, said environment information having been obtained upon a first movement of the mobile device; providing environmental information, in particular an image, of the environment at least in the region of the existing boundary, said environmental information being obtained or obtained when the mobile device moves again along and / or within the existing boundary; based on existing boundaries and environmental information, in particular images, obtained during the re-movement of the mobile device, changed, in particular extended boundaries are automatically determined, and changed boundaries are provided, in particular as a map, in particular for the user.

Description

Technical Field

[0001] The invention relates to a method for determining a changing boundary, which at least partially delimits a working area of ​​a mobile device, in particular an at least partially automatically moving vehicle or robot, in particular a mowing robot, a computing unit and a computer program for executing the method, and a mobile device. Background Art

[0002] Mobile or working devices, such as at least partially automatically moving vehicles or robots, typically move in an environment, in particular an environment to be processed or a work area, such as in a home, in a garden, in a factory hall or on the street, in the air or in water. Summary of the invention

[0003] According to the invention, a method for determining a changed boundary of an operating range of a mobile device, a computing unit and a computer program for carrying out the method, and a mobile device are proposed with the features of the independent claims. Advantageous embodiments are the subject matter of the dependent claims and the subsequent description.

[0004] The invention generally relates to a mobile device that moves or at least can move in an environment or there, for example in a work area. Mobile working devices can then also be mentioned. Examples of such mobile devices (or mobile working devices) are, for example, robots and / or drones and / or vehicles that also move partially or (fully) automatically (on land, on water or in the air). As robots, for example, household robots, cleaning robots such as (for example in the form of vacuuming and / or mopping robots), floor or street cleaning devices, construction robots or lawn mowing robots are considered, but other so-called service robots are also considered, as vehicles that move at least partially automatically, such as passenger vehicles or freight vehicles (also so-called ground transport vehicles, for example in warehouses), as well as aircraft or watercraft such as so-called drones.

[0005] Such a mobile device has in particular a control or regulating unit and a drive unit for moving the mobile device so that it can be moved in an environment, in particular along a movement path. For this purpose, navigation information can be determined based on the movement path, i.e., for example, specific instructions: in which direction the mobile device should be driven in order to follow the movement path. This navigation information can then be implemented via the control or regulating unit and the drive unit.

[0006] For navigation, for example, a map of the environment can be used, which can be obtained or determined in particular with the aid of SLAM. The following method in robotics is called SLAM ("Simultaneous Localization and Mapping", roughly: simultaneous localization and mapping), in which a mobile device, such as a robot, can or must simultaneously create a map of its environment and estimate its spatial position within this map. It is therefore used to detect obstacles and thus supports autonomous navigation.

[0007] Furthermore, the mobile device or working device may have one or more sensors, by means of which the environment or information in the environment can be detected. These sensors may be, for example, cameras, lidar sensors or inertial sensors, by means of which the environment and / or the movement of the mobile device can be detected, for example, in two or three dimensions.

[0008] Furthermore, such a mobile device can be configured to receive data and / or to send data by means of a communication connection, i.e. to communicate or exchange data in general. In this way, communication with the mobile device can be carried out, for example, instructions can be given to the mobile device, other data can be transmitted to the mobile device or data or information can be received from the mobile device. In particular, wireless communication connections are considered here. For this purpose, the mobile device can, for example, have corresponding (if necessary different) modules for wireless communication, but these modules can also be integrated into the computing unit, for example.

[0009] In such mobile devices, it is also common that there are functions that the mobile device can perform, ie the mobile device is configured to perform one or more, preferably different functions. Instead of functions, it is also possible to talk about applications. Such functions can be, for example, work functions or learning functions.

[0010] The working function comprises, for example, automatically moving the mobile device in an environment and, at least temporarily during the movement, performing a working process in the environment. In the case of a lawn mower robot, the working function may thus comprise, for example, mowing the lawn (during the movement). For this purpose, a start instruction for performing the working function may be issued to the mobile device, for example, by sending corresponding data via a wireless communication connection.

[0011] The learning function includes, for example, manually controlled movement of the mobile device in the environment, in particular when no work process is being performed in the environment. In the case of a mowing robot, the learning function can therefore include, for example, (manual) movement or control along the boundaries of the working area in order to teach the mowing robot these boundaries. For this purpose, control instructions for navigating or moving in the environment can be issued to the mobile device, for example (continuously), for example, by sending corresponding data via a wireless communication connection. In this context, we also talk about learning or so-called "teaching".

[0012] One aspect of this learning is to find or predetermine a boundary (or limit) that surrounds a working area, ie, for example, a garden. In the working function, this boundary can then be used to delimit the automated movement of the mobile device.

[0013] For a mobile device, a possibility is provided for determining such a boundary for at least partially delimiting a working area of ​​the mobile device.

[0014] For this purpose, in a first step, environmental information of an environment is provided, in which the mobile device moves or should move. These environmental information can be, for example, positions, orientations, etc. that can be used for SLAM. However, these environmental information can also include, for example, images or environmental images detected by or already detected by the camera of the mobile device. Here, these environmental information (in the first step) have been obtained when the mobile device is moved from the outside, more precisely, when the mobile device is moved manually, as mentioned above about the learning function. Here, the mobile device can therefore be controlled, for example, via a smartphone and a Bluetooth connection. Here, the user can therefore, for example, control the mowing robot at least approximately or as well as possible along the edge of the desired working area and follow the mowing robot at the same time. In the case of such movement of the mobile device in the first step, the first movement can also be mentioned.

[0015] Based on the environment information from the first step, the boundaries of the working area of ​​the mobile device within the environment are then determined. The boundary can be a temporary or rough boundary, but the boundary can in principle be used for working functions. Since the boundary is based on manual control of the mobile device, it may happen that the boundary is not very precise.

[0016] The environmental information from the first step can be obtained here in particular by means of SLAM, in particular visual SLAM. For this purpose, for example, the detected image can be processed accordingly. Preferably, the boundary is determined and / or refined based on the environmental information from the first step using loop closure.

[0017] So-called loop closure is an aspect related to SLAM. In this case, it is assumed that the mobile device reaches a certain position in the environment during movement in the environment or in the working area, and the mobile device has been in this position once before. According to environmental information, such as an image or for example, there is also spacing information from the corresponding position of the mobile device, and by comparing such environmental information with previous environmental information, it can be determined whether the mobile device has been in a position once before, or the mobile device has been in a position once before. If this is found out, the map created within the scope of SLAM or the path marked as passing in the map of the mobile device must also be explained: the mobile device is located at the same position again. However, due to possible deviations, it is often not the case. More precisely, in the map created, the current position will deviate from the previous position. Here, the map and therefore the boundary can be adjusted now so that the two separated positions are consistent again. In this case, so-called loop closure is mentioned.

[0018] In this context, it is also particularly expedient if the externally controlled movement of the mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a predetermined tolerance. For example, the starting point and the end point can be at a docking station. This allows a loop to be closed and the boundaries to be particularly well defined in the first step.

[0019] In a second step, environmental information, in particular images, of the environment at least in the region of the border are then provided. This environmental information is already obtained when the mobile device is moved again along the border and / or within the border. This re-movement is in particular an automated movement, i.e., for example within the scope of the mentioned working functions, i.e., processing can be performed here. This can, for example, include a first mowing of the (possibly entire) working area. In the case of this (re)movement of the mobile device in the second step, another movement can also be mentioned.

[0020] Based on the environmental information from the second step, the boundary (obtained in the first step) is then confirmed and / or adjusted. If the boundary is confirmed, the boundary obtained in the first step can continue to be used, which can be referred to as a confirmation. If the boundary is adjusted, the boundary obtained in the first step can be adjusted, for example, based on the environmental information obtained in the second step. Thus, for example, the following information can be obtained based on the environmental information (preferably an image) from the second step: there is also a lawn at a certain location outside the boundary from the first step, so that the boundary can be adjusted accordingly there. It is easy to understand that parts of the boundary can be confirmed and other parts of the boundary can be adjusted.

[0021] As mentioned, the environmental information from the second step may include an image of the environment. In one embodiment, confirming and / or adjusting the boundary then includes determining a map of the environment including at least a portion of the boundary based on the image from the second step. This can be done, for example, using an orthogonal projection.

[0022] In this way, in the first step, no or at most a small number of images must be detected and stored in order to create the map later. More precisely, the determination of the boundary is carried out only via (visual) SLAM. This saves computing resources. The environmental information, especially the images, from the second step can then be used for verification or adjustment. Since the second step can be carried out within the scope of the working process, the mobile device with the actual working function can be used more quickly than when, for example, an additional step for verification is carried out (before the working function).

[0023] However, instead of these two steps, it is also possible to provide environmental information and images of the environment in which the mobile device moves or should move, which are obtained or have been obtained during the movement of the mobile device controlled from the outside. The boundaries of the working area of ​​the mobile device within the environment are then determined based on these environmental information and images. Therefore, in this case, images are also detected and stored and processed if necessary. This allows particularly accurate boundaries to be determined during only one movement of the mobile device, although more computing resources may be required here. Compared with the above-described method with two steps, only the first step is required here, but images are also detected there.

[0024] In one embodiment, in this case, environmental information, in particular images, of the environment at least in the region of the boundary can also be provided, which environmental information is or has been obtained during another (in particular automated) movement of the mobile device along the boundary and / or within the boundary. In the case of the other movement, the mobile device in particular performs a processing process. This can correspond to the second step of the previously described variant. With this environmental information or image, it is then not necessary to confirm and / or adjust the boundary (although this may be the case again). More precisely, a map of the environment can be determined in this way, which includes at least a part of the boundary, which can be achieved, as already mentioned above, for example, using orthogonal projections.

[0025] As has been shown, particularly precise boundaries can also be obtained by targeted movement of the mobile device, depending on whether, for example, both steps or only one step for determining the boundary is used.

[0026] In this case, environmental information and / or images of the environment are provided, in which the mobile device moves or should move. Here, these environmental information and / or images are obtained when the mobile device moves: this movement can be controlled from the outside, manually or automatically.

[0027] However, the movement of the mobile device here includes at least one movement section in which the mobile device is moved at least partially outside the working area on a ground surface that is different from the ground surface of the working area (or the expected working area). This can be, for example, a road or a sidewalk, while the (expected) working area has, for example, a lawn as the ground surface. In other words, the mobile device is thus moved in a targeted manner outside the (actual or expected) working area, whereas this is usually only done within the (actual or expected) working area.

[0028] Based on this environmental information and / or image, the boundaries of the working area of ​​the mobile device within the environment are then determined. Through this targeted movement of the mobile device outside the (actual or desired) working area, it is possible to detect the boundaries of the (desired) working area more accurately, so that the boundaries (whether temporary or final boundaries) can also be determined significantly more accurately. It should be mentioned in this context that the boundaries here do not have to be directly based on the movement of the mobile device, for example a movement controlled from the outside, but can be adjusted accordingly if necessary in order to limit the boundaries to the (desired) working area.

[0029] It is advantageous if the ground surface different from the ground surface of the working area adjoins the ground surface of the working area and / or has the same height within a predetermined tolerance. The height in this case depends in particular on the direction of gravity. In this way, the mobile device can be moved without problems from one ground surface to the other, or even on both ground surfaces simultaneously.

[0030] It should be mentioned that often only a part of the boundary can be determined in this way, since for example parts of the ground outside the (desired) working area may also be inaccessible; for example, in certain places fences, walls or water may limit the (desired) working area.

[0031] As mentioned, a map of the environment, more precisely a visual map, can be determined using orthogonal projections. A visual map is to be understood here in particular as a map that graphically shows the environment, rather than a SLAM map with nodes and edges, etc. In this context, a structural map can also be mentioned, which thus shows the structure of the environment, for example areas with lawns and areas without lawns.

[0032] For this purpose, an image of the environment is provided, wherein the image is or has been obtained while the mobile device is moving in the environment. Such movement may be from outside, manual or automated. An orthogonal projection view of the image of the environment is then determined. Based on the orthogonal projection view of the image of the environment, a visual map of the environment is then provided; this visual map can then be used as explained above.

[0033] In one embodiment, a visual map of the environment is determined using a computer vision algorithm, such as in particular segmentation, for example semantic segmentation. By applying the computer vision algorithm, the image can be preprocessed. In particular, coherent regions can be determined in the image, in particular even in the case of segmentation. In this case, the image of the environment can be segmented or generally processed with the aid of a computer vision algorithm before the orthogonal projection view is determined, in order to obtain a segmented image of the environment or a generally preprocessed image. The orthogonal projection view of the image of the environment is then determined based on the segmented image of the environment or the preprocessed image.

[0034] However, it is also possible to determine the orthogonal projection views of the image of the environment before segmentation or application of the computer vision algorithm. The orthogonal projection views of the image of the environment are then segmented or preprocessed.

[0035] In one specific embodiment, the segmentation is performed using a machine learning algorithm, such as an artificial neural network, but classical use of computer vision algorithms is equally possible.

[0036] If the mobile device is a lawn mowing robot, and the image of the environment includes an image of grass (or lawn), the boundaries of the grass can be determined when segmenting or applying a computer vision algorithm.

[0037] In this way, the limits of the (desired) working area, i.e., the lawn, for example, can be determined particularly accurately, so that the visual map can present the working area particularly accurately. In particular, the boundaries can therefore be determined particularly accurately, i.e., for example, via the boundaries between the lawn and other ground surfaces. This allows particularly accurate determination or adjustment of the boundaries, as mentioned above.

[0038] Here, an orthogonal projection of an image (or a preprocessed or segmented image) is to be understood in particular as a projection of the image onto the ground, i.e. from top to bottom. It should be taken into account that the camera of the mobile device, such as a lawn mower, is usually oriented at least approximately horizontally. Therefore, in the image captured or recorded with such a camera, the ground is only presented at the lower edge and distortedly. Taking into account the camera parameters, such as the specific arrangement and field of view of the camera on the mobile device, etc., the part of the image showing the ground can be corrected and presented in the sense of a top view. In this presentation of the orthogonal projection, the specific and precise limits of the working area can be clearly better recognized.

[0039] In the case of segmentation, in particular so-called semantic segmentation, each image point (pixel) in the image can be assigned one of a number of classes, i.e. the image points can be labeled or classified. For this purpose, typically first features of the image points are determined or extracted, and then the classes are assigned based on these features. Such features can be, for example, shape, color, context, pattern, light changes and / or image context. Thus, for example, an image point can be assigned the color green, and the surrounding image points can also all be green. This would indicate a lawn.

[0040] Different features are considered as categories. In the example of a lawnmower robot, two categories can be used, for example, in a simple case, namely "lawn" and "non-lawn", i.e. for each image point in the image it is determined whether lawn is shown. It is self-evident that more than two categories can also be used, for example in order to also identify roads, streets, buildings, vehicles or people and assign them to image points as necessary. Based on the category "non-lawn", the category "background" is also conceivable, to which everything that is not assigned to another category is assigned. All these markings add more information compared to pure object recognition, which only recognizes objects in the image.

[0041] In principle, the classification does not have to be performed for each individual pixel, but can also be performed for segments or parts of the image, which each include a plurality of pixels. Such segments can be fixedly predetermined or can also be formed within the scope of semantic segmentation, possibly also having different sizes within the image. For this purpose, one or more artificial neural networks or generally pattern recognition methods based on artificial intelligence are preferably used.

[0042] The segmented image thus allows a simpler and faster determination of where the boundaries between different ground surfaces are.

[0043] In one embodiment, a changed, in particular extended, boundary is also determined, more precisely based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. The determination of the changed boundary is here particularly automated. The changed boundary is then provided to the user, in particular as a map. Thus, for example, a suggestion of the changed boundary can be provided. For this purpose, the changed boundary can be displayed, for example, on a smartphone or other input device.

[0044] In one embodiment, it is provided that a modified, in particular extended, boundary is determined based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. The determination of the modified boundary is here particularly automated. Here, the modified boundary is determined as an adjusted boundary. Thus, the improvement of the boundary can be performed here automatically, in particular without changes by the user.

[0045] The determination of the changed boundaries can also be carried out, in particular, independently of how exactly the existing boundaries have been obtained. For this purpose, the existing boundaries of the working area within the environment in which the mobile device is moving or is to be moved can be provided. The existing boundaries are determined at least on the basis of environmental information about the environment, which has been obtained during the first movement of the mobile device.

[0046] Then, environmental information, in particular an image, of the environment at least in the area of ​​the existing border is provided. This environmental information is then obtained during a new, in particular automated, movement of the mobile device along the existing border and / or within the existing border. Then, the changed border is automatically determined, more precisely based on the existing border and the environmental information, in particular the image, obtained during the new movement of the mobile device. The changed border is provided, in particular, as a map (for example the visual map mentioned), in particular for the user. This can be done, for example, on a smartphone or other input or display device. Thus, the user can receive suggestions for improved borders.

[0047] This changed boundary can be in particular an expanded boundary or an improved boundary. It is thus possible to obtain a better boundary particularly simply and quickly (because it is automated) that allows a more precise coverage of the desired working area.

[0048] It can also be determined whether the deviation between the changed boundary and the existing boundary meets a predetermined criterion; for example, the average distance or cumulative distance between the two boundaries can be determined and compared with a threshold value. Only if the deviation meets the predetermined criterion is the changed boundary provided. Thus, for example, a suggestion is only presented to the user if this appears reasonable.

[0049] Furthermore, it can be provided that correction information is obtained from the outside. For this purpose, the user can, for example, make and transmit a change to the suggestion based on the suggestion. Then, the adjusted boundary is determined based on the boundary (depending on which one exists, either the existing boundary and the proposed changed boundary) and based on the correction information.

[0050] In one embodiment, it is provided that the environment information from the first step includes information about reference points in the environment, in particular images. The information about the reference points can then be used, in particular, in determining the boundary and / or in determining the visual map. Such reference points can be, for example, checkpoints, in particular also only a few checkpoints, not along the entire path, but for example at certain intervals and / or at certain maneuvers (90° bends, etc.). This allows a more precise verification or adjustment of the boundary or map, while only a small number of images must be stored in the first step.

[0051] In addition to the above-mentioned possibility of determining a boundary that at least partially delimits the working area, which in particular relates to an external boundary, it is also possible to determine an additional boundary, in particular an internal boundary, i.e. a boundary that is enclosed by the existing boundary, starting from the existing boundary (which has then been determined, for example, as described above). Such an additional boundary can, for example, exclude trees or other obstacles present in the lawn from the working area to be processed.

[0052] For this purpose, environmental information (e.g. position, orientation) and / or images of the environment in which the mobile device moves or is to move are provided. This environmental information or images are obtained when the mobile device moves. The movement of the mobile device in this case includes a movement starting from a starting point, wherein the starting point corresponds to a point in the environment at which the mobile device is stopped or has been stopped when moving along an existing boundary or at an existing boundary (this can be a boundary, as mentioned above, whether it has been adjusted or changed). The movement of the mobile device in this case is carried out in particular automatically. The stopping of the mobile device in this case is carried out actively and / or manually, that is to say consciously.

[0053] Then, based on the environment information and / or the image, and if necessary also based on existing boundaries, additional boundaries of the working area of ​​the mobile device within the environment are determined. In other words, the mobile device is stopped during its (automated) movement along an already existing boundary or at least (briefly) at such an existing boundary and is controlled from there, for example, around an obstacle. In this case, the mobile device is at least partially controlled from the outside (for example manually), for example as described above for the first step. In this way, it is known for the additional boundary how the additional boundary is located relative to the existing boundaries.

[0054] Preferably, the movement of the mobile device further includes a movement to an end point, wherein the starting point and the end point are identical at least within a predetermined tolerance. Thus, the additional boundary can be optimized, for example, by means of loop closure.

[0055] Preferably, the movement from the starting point to the intermediate point comprises an externally controlled (e.g. manual) movement of the mobile device, while the movement from the intermediate point to the end point then comprises, for example, an automated movement of the mobile device. In particular, the intermediate point corresponds, at least within a predetermined tolerance, to a point that the mobile device has already passed during the movement from the starting point to the intermediate point. In this way, it is ensured that the mobile device has completely circumvented the obstacle.

[0056] Furthermore, what has already been said about the other boundaries then applies to this additional boundary, ie this additional boundary can be determined, for example, with the aid of SLAM.

[0057] Based on the boundaries of the environment (if necessary, also additional boundaries and / or changed or adjusted boundaries) and / or the (if necessary visual) map, navigation information can then be determined for the mobile device. Based on this navigation information, control information for moving the mobile device can in turn be determined. The control information can then be provided and / or the mobile device can be moved based on the control information.

[0058] A computing unit according to the invention, ie a system for data processing in general, such as a control unit or a control unit of a mobile device, or a central server or another computing system is configured in particular in terms of programming to carry out the method according to the invention.

[0059] The invention also relates to a mobile device, which is configured to obtain control information, which is determined as mentioned above. In addition or alternatively, the mobile device has a computing unit according to the invention. In addition, the mobile device has a control or regulation unit and a drive unit for moving the mobile device. In addition, the mobile device can have a detection device for detecting environmental information of the environment, such as the mentioned camera or cameras, in which the mobile device moves or is to move.

[0060] Preferably, the mobile device is configured as a vehicle that moves at least partially automatically, in particular as a passenger vehicle or as a freight vehicle, and / or as a robot, in particular as a household robot, such as a cleaning robot, a floor or road cleaning device or a lawn mowing robot, and / or as a drone.

[0061] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this results in particularly low costs, especially when the executed control device is also used for other tasks and therefore always exists. Finally, a machine-readable storage medium is provided, which has a computer program as described above stored thereon. Suitable storage media or data carriers for providing the computer program are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. The program can also be downloaded via a computer network (Internet, intranet, etc.). Such downloading can be performed here either wired or cable-connected or wirelessly (e.g., via a WLAN network, 3G, 4G, 5G or 6G connection, etc.). BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Further advantages and embodiments of the invention are apparent from the description and the drawings.

[0063] The invention is schematically represented by means of exemplary embodiments in the drawings and is described below with reference to the drawings.

[0064] Figure 1a A mobile device is schematically shown for explaining the present invention.

[0065] Figure 1b Another view schematically shows Figure 1a Mobile devices in .

[0066] Figure 2a , 2b 2c schematically shows an environment with a mobile device for explaining the present invention.

[0067] Figure 3a , 3b 3c and 3d schematically show images for explaining embodiments of the present invention.

[0068] Figure 4a , 4b A map on an input device is schematically shown.

[0069] Figure 5 The flow of the method for explaining the embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0070] exist Figure 1a A mobile device 100, in particular a working device, for explaining the present invention is presented schematically and exemplarily in FIG. Figure 1b In the embodiment, the mobile device 100 is presented in another view and using other aspects. Figure 1a and Figure 1b .

[0071] The mobile working device 100 is exemplarily a lawnmower robot having a control or regulating unit 102 and a drive unit 104 (with wheels) for navigating or moving the lawnmower robot 100 in an environment 120 and there, in particular in a working area 122, such as a lawn or a garden. The lawnmower robot 100 can move or be moved, for example, along a movement path or trajectory 130. In addition, the lawnmower robot 100 exemplarily has a sensor 106 designed as a camera. With the aid of the camera 106, images of the environment can be detected, which in turn can be used for navigation. In addition, a docking station 110 is exemplarily provided, at which the lawnmower robot can be charged, for example.

[0072] exist Figure 1b In the environment, buildings 124 , trees 126 , and persons 128 are depicted by way of example, which may be considered as objects or obstacles during the navigation of the mowing robot.

[0073] Furthermore, the mowing robot 100 has a computing unit 108, for example a control device, by means of which data can be received and / or sent. As already mentioned, this can be done, for example, via different types of wireless communication connections. These types of wireless communication connections are Figure 1a 170, 172, 174. By way of example, these are a mobile radio connection 170, a Bluetooth connection 172 and a WLAN connection or WiFi connection 174. For this purpose, the computing unit 108 can, for example, have a corresponding radio module or be connected to such a radio module, which is then part of the mowing robot 100.

[0074] In addition, Figure 1a In the embodiment of the present invention, a mobile input device 140, such as a smartphone, a central computing system 150 (or a server which can represent a so-called cloud) and a WLAN router 160 in a building 162 (which can be a different building from the building 124 or can also be the same building). Wireless communication connections 170, 172, 174 of the type already mentioned are provided between the mowing robot 100 or its computing unit 108, the mobile input device 140, the computing system 150 and the WLAN router 160. It should be noted that the mobile communication connection 170 is established via a mobile radio transmitter 152, which in turn is connected to the computing system 150.

[0075] A Bluetooth connection 172 is provided between the mowing robot 100 or its computing unit 108 and the mobile input device 140, ie a Bluetooth connection can be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140. It is self-evident that the mobile input device 140 has a suitable radio module for this purpose.

[0076] A WLAN connection 174 is provided between the mowing robot 100 or its computing unit 108 and the WLAN router, i.e. a WLAN connection can be established between the mowing robot 100 or its computing unit 108 and the WLAN router 160. It is self-evident that the WLAN router 160 has a suitable radio module for this purpose. The WLAN router 160 can in turn be connected to the Internet via, for example, a wired connection.

[0077] A mobile radio connection 170 is provided between the mowing robot 100 or its computing unit 108 and the computing system 150, i.e. a mobile radio connection can be established between the mowing robot 100 or its computing unit 108 and the computing system 150. It should be mentioned here that such a mobile radio connection 170 is established by the mowing robot 100 or its computing unit 108 or a radio module therein, for example with a mobile radio transmitter 152 or a corresponding mobile radio station, to which the computing system 150 is in turn connected, for example also via mobile radio or wired.

[0078] Furthermore, a mobile radio connection 170 is also provided between the mobile input device 140 and the computing system 150, that is, a mobile radio connection can be established between the mobile input device 140 and the computing system 150. It also applies here that such a mobile radio connection 170 is established by the mobile input device 140 or a radio module therein, for example, with a mobile radio transmitter or a corresponding mobile radio station, to which the computing system 150 is in turn connected. It should be mentioned that the mobile input device 140 includes a radio module for WLAN for this purpose, as is usually the case in typical smartphones.

[0079] Likewise, a WLAN connection 174 may also be provided between the mobile input device 140 and the computing system 150, that is, a WLAN connection may be established between the mobile input device 140 and the computing system 150. For this purpose, a WLAN connection 174 may be provided between the mobile input device 140 and a WLAN router. The WLAN router 160 may again be connected to the Internet via, for example, a wired connection, as already mentioned, and to the computing system 150 via the Internet.

[0080] That is, the mowing robot 100 or its computing unit 108 can therefore send and receive data, i.e. exchange data, via any of the mentioned wireless communication connections 170, 172, 174. With the mobile input device 140, data can be exchanged not only via a Bluetooth connection, but also via a mobile radio connection 170 or a WLAN connection 174, i.e. indirectly via the computing system 150. It should be mentioned that in principle a WLAN connection can also be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140.

[0081] The mobile working device or lawn mowing robot 100 can be configured to perform one or more functions, which in one embodiment include a working function, such as a lawn mowing function, and a learning function.

[0082] The working function includes, for example, the automated movement of the mowing robot 100 in the environment 120 and the execution of a working process (or processing process) in the environment, i.e., for example, mowing a lawn at least sometimes during the movement in the environment (depending on the situation, the mowing robot first drives to a certain location without mowing in order to start or continue mowing there).

[0083] The mobile radio connection 170 can be assigned to the operating function. This means that the mowing robot 100 or its computing unit 108 at least partially receives and / or transmits the data necessary for performing the operating function via the mobile radio connection. This can include, for example, a start command for starting a mowing process, which is transmitted to the mowing robot 100 by the mobile input device (optionally via the computing system 150).

[0084] This may also include obtaining, for example, trajectories (or navigation information in general) from the computing system 150, which are determined in the computing system 150 and which the mowing robot is then to follow for the mowing process. However, it may also be provided that the trajectories (or navigation information in general) are determined in the mowing robot 100 or its computing unit 108 itself or are obtained there in another way. Instead of trajectories, however, the computing unit 108 may also (only) receive control information that has been determined based on the trajectories (which in turn may have been determined in the computing system 150), and based on this control information, the control or regulating unit 102 via the drive unit 104 may move the mowing robot 100 in order to follow a trajectory, such as the trajectory 130 that is merely indicated here by way of example.

[0085] The mowing robot 100 can then, for example, independently move over the working area 122 or in the environment 120 or navigate there and at the same time mow the lawn. As already mentioned, various objects can be detected by the mowing robot 100 or its camera 106 (i.e., these objects are visible in the images recorded by the camera) and then taken into account during navigation, i.e., when determining navigation information. For this purpose, the images detected here or corresponding data can be transmitted to the computing system 150, for example, via the mobile radio connection 170.

[0086] The learning function includes, for example, manually controlled movements of the mowing robot in the environment, in particular along the boundaries of the working area 122, in order to teach the mowing robot these boundaries. In this case, the working function, i.e. mowing, does not have to be used (or should not be used). For this purpose, control instructions (or control instructions) for navigation or movement can be transmitted to the mowing robot 100 (continuously), for example. Such control instructions or corresponding data are sent to the mowing robot from the mobile input device 140 via the Bluetooth connection 172. Likewise, the mowing robot 100 or its computing unit 108 can send data back to the mobile input device 140 via the Bluetooth connection. In this case, we also talk about learning or so-called "teaching".

[0087] exist Figure 2a In order to explain the present invention, an environment 220 having a building 224 and a tree 226 is shown, and Figure 1a , 1b In addition, the desired working area, such as a lawn to be mowed, is shown by way of example at 221. The mobile device or mowing robot 200 and the docking station 210 are also shown again by way of example.

[0088] It is then typically desired that the desired working area 221 is covered completely and also as accurately as possible by the mobile device during processing, so that the entire lawn is mowed as far as possible.

[0089] For this reason, the boundaries of the (actual) working area should now be determined, which can then be processed by the mobile device. In other words, a boundary or limit should be determined, within which the mobile device should move and should process.

[0090] Within the scope of the aforementioned learning function, the mobile device 200 can now be controlled by the user, for example by means of a smartphone, from the docking station 210 along the edge of the desired working area 221 until the mobile device 200 reaches the docking station 210 again.

[0091] Based on this movement and the environmental information obtained at the same time, such as an image, a boundary 223 of the (actual) working area 222 can be determined in a first step, as shown in FIG2. It can be seen in this case that this boundary 223 deviates from the actual edge of the desired working area 221.

[0092] This may be due in part in particular to the user moving the mobile device 200 too far from the actual edge of the desired working area 221. This is illustrated in FIG. 2 at the bottom left and top right, where the boundary 223 is particularly far from the actual edge of the desired working area 221.

[0093] It should be mentioned here that during a movement of mobile device 200 , its position is determined, if necessary relative to a center point, wherein the geometric dimensions of the mobile device or its processing tool are to be taken into account for the boundaries.

[0094] However, in a second step, environmental information, in particular an image of the environment, can now be obtained again, more precisely when mobile device 200 is moved again along boundary 223 and / or within boundary 223. This can be the case, for example, within the scope of the aforementioned operating function, in which mobile device moves according to trajectory 330.

[0095] Based on the environmental information or images obtained in the meantime, the boundary 223 can then be confirmed and / or adjusted. Figure 2a It is outlined at the bottom left and top right in FIG. 2 that the boundary 223 is, as mentioned, particularly far from the actual edge of the desired working area 221 . The adjusted boundary is denoted there by 233 .

[0096] When the mobile device 200 moves, for example, along the trajectory 330, images are also detected from the lower left portion. By analyzing these images, it can be recognized that the actual edge of the desired working area 221 deviates from the boundary 223.

[0097] Based on the images, a map of the environment including at least a portion of the boundary may also be determined.

[0098] As already mentioned, instead of these two steps, it is also possible to use only one step in which the mobile device is controlled from the outside, wherein an image of the surroundings is then also recorded, however.

[0099] exist Figure 2b In the Figure 2a231, a path or a movement path is now shown here, along which the mobile device 200 moves, for example in the case of a manual movement controlled from the outside. It can be seen here that the mobile device comprises a movement section or a movement path section 232, in which the mobile device moves at least partially outside the (desired) working area 221, more precisely on a ground 221b that is different from the ground 221a of the working area 221. For example, the ground 221a can be a lawn, while the ground 221b is a road or a sidewalk.

[0100] By means of this movement, the boundary, here indicated by 223', can be determined more precisely, i.e. there is no such movement outside the working area 221. It should be mentioned here that the boundary 223' is shown here slightly spaced apart from the edge of the (desired) working area 221, although this can be significantly better matched in practice.

[0101] exist Figure 2c In the Figure 2a The part of the environment 220 in the embodiment. Here the determination of the additional boundary will now be explained. For example, the boundary 223 determined as explained above can be assumed to be the existing boundary.

[0102] When the mobile device 200 moves along the boundary 223, more precisely, in particular, automatically, the mobile device can be stopped, more precisely, for example, at point P1. From there, that is, point P1 as the starting point, the mobile device 200 can then be manually moved around the tree 226, that is, the obstacle, for example, via the aforementioned Bluetooth connection to the smartphone; this is shown by roads 235 and 236. This manual movement is preferably carried out until point P3, that is, the intermediate point, which is at least within a predetermined tolerance of the point (here point P2) that has been previously traveled (i.e., located on roads 235, 236). From there, the mobile device can then, for example, be automatically driven again to the end point, here point P4, which coincides with the starting point at least within a predetermined tolerance. This is shown by road 237.

[0103] In this way, an additional boundary can be determined, more precisely in particular as that part of the road which is located between the points P2 and P3; this is an at least approximately closed loop; Figure 2c , the additional boundary corresponds to road 236 .

[0104] Boundary 223 or 233 is therefore in particular an outer boundary, whereas boundary 236 can be referred to as an inner boundary.

[0105] exist Figure 3a , 3b, 3c, 3d schematically show orthogonal projections and segmented images as can be used in embodiments of the present invention for explaining the mentioned examples of application of computer vision algorithms.

[0106] exist Figure 3a , an image 300a is shown, which may be detected by a mobile device or its camera. For example, the image 300a may be approximately as in Figure 2a or Figure 2b Detected in the situation shown in . In this, in particular, a building 324 can be seen in the front view, and a section of the desired working area 322 with, for example, a lawn as ground 321a can be seen at the lower edge. A road with a different ground 321b stretches next to the working area 322.

[0107] As can be discerned from image 300 a , it may be difficult to directly determine a boundary based thereon which should correspond as precisely as possible to the desired working area 322 or its edge.

[0108] exist Figure 3b , which is an orthogonal projection of image 300a. Here, it can be seen that the section of the desired working area 322 with lawn as ground surface 321a and the road with other ground surface 321b are shown in a top view (or a top-down view). There, the geometry and the size or extent of the edge of the desired working area 322 (i.e. the line denoted here by 322) at least approximately correspond to the actual conditions.

[0109] exist Figure 3c , which is a semantic segmentation of image 300a. Here, only two classes are distinguished, namely, for example, lawn 321a and background or the remaining part which is not represented here. There, for example, buildings, roads and all other parts except lawns belong to this class. It can be seen here.

[0110] exist Figure 3d 300d is shown, which corresponds to image 300a with semantic segmentation and orthogonal projection. This can be obtained from image 300b by semantic segmentation or from image 300c by orthogonal projection, for example. The result will be at least approximately the same. It is easy to understand that different small differences may occur.

[0111] In any case, the orthogonal projection of the image, in particular with (semantic) segmentation, allows a particularly precise determination of the edge of the (desired) working area, whereby this boundary can be determined particularly precisely. Likewise, an exact (visual) map of the environment can thus be determined, in which map at least a part of this boundary is shown, for example. This boundary can then be displayed to the user, for example on a smartphone.

[0112] exist Figure 4a An input device 440, such as a smart phone, is shown in FIG. Figure 1a In addition to controlling the mobile device during learning, for example, the input device can also display the information in the input device. Figure 4a 420 is shown by way of example in the visual map 420. For example, an area 421a of a lawn or grass field and an area 421b outside of it can be seen there. Furthermore, a line 423 is shown in the map, which line can correspond, for example, to a boundary, which has been created in the first step mentioned above; this can correspond, for example, to a boundary according to Figure 2a Boundary 223.

[0113] Furthermore, a line 433 is shown, which can represent a changed, in particular extended and / or improved boundary; this can correspond, for example, to Figure 2a Such a changed or extended boundary can be determined, as mentioned, in particular automatically, and then displayed to the user as a suggestion in the map, as here in Figure 4a As presented in .

[0114] The user will then be able to confirm, for example, via a suitable input instruction on input device 440, that he agrees with the suggestion of boundary 433. Likewise, the user will be able to reject the suggestion with boundary 433. Boundary 423 may then be retained, for example.

[0115] Users can also suggest changes, for example. Figure 4b , more precisely, the boundary of the change is represented by line 434. For this purpose, the user can move line 433 (for example at a suitable point of the line) by means of a touch display of input device 440, for example, so that this leads to line 424.

[0116] exist Figure 5 The flow of the method for determining a boundary is schematically shown in FIG. Figure 1a , 1b The working area of ​​the lawn mowing robot 100 of 2a, 2b, 2c is delimited. This process will be explained in more detail later with reference to the figures explained above.

[0117] For this purpose, in step 500, i.e. the first step, environmental information 502 of the environment in which the mobile device moves or is to move is provided. This environmental information may include, for example, the position, the orientation, and may, for example, have been determined based on images captured by means of a camera of the mobile device. This may be achieved, for example, by means of SLAM, in particular visual SLAM. In this case, the mobile device is already controlled from the outside, as described above, for example, with regard to Figure 2a In this case, the mobile device can also be moved outside the desired working area on other surfaces, such as for example Figure 2b Explained.

[0118] In step 504, based on the environment information 502 from the first step, the boundaries of the operating area of ​​the mobile device within the environment are then determined, such as for example with respect to Figure 2a This can be achieved, for example, when using a loop closure, especially when the starting point and the end point are identical or close to each other in the case of an externally controlled movement, for example at a docking station. A boundary is thus also provided.

[0119] In step 506, the second step, then environmental information 508, such as an image, of the environment at least in the region of the border is provided. This environmental information or image 508 has been obtained when the mobile device is moved along the border and / or within the border, as described above, for example, with respect to Figure 2a Explained.

[0120] In step 510, a map, in particular a visual map, of the environment may be determined based on or from image 508. This may be achieved using orthogonal projection and segmentation. An example of such an image is Figure 3a An example of such a map is shown in Figure 4a Map 420 shown in .

[0121] In this case, in step 512, orthogonal projection views 514 of an image of the environment can be determined from image 508, as shown, for example, using image 300b in FIG. 3. In step 516, these orthogonal projection images 514 or views thereof can then be segmented, such as, for example, using Figure 3d This map can thus be created and also provided.

[0122] Based on this map or only the segmented orthogonal projection images, the existing boundary (from the first step) can then be confirmed (or verified) in step 518. If necessary, however, the existing boundary can be adjusted in step 520. Then, in step 522, the boundary obtained here (i.e., for example confirmed or adjusted) can be provided, for example, for further use.

[0123] Alternatively, however, it can also be provided that in step 500', the environmental information 502 and the image 508, which have been obtained in the case of an externally controlled movement, are provided. In step 510', a map, in particular a visual map, can be determined based on or from the image 508, as also described above. In particular, steps 512 to 516 can also be provided here.

[0124] Then, in step 518', a boundary may be determined based on the environmental information and the image. Then, in step 522', the boundary may be provided, for example, for further use.

[0125] Based on the boundaries provided in step 522 or 522' or based on the boundaries determined or provided in step 504, a changed, in particular extended, boundary can be determined in step 524, more precisely based on the boundaries from the first step and / or the environment information, in particular the image, from the second step, or generally determined from existing boundaries, such as, for example, with respect to Figure 2a A further mobile environment information, in particular an image, can then also be provided (for example during the processing process).

[0126] The determination of the changed boundaries is here particularly automated. Then, in particular as a map, the changed boundaries are provided to the user, step 526, such as for example with respect to Figure 4a Explained.

[0127] The user can then make changes or corrections, such as regarding Figure 4b Then, in step 528, correction information 530 is received externally (ie, for example, from a smartphone). Then, in step 532, the boundaries determined in step 518 or obtained in step 518' can be adjusted. However, the adjusted map according to step 520 can also be readjusted.

[0128] Likewise, the adjusted map according to step 520 can be obtained in such a way that the adjusted map is first obtained using the user's correction. Therefore, the adjusted boundary is then determined based on this boundary and based on this correction information. However, the changed boundary can also be determined without correction information as an alternative.

[0129] Based on the boundary (e.g., first obtained, adjusted, or changed boundary), navigation information 536 can then be determined for the mobile device in step 534. Based on this navigation information, control information 540 for moving the mobile device can then be determined in step 538 and can then be provided.

[0130] After obtaining a boundary, whether an initial boundary (see step 522 or 522') or an adjusted or changed boundary (see step 532), additional, for example internal boundaries may also be determined based on the boundary as an existing boundary.

[0131] To this end, in step 550, environmental information 552 (similar to environmental information 502, such as position, orientation) and / or an image 554 (similar to image 508) of an environment in which the mobile device moves or is to move is provided. This environmental information or image is obtained when the mobile device moves, wherein the movement of the mobile device includes movement from a starting point, wherein the starting point corresponds to a point in the environment at which the mobile device is stopped or has been stopped when moving (especially automatically) at an existing boundary. This is about Figure 2c Explain in more detail.

[0132] Then, in step 556, additional boundaries of the operating area of ​​the mobile device within the environment are determined based on the environment information and / or the image, such as, for example, also with respect to Figure 2c This additional boundary can then be provided in step 558 and used again to move the mobile device, for example, from step 534. A changed boundary can also be determined in the case of this additional boundary, as has already been described above with respect to boundary 223 and changed boundary 233.

Claims

1. A method for determining a changed boundary (223) which at least partially delimits a working area (122, 222) of a mobile device (100), in particular a vehicle or a robot which moves at least partially automatically, in particular a lawnmower robot, the method comprising: providing (504) an existing boundary (223) of the working area within an environment in which the mobile device is moving or is to be moving, wherein the existing boundary (223) has been determined based on environmental information of the environment, the environmental information having been obtained when the mobile device was first moved, providing (506) environmental information (508), in particular an image, of the environment at least in the region of the existing border, the environmental information being obtained or having been obtained when the mobile device is re-moved along the existing border and / or within the existing border, Automatically determining (524) a changed, in particular extended, boundary (233) based on the existing boundary (223) and environmental information, in particular images, acquired during the renewed movement of the mobile device, and In particular as a map, in particular for a user, the changed boundaries are provided (526). 2 . The method of claim 1 , wherein the first movement of the mobile device comprises an externally controlled movement of the mobile device. 3 . The method according to claim 1 , wherein the environmental information of the existing boundary is obtained or has been obtained by means of SLAM, in particular visual SLAM.

4. The method according to claim 3, wherein the existing boundary (223) is determined and / or refined or has been determined and / or refined using loop closure.

5. The method of any of the preceding claims, wherein the first movement of the mobile device comprises an externally controlled movement of the mobile device.

6. The method according to any of the preceding claims, wherein the re-moving of the mobile device comprises an automated moving of the mobile device.

7. The method according to any one of the preceding claims, further comprising: obtaining (528) correction information (530) from an external source, and An adjusted boundary is determined (532) based on the boundary and / or the changed boundary and based on the correction information.

8. The method according to any one of the preceding claims, further comprising: determining whether a deviation between the changed boundary and the existing boundary satisfies a predetermined criterion, In this case, the limits of the change are only provided if the deviation meets the predefined criterion.

9. The method according to any one of the preceding claims, further comprising: determining (534) navigation information (536) for the mobile device based on the existing boundary and / or the changed boundary, determining (538) control information (540) for moving the mobile device based on the navigation information, and The control information is provided and / or the mobile device is moved based on the control information.

10. A computing unit (108) comprising means for executing the method according to any one of the preceding claims.

11. A mobile device (100) configured to obtain control information determined by means of the method according to claim 9, and / or comprising a calculation unit according to claim 10, and A mobile device having a drive system (104) and a control or regulating unit (102) for controlling the drive system, and in particular a detection device (106) for detecting environmental information of an environment in which the mobile device moves or is to move, The mobile device is in particular designed as a vehicle that moves at least partially automatically, in particular as a passenger vehicle or as a freight vehicle, and / or as a robot, in particular as a domestic robot, such as a cleaning robot, a floor or road cleaning device or a lawn mowing robot, and / or as a drone.

12. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to claims 1 to 9.

13. A computer-readable storage medium having stored thereon the computer program according to claim 12.

Images