Method for determining boundary of working area of mobile device

Abstract

A method for determining a boundary of a working area of a mobile device. The invention relates to a method for determining a boundary (223, 233) which at least partially delimits a working area (122, 222) of a mobile device (200), in particular an at least partially automatically moving vehicle or robot, in particular a mowing robot, comprising: providing environment information (502) and / or an image of an environment in which the mobile device moves or is to move, the environment information and / or images are acquired during a movement of the mobile device or during a movement of the mobile device, the movement of the mobile device (200) comprising at least one movement part (232) in which the mobile device moves at least partially outside the working area on a base surface (221b) different from the base surface (221a) of the working area, and determining a boundary of a working area of the mobile device within the environment based on the environment information and / or the image (223).

Description

Technical Field

[0001] The invention relates to a method for determining boundaries which at least partially delimit a working area of ​​a mobile device, in particular an at least partially autonomous vehicle or a robot, in particular a lawnmower robot, a computing unit and a computer program for executing the method, and a mobile device. Background Art

[0002] Mobile or working devices, such as at least partially autonomous vehicles or robots, usually move in an environment, in particular an environment to be processed or a work area, such as an apartment, a garden, a factory hall or on the street, in the air or in water. Summary of the invention

[0003] According to the invention, a method for determining the boundaries of an operating area of ​​a mobile device, a computing unit and a computer program for carrying out the method, and a mobile device are proposed with the features of the independent patent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The invention generally relates to a mobile device that moves or at least can move in an environment or there, for example in a work area. The mobile device may also be referred to as a mobile working device. Examples of such mobile devices (or mobile working devices) are, for example, robots and / or drones and / or vehicles that move partially or (fully) automatically (on land, in water or in the air). Robots include, for example, household robots, such as cleaning robots (for example in the form of vacuum and / or wiping robots), floor or street cleaning devices, construction robots or lawn mowing robots, but also other so-called service robots as at least partially automatically moving vehicles, such as passenger vehicles or freight vehicles (also referred to as so-called land transport vehicles, for example in warehouses), and also aircraft, such as so-called drones or ships.

[0005] Such a mobile device has, in particular, a control or regulating unit and a drive unit for moving the mobile device, so that the mobile device can be moved in an environment, in particular along a movement path. To this end, navigation information, i.e., specific instructions, for example, about the direction in which the mobile device should travel in order to follow the movement path, can be determined based on the movement path. This navigation information can then be implemented by the control or regulating unit and the drive unit.

[0006] For example, a map of the environment can be used for navigation, which can be acquired or determined in particular by means of SLAM. A method in robotics is called SLAM (“Simultaneous Localization and Mapping”, roughly meaning: simultaneous localization and mapping), in which a mobile device (such as a robot) can or must simultaneously create a map of its environment and estimate its spatial position within this map. The method is therefore used to identify obstacles and thus support autonomous navigation.

[0007] Furthermore, the mobile device or working device may have one or more sensors, by means of which the environment or information in the environment can be acquired. This may be, for example, a camera, a lidar sensor or even an inertial sensor, by means of which the environment and / or the movement of the mobile device can be acquired, for example, in two or three dimensions.

[0008] Furthermore, such a mobile device can be configured to receive data and / or send data by means of a communication connection, i.e. generally communicate or exchange data. In this way, the communication with the mobile device can, for example, issue instructions to the mobile device, transmit other data to the mobile device or receive data or information from the mobile device. In this context, wireless communication connections are particularly contemplated. For this purpose, the mobile device can, for example, have corresponding (possibly different) modules for wireless communication, but the modules can also be integrated into the computing unit, for example.

[0009] In the case of such mobile devices, it is also common that there are functions that the mobile device can perform, i.e. the mobile device is set to perform one or more, preferably different functions. In addition to functions, applications can also be discussed. Such functions can be, for example, work functions or learning functions.

[0010] The working function includes, for example, the automatic movement of the mobile device in the environment and the execution of a working process in the environment at least temporarily during the movement in the environment. In the case of a lawn mower robot, the working function may thus include, for example, mowing the lawn (during the movement). For this purpose, the working function may be executed, for example, by sending corresponding data via a wireless communication connection, for example, issuing a start command to the mobile device.

[0011] The learning function includes, for example, manually controlled movement of the mobile device in an environment, in particular in which no work process is performed. In the case of a mowing robot, the learning function can therefore, for example, include (manual) movement or control along the boundaries of the working area in order to teach the mowing robot the boundaries. For this purpose, control commands for navigating or moving in the environment can be (continuously) issued to the mobile device, for example, by sending corresponding data via a wireless communication connection. This is also referred to here as learning or so-called "Teach-In".

[0012] One aspect of this learning is that a boundary (or limit) should be found or predefined, which surrounds the working area, ie, for example, a garden. In the case of a working function, this boundary can then be used to limit the automatic movement of the mobile device.

[0013] For a mobile device, a possibility is proposed to determine a boundary which at least partially delimits a working area of ​​the mobile device.

[0014] For this purpose, in a first step, environmental information of the environment in which the mobile device moves or should move is provided. This environmental information can be, for example, position, orientation, etc., which can be used, for example, for SLAM. However, in particular, the environmental information can also include an image or an environmental image, which is, for example, collected by a camera of the mobile device or collected by a camera of the mobile device. Here, the environmental information is acquired (in the first step) during the external controlled movement of the mobile device, and in particular during the manual movement of the mobile device, as mentioned above with respect to the learning function. Here, the mobile device can therefore be controlled, for example, via a smartphone and a Bluetooth connection. The user can therefore, for example, control the mowing robot at least approximately or as far as possible along the edge of the desired working area and follow the mowing robot here. The movement of the mobile device for the first step can also be referred to as the first movement.

[0015] Based on the environment information from the first step, the boundaries of the working area of ​​the mobile device within the environment are then determined. This can be a temporary or rough boundary, but in principle it can be used for working functions. Since the boundary is based on manual control of the mobile device, the boundary may not be very accurate.

[0016] The environmental information from the first step can be obtained here, in particular, by means of SLAM, in particular visual SLAM. For this purpose, for example, the collected images can be processed accordingly. Preferably, the boundary is determined and / or refined based on the environmental information from the first step using loop closure.

[0017] So-called loop closure is an aspect related to SLAM. In this case, it is based on: the mobile device reaches a specific position in the environment where it has been previously located during its movement in the environment or in the work area. According to environmental information such as images or distance information seen from the corresponding position of the mobile device, and by comparing such environmental information with earlier environmental information, it can be determined whether the mobile device has been in a position before or the mobile device has been in a position before. If this is determined, the map created within the scope of SLAM or the path of the mobile device marked as traveled therein must also indicate that the mobile device is located at the same position here. However, due to possible deviations, this is often not the case. More precisely, the current position in the created map may be different from the earlier position. Here, the map and therefore the boundary can now be adjusted so that two positions separated from each other are consistent again. In this case, it is called so-called loop closure.

[0018] In this respect, it is also particularly expedient if the externally controlled movement of the mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a predetermined tolerance. For example, the starting point and the end point can be at a docking station. Thus, loop closure can be performed and the boundaries can be determined particularly well in a first step.

[0019] In a second step, then, environmental information, in particular an image, of the environment is provided at least in the border region. This environmental information is acquired when the mobile device is moved again along and / or within the border. The further movement is in particular an automatic movement, i.e., for example within the scope of the mentioned working functions, i.e., processing can be performed here. For example, this can include a first trimming of the (possibly entire) working area. This (further) movement of the mobile device for the second step can also be referred to as a further movement.

[0020] Based on the environmental information from the second step, the boundary (obtained in the first step) is then confirmed and / or adjusted. If the boundary is confirmed, the boundary obtained in the first step can be further used, so to speak, verified. When adjusting the boundary, the boundary obtained in the first step can be adjusted, for example, based on the environmental information obtained in the second step. Thus, for example, based on the environmental information (preferably an image) from the second step, the following information can be obtained, namely that there is still a lawn at a certain position outside the boundary from the first step, so that the boundary there can be adjusted accordingly. It goes without saying that part of the boundary can be confirmed and other parts of the boundary can be adjusted.

[0021] As mentioned, the environmental information from the second step may include an image of the environment. In one embodiment, confirming and / or adjusting the boundary then includes determining a map of the environment including at least a portion of the boundary based on the image from the second step. For example, this can be done using an orthographic projection.

[0022] In this way, in the first step, it is not necessary or at most only a few images need to be collected and stored in order to use them to create a map later. More precisely, the determination of the boundaries is carried out only by (visual) SLAM. This saves computing resources. The environmental information from the second step, in particular the images, can then be used for verification or adjustment. Since the second step can be carried out within the scope of the working process, the mobile device with the actual working function can be used more quickly overall, compared to, for example, additional steps for verification that may be taken (before the working function).

[0023] However, as an alternative to these two steps, environmental information and images of the environment in which the mobile device is moving or should be moving can also be provided, which are acquired during external controlled movement of the mobile device or are acquired during external controlled movement of the mobile device. Based on the environmental information and images, the boundaries of the working area of ​​the mobile device in the environment are then determined. In this case, images are therefore also acquired and, if necessary, stored and processed. This allows particularly precise boundaries to be determined in just one movement of the mobile device, although more computing resources may be required here. Compared to the above two-step approach, only the first step is required here, but the image is also acquired there.

[0024] In one embodiment, in this case, environmental information, in particular an image of the environment, can also be provided at least in the border area, which environmental information is acquired when the mobile device is moved further (in particular automatically) along the border and / or within the border or when the mobile device is moved further (in particular automatically) along the border and / or within the border. During the further movement, the mobile device in particular performs a processing process. This can itself correspond to the second step of the previously described variant. With the environmental information or image, it is then not necessary to confirm and / or adjust the border (although this may be the case). More precisely, a map of the surrounding area can be determined in this way, which includes at least a part of the border; as already mentioned above, this can be done, for example, using an orthographic projection.

[0025] As has already been shown, particularly precise boundaries can also be acquired by targeted movement of the mobile device, and this depends, for example, on whether the boundary is determined using two steps or only one step.

[0026] In this case, environmental information and / or images of the environment in which the mobile device moves or should move are provided. Here, the environmental information and / or images are acquired when the mobile device moves: this movement can be external, manual or automatic.

[0027] However, the movement of the mobile device here includes at least one movement section in which the mobile device is at least partially moved outside the working area on a base surface that is different from the base surface of the working area (or the expected working area). This can be, for example, a path or a road surface, while the (expected) working area has, for example, a lawn as a base surface. In other words, the mobile device is thus moved in a targeted manner outside the (actual or expected) working area, whereas this usually only occurs within the (actual or expected) working area.

[0028] The boundaries of the working area of ​​the mobile device within the environment are then determined based on the environment information and / or the image. By means of this targeted movement of the mobile device outside the (actual or desired) working area, the edges of the (desired) working area can be detected more accurately, so that the boundaries (e.g. preliminary or final boundaries) can also be determined significantly more accurately. It should be mentioned here that the boundaries do not have to be directly based on the movement of the mobile device, e.g. externally controlled movement, but can be adjusted accordingly if necessary to limit the boundaries to the (desired) working area.

[0029] It is advantageous if a base surface different from the base surface of the working area adjoins the base surface of the working area and / or has the same height within a predetermined tolerance. The height in this case depends in particular on the direction of gravity. In this way, the mobile device can be moved without problems from one base surface to the other, or even simultaneously on both base surfaces.

[0030] It should be mentioned that usually only part of the boundary can be determined in this way, since for example parts of the base surface outside the (desired) working area may not be traversable; for example, fences, walls or water may limit the (desired) working area in certain places.

[0031] As mentioned, the environment map can be determined using an orthographic projection, i.e. a visual map. A visual map is to be understood here in particular as a map that displays the surroundings in an image format, rather than a SLAM map with nodes and edges, etc. In this case, it can also be referred to as a structure map, which displays the structure of the environment, for example areas with lawn and areas without lawn.

[0032] For this purpose, an image of the environment is provided, wherein an image of the movement of a mobile device in the environment is acquired or has been acquired. The movement may be external, manual or automatic. An orthographic projection view of the image of the environment is then determined. Based in turn on the orthographic projection view of the image of the environment, a visual map of the environment is then provided; the visual map can then be used as explained above.

[0033] In one embodiment, a visual map of the environment is determined using a computer vision algorithm, such as, in particular, segmentation, for example, semantic segmentation. By applying the computer vision algorithm, the image can be preprocessed. In particular, continuous regions in the image can be determined, which is the case in particular in the case of segmentation. Here, the image of the environment can be segmented before determining the orthographic projection view, or generally processed with the aid of a computer vision algorithm to obtain a segmented image of the environment or a generally preprocessed image. The orthographic projection view of the image of the environment is then determined based on the segmented image or the preprocessed image of the environment.

[0034] However, the orthographic view of the image of the environment may also be determined before segmentation or application of a computer vision algorithm. The orthographic view of the image of the environment is then segmented or pre-processed.

[0035] In one embodiment, the segmentation is performed with the aid of a machine learning algorithm, such as an artificial neural network. However, the classical application of computer vision algorithms can also be implemented.

[0036] If the mobile device is a lawn mowing robot and the image of the environment includes an image of grass (or lawn), the boundaries of the grass may be determined when segmenting or applying a computer vision algorithm.

[0037] In this way, the boundaries of the (desired) working area (for example, the lawn) can be determined particularly accurately, so that the visual map can represent the working area particularly accurately. However, in particular, the boundaries can be determined particularly accurately, that is, for example, with regard to the boundary between the grass and other base surfaces. As mentioned above, this allows the boundaries to be determined or adjusted particularly accurately.

[0038] The orthographic projection of an image (or a preprocessed or segmented image) is to be understood here in particular as the projection of the image onto a base surface, i.e. from top to bottom. In this context, it should be taken into account that the camera of a mobile device, such as a lawn mower, is usually aligned at least approximately horizontally. Therefore, in an image captured or recorded with such a camera, the background is only displayed at the bottom edge and is distorted. Taking into account the camera parameters, such as the specific arrangement and field of view of the camera at the mobile device, the part of the image that represents the base surface can be corrected and represented in the sense of a top view. In this representation, the orthographic projection, the specific and precise boundaries of the working area can be clearly better recognized.

[0039] In segmentation, in particular so-called semantic segmentation, each image point (pixel) in an image can be assigned one of several classes, i.e. the image point is labeled or classified. To this end, features of the image point are usually first determined or extracted, and the class is then assigned based on these features. Such features can be, for example, shape, color, context, pattern, light changes and / or image context. Thus, for example, a pixel can be assigned the color green, and the surrounding image points can also all be green. This could indicate a lawn.

[0040] Different features are considered as classifications. Taking a lawnmower robot as an example, two classes can be used in a simple case, namely "lawn" and "non-lawn", i.e. for each image point in the image it is determined whether it shows lawn or not. Of course, more than two classes can also be used, in order to, for example, also recognize roads, streets, buildings, vehicles or people and assign them to image points if necessary. It is also conceivable that - based on the class "non-lawn" - a class "background" is assigned to which all content that is not assigned to another class is assigned. Compared to pure object recognition, which in this case only recognizes objects in the image, more information is added by all these labelings.

[0041] In principle, it is not necessary to classify each individual pixel, but rather it is also possible to classify segments or parts of the image, each of which includes a plurality of pixels. Such segments can be predetermined or can also be formed within the scope of semantic segmentation, and if necessary also have different sizes within the image. For this purpose, one or more artificial neural networks or generally artificial intelligence-based pattern recognition methods are preferably used.

[0042] Segmenting the image thus allows to more easily and quickly determine where the boundaries between different base surfaces are.

[0043] In one embodiment, a changed, in particular extended, boundary is determined, and this is determined based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. The determination of the changed boundary is here in particular automatic. The changed boundary is then provided to the user, in particular as a map. Thus, for example, a suggestion for a modified boundary can be provided. For this purpose, the changed boundary can be displayed, for example, on a smartphone or other input device.

[0044] In one embodiment, it is provided that a modified, in particular extended, boundary is determined based on the boundary from the first step and / or the environmental information, in particular the image, from the second step. The determination of the modified boundary is here in particular automatic. The modified boundary is here determined as an adjusted boundary. Here, the improvement of the boundary can therefore be carried out automatically and in particular without the need for a user to make a change.

[0045] The determination of the changed boundaries can also be independent of how exactly the existing boundaries were acquired. For this purpose, existing boundaries of the working area within the environment in which the mobile device is moving or is to move can be provided. The existing boundaries are determined here at least based on environmental information of the environment, which was acquired during the first movement of the mobile device.

[0046] Environmental information, in particular an image, of the environment is then provided at least in the area of ​​the existing border. The environmental information is then acquired when the mobile device is moved again, in particular automatically, along the existing border and / or within the existing border. The changed border is then automatically determined and determined based on the existing border and the environmental information, in particular the image, which was acquired when the mobile device was moved again. In particular, the changed border is provided to the user, in particular as a map (e.g. the mentioned visual map). For example, this can be done on a smartphone or other input or display device. The user can thus obtain suggestions for improved borders.

[0047] The changed boundary can be, in particular, an expanded boundary or an improved boundary. Thus, a better boundary can be obtained particularly easily and quickly (due to automation), which allows a more precise coverage of the desired working area.

[0048] It can also be determined whether the deviation between the changed boundary and the existing boundary meets a predetermined criterion; for example, the average distance or cumulative distance of the two boundaries can be determined and compared to a threshold. The changed boundary is only provided if the deviation meets the predetermined criterion. That is, for example, a suggestion is only made to the user if it seems to make sense.

[0049] Furthermore, it can be provided that correction information is obtained from the outside. For this purpose, the user can, for example, make and submit a change to the suggestion based on the suggestion. The adjusted boundary is then determined based on the boundary (depending on which one exists, this can also be the existing boundary and the proposed changed boundary) and based on the correction information.

[0050] In one embodiment, it is provided that the environment information from the first step includes information about reference points in the environment, in particular images. The information about the reference points can then be used, in particular when determining the boundaries and / or when determining the visual map. Such reference points can be, for example, checkpoints, in particular also only a few checkpoints, which are not along the entire route, but for example at specific intervals and / or at specific maneuvers (90° bends, etc.). This enables a more precise verification or adjustment of boundaries or even maps, while only a small number of images need to be stored in the first step.

[0051] In addition to the above-mentioned possibilities for determining a boundary that at least partially delimits the working area (this applies in particular to an external boundary), it is also possible to determine an additional boundary, in particular an internal boundary, i.e. a boundary that is, for example, surrounded by the existing boundary, based on the existing boundary (which is then determined, for example, as described above). Such an additional boundary can, for example, exclude trees or other obstacles present in the lawn from the working area to be processed.

[0052] To this end, environmental information (e.g., position, orientation) and / or images of the environment in which the mobile device moves or should move are provided. These environmental information or images are acquired when the mobile device moves. Here, the movement of the mobile device includes movement starting from a starting point, wherein the starting point corresponds to the following point in the environment, at which the mobile device stops or stops at an existing boundary or along an existing boundary (this can be a boundary as described above, whether it is adjusted or changed). The movement of the mobile device is in this case especially automatic. The stopping of the mobile device is in this case actively and / or manually (i.e. consciously) performed.

[0053] Then, based on the environment information and / or the image, and if necessary also based on existing boundaries, additional boundaries of the working area of ​​the mobile device within the environment are determined. In other words, the mobile device is stopped during its (automatic) movement along an already existing boundary or at least (temporarily) at such an existing boundary, and from there it is controlled, for example, around an obstacle. In this case, the mobile device is at least partially controlled externally (for example manually), for example as described above for the first step. In this way, the position of this additional boundary relative to the existing boundary can be known.

[0054] Preferably, the movement of the mobile device also comprises a movement to an end point, wherein the start point and the end point are identical at least within a predetermined tolerance.Additional boundaries can thus be optimized, for example by means of loop closure.

[0055] The movement from the starting point to the intermediate point preferably comprises an externally controlled (e.g. manual) movement of the mobile device, while the movement from the intermediate point to the end point comprises, for example, an automatic movement of the mobile device. In particular, the intermediate point corresponds, at least within a predetermined tolerance, to a point that the mobile device has already passed during the movement from the starting point to the intermediate point. In this way it is ensured that the mobile device completely avoids obstacles.

[0056] Furthermore, for the additional boundaries what has already been done for the other boundaries then applies, ie this can be determined, for example, with the aid of SLAM.

[0057] Navigation information for the mobile device can then be determined based on the boundaries (if necessary also additional and / or changed or adjusted boundaries) and / or the (possibly visual) map of the environment. Based on the navigation information, control information for moving the mobile device can in turn be determined. The control information can then be provided and / or the mobile device can be moved based on the control information.

[0058] A computing unit according to the invention, ie generally a system for data processing, for example a control unit or a control unit of a mobile device or a central server or another computing system, is configured, in particular in terms of programming, to carry out the method according to the invention.

[0059] The invention also relates to a mobile device, which is set up to acquire control information determined as mentioned above. In addition or alternatively, the mobile device has a computing unit according to the invention. In addition, the mobile device has a control or regulation unit and a drive unit for moving the mobile device. In addition, the mobile device can have a detection device for detecting environmental information of the environment in which the mobile device moves or should move, for example the mentioned camera or even a plurality of cameras.

[0060] The mobile device is preferably designed as an at least partially autonomous vehicle, in particular a passenger or freight vehicle, and / or a robot, in particular a household robot, such as a cleaning robot, a floor or street cleaning robot or a lawn mowing robot, and / or a drone.

[0061] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for performing all method steps, because this results in particularly low costs, especially if the execution control device is still used for other tasks and therefore exists in any case. Finally, a machine-readable storage medium is provided, with a computer program as described above stored thereon. Suitable storage media or data carriers for providing computer programs are especially magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download programs via computer networks (the Internet, intranets, etc.). Such downloading can be carried out here by wired or cabled or wireless (e.g., by WLAN networks, 3G, 4G, 5G or 6G connections, etc.) mode.

[0062] Further advantages and embodiments of the invention are apparent from the description and the accompanying drawings.

[0063] The invention is schematically illustrated in the drawings based on exemplary embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1a A mobile device is schematically shown for explaining the present invention.

[0065] Figure 1b Another view schematically shows Figure 1a Mobile devices in .

[0066] Figure 2a , 2b 2c schematically shows an environment with a mobile device for explaining the present invention.

[0067] Figure 3a , 3b 3c and 3d are schematic diagrams for explaining embodiments of the present invention.

[0068] Figure 4a , 4b A map on an input device is schematically shown.

[0069] Figure 5 The flow of a method for explaining an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0070] exist Figure 1a In FIG. 1 , a mobile device 100, in particular a working device, is schematically and exemplarily shown to explain the present invention. Figure 1b , the mobile device 100 is shown in another view and in other aspects. Figure 1a and 1b .

[0071] The mobile working device 100 is exemplarily a mowing robot with a control or regulating unit 102 and a drive unit 104 (with wheels) for navigating or moving the mowing robot 100 in an environment 120 and there, in particular in or on a working area 122 (e.g. a lawn or a garden). The mowing robot 100 can move or be moved, for example, along a movement path or trajectory 130. In addition, the mowing robot 100 exemplarily has a sensor 106 designed as a camera. With the aid of the camera 106, images of the environment can be acquired, which can be used again for navigation. In addition, a docking station 110 is provided exemplarily, at which the mowing robot can be charged, for example.

[0072] exist Figure 1b In FIG. 1 , buildings 124 , trees 126 and persons 128 are also depicted in the environment by way of example, which can be taken into account as objects or obstacles when navigating the mowing robot.

[0073] Furthermore, the mowing robot 100 has a computing unit 108, for example a control device, by means of which data can be received and / or sent. As already mentioned, this can be done, for example, by means of various types of wireless communication connections. These types of wireless communication connections are Figure 1a 170, 172, 174. By way of example, these are mobile radio connection 170, Bluetooth connection 172 and WLAN connection or WiFi connection 174. For this purpose, computing unit 108 can have corresponding radio modules or be connected to them, which are then part of mowing robot 100.

[0074] In addition, Figure 1a In the embodiment of the present invention, a mobile input device 140 (e.g. a smartphone), a central computing system 150 (or a server, which can represent a so-called cloud) and a WLAN router 160 are located in a building 162 (which can be a different building from the building 124 or the same building). Wireless communication connections 170, 172, 174 of the type already mentioned are provided between the mowing robot 100 or its computing unit 108, the mobile input device 140, the computing system 150 and the WLAN router 160. It should be noted that the mobile radio connection 170 is established via a mobile radio transmitter 152, which in turn is connected to the computing system 150.

[0075] The Bluetooth connection 172 is provided between the mowing robot 100 or its computing unit 108 and the mobile input device 140, ie a Bluetooth connection can be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140. It goes without saying that the mobile input device 140 has a suitable radio module for this purpose.

[0076] The WLAN connection 174 is provided between the mowing robot 100 or its computing unit 108 and a WLAN router, i.e., a WLAN connection can be established between the mowing robot 100 or its computing unit 108 and a WLAN router 160. It goes without saying that the WLAN router 160 has a suitable radio module for this purpose. The WLAN router 160 can in turn be connected to the Internet via, for example, a wired connection.

[0077] A mobile radio connection 170 is provided between the mowing robot 100 or its computing unit 108 and the computing system 150, i.e. a mobile radio connection can be established between the mowing robot 100 or its computing unit 108 and the computing system 150. It should be mentioned here that such a mobile radio connection 170 is established by the mowing robot 100 or its computing unit 108 or a radio module therein, for example, with a mobile radio transmitter 152 or a corresponding mobile radio station, to which the computing system 150 is in turn connected, for example, also via a mobile radio and / or a wired connection.

[0078] Furthermore, a mobile radio connection 170 is provided between the mobile input device 140 and the computing system 150, i.e. a mobile radio connection can be established between the mobile input device 140 and the computing system 150. It also applies here that such a mobile radio connection 170 is established by the mobile input device 140 or a radio module therein, for example with a mobile radio transmitter or a corresponding mobile radio station, to which the computing system 150 is in turn connected. It should be mentioned that the mobile input device 140 includes a radio module for WLAN for this purpose, which is usually the case in the case of a typical smartphone.

[0079] Likewise, a WLAN connection 174 may be provided between the mobile input device 140 and the computing system 150, i.e. a WLAN connection may be established between the mobile input device 140 and the computing system 150. To this end, a WLAN connection 174 may be provided between the mobile input device 140 and a WLAN router. As already mentioned, the WLAN router 160 in turn may be connected to the Internet via, for example, a wired connection and to the computing system 150 via a wired connection.

[0080] Thus, the mowing robot 100 or its computing unit 108 can send and receive data, i.e. exchange data, via each of the mentioned wireless communication connections 170, 172, 174. With the mobile input device 140, data can be exchanged here not only via a Bluetooth connection, but also via a mobile radio connection 170 or a WLAN connection 174, i.e. indirectly via the computing system 150. It should be mentioned that in principle a WLAN connection can also be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140.

[0081] The mobile working device or mowing robot 100 can be set up to perform one or more functions. In one embodiment, these functions include working functions, such as mowing functions and learning functions.

[0082] The working functions include, for example, the automatic movement of the mowing robot 100 in the environment 120 and the execution of a working process (or processing process) in the environment, i.e., for example, mowing the lawn at least partially while moving in the environment (depending on the situation, the mowing robot may also first drive to a specific location without mowing the lawn in order to start or continue mowing the lawn there).

[0083] The mobile radio connection 170 can be assigned to the working function. This means that the mowing robot 100 or its computing unit 108 acquires and / or sends data required for performing the working function at least partially via the mobile radio connection. This can, for example, include a start command for starting a mowing process, which is sent from a mobile input device (possibly via the computing system 150) to the mowing robot 100.

[0084] This can also include acquiring, for example, a trajectory (or navigation information in general) from the computing system 150, which is determined in the computing system 150 and which the mowing robot should subsequently follow for the mowing process. However, it can also be provided that the trajectory (or navigation information in general) is determined in the mowing robot 100 or its computing unit 108 itself, or that the trajectory is acquired there in another way. However, instead of the trajectory, the computing unit 108 can also (only) acquire control information that has been determined based on the trajectory (which in turn can have been determined in the computing system 150), and based on this control information, the control or regulating unit 102 can move the mowing robot 100 via the drive unit 104 in order to follow a trajectory, such as the trajectory 130 indicated here only by way of example.

[0085] The mowing robot 100 can then move independently, for example, over the working area 122 or in the environment 120 or navigate there and mow the lawn. As already mentioned, various objects can be detected by the mowing robot 100 or its camera 106 (so they are visible in the images recorded by means of the camera) and then be taken into account during navigation, that is, when determining navigation information. For this purpose, the images detected here or the corresponding data can be transmitted to the computing system 150, for example, via the mobile radio connection 170.

[0086] The learning function includes, for example, manually controlled movement of the mowing robot in the environment, in particular movement along the boundaries of the working area 122, in order to teach the mowing robot the boundaries. In this case, the working function, i.e. mowing, does not necessarily have to be used (or should be used). For this purpose, for example, control commands (or steering commands) for navigation or movement can be sent (continuously) to the mowing robot 100. Such control commands or corresponding data are sent to the mowing robot from the mobile input device 140 via the Bluetooth connection 172. Likewise, the mowing robot 100 or its computing unit 108 can send data back to the mobile input device 140 via the Bluetooth connection. In this case, this is also called learning or so-called "Teach-In".

[0087] exist Figure 2a In order to explain the present invention, an environment 220 having a building 224 and a tree 226 is shown, similar to the environment 220 according to Figure 1a , 1b In addition, the desired working area, such as a lawn to be mowed, is shown by way of example with 221. The mobile device or lawn mowing robot 200 and the docking station 210 are also shown again by way of example.

[0088] It is then generally desirable that the desired working area 221 is covered completely and as accurately as possible by the mobile device during processing, so that as far as possible the entire lawn is mowed.

[0089] For this purpose, the boundaries of the (actual) working area are now to be determined, which can then be processed by the mobile device. In other words, the boundaries or limits within which the mobile device should move and within which processing should be performed are to be determined.

[0090] Within the scope of the aforementioned learning function, the mobile device 200 can now be controlled, for example by means of a smartphone of the user, from the docking station 210 along the edge of the desired working area 221 until the mobile device 200 reaches the docking station 210 again.

[0091] Based on this movement and the environmental information (eg images) acquired here, a boundary 223 of the (actual) working area 222 can be determined in a first step, as shown in FIG2 . It can be seen in this case that this boundary 223 deviates from the actual edge of the desired working area 221 .

[0092] This may be caused in particular by the user moving the mobile device 200 too far from the actual edge of the desired working area 221. This is shown by way of example at the bottom left and top right in FIG. 2, where the border 223 is particularly far from the actual edge of the desired working area 221.

[0093] It should be mentioned here that when the mobile device 200 is moved, the position of the mobile device is determined relative to a center point if necessary, while the geometric dimensions of the mobile device or its processing tool are taken into account for the boundaries.

[0094] However, in a second step, environmental information, in particular images of the environment, can now be acquired again, i.e., when the mobile device 200 is used again, along the boundary 223 and / or within the boundary 223. This can be the case, for example, within the scope of the mentioned working functions, in which the mobile device is moved according to the trajectory 330.

[0095] Based on the surrounding information or images acquired here, the boundary 223 can then be confirmed and / or adjusted. Figure 2a It is shown at the bottom left and top right in FIG. 2 that the above-mentioned boundary 223 is particularly far from the actual edge of the desired working area 221 . The adjusted boundary is marked with 233 there.

[0096] If the mobile device 200 is moving along the trajectory 330 at this time, for example, an image from the lower left position is also collected. By analyzing these images, it can be seen that the actual edge of the desired working area 221 deviates from the boundary 223.

[0097] Based on these images, a map of the environment including at least part of the boundary may also be determined.

[0098] As already mentioned, instead of these two steps, it is also possible to use only one step in which the mobile device is controlled from the outside, wherein an image of the surroundings is then also acquired.

[0099] exist Figure 2b In the Figure 2a 231 is now used to indicate a route or movement path along which the mobile device 200 moves, for example during an externally controlled manual movement. It can be seen here that the mobile device comprises a movement portion or movement path portion 232, wherein the mobile device moves at least partially outside the (desired) working area 221, i.e. on a base surface 221b that is different from the base surface 221a of the working area. For example, the base surface 221a can be a lawn, while the base surface 221b can be a route or a road surface.

[0100] By this movement, the boundary indicated here by 223′ can be determined more precisely, i.e. without having to perform the movement outside the working area 221. It should be mentioned here that the boundary 223′ is shown here as being slightly spaced apart from the edge of the (desired) working area 221, although this can clearly be more consistent in practice.

[0101] exist Figure 2c In the Figure 2a The part of the environment 220 in the embodiment. The determination of the additional boundary should now be explained here. For example, the boundary 223 determined as described above can be assumed to be an existing boundary.

[0102] During the movement of the mobile device 200 along the border 223, and in particular the automatic movement, the mobile device can stop, and for example at point P1. From there, i.e. point P1 as the starting point, the mobile device 200 can then be manually controlled around the tree 226, the obstacle, for example via the mentioned Bluetooth connection to the smartphone; this is shown by routes 235 and 236. Suitably, this manual movement takes place up to point P3, an intermediate point, which is at least within a predetermined tolerance of the point that has been traveled before (here point P2) (i.e. on routes 235, 236). From there, the mobile device can then, for example, automatically drive again to the end point, here point P4, which coincides with the starting point at least within a predetermined tolerance. This is shown by route 237.

[0103] In this way, an additional boundary can be determined, and in particular as a route portion located between points P2 and P3; in this case this is an at least approximately closed loop; Figure 2c , this additional boundary corresponds to route 236.

[0104] Boundary 223 or 233 is therefore in particular an outer boundary, while boundary 236 can be referred to as an inner boundary.

[0105] exist Figure 3a , 3b , 3c, 3d, images are schematically shown for explaining the mentioned orthographic projection and segmentation as examples of application of computer vision algorithms, as can be used in embodiments of the present invention.

[0106] exist Figure 3a 3 shows an image 300a that can be captured by a mobile device or its camera. Figure 2a 2b. Here, in particular, a building 324 can be seen in the front view, and a desired working area 322 can be seen at the bottom edge, with, for example, a lawn as a base surface 321a. A route with a different base surface 321b extends next to the working area 322.

[0107] As can be seen from the image 300 a , it may be difficult to determine a boundary directly based thereon, which should correspond as precisely as possible to the desired working area 322 or its edge.

[0108] exist Figure 3b , which is an orthographic projection of image 300a. Here, it can be seen that a desired working area 322 with a lawn as a base surface 321a and a route with other base surfaces 321b are shown in a top view (or a top-down view). There, the geometrical relations and the size or course of the edge of the desired working area 322 (i.e., the line denoted here by 322) at least approximately correspond to the actual situation.

[0109] exist Figure 3c 3 shows an image 300c, which is a semantic segmentation of image 300a. Here, only two classes are distinguished, namely, for example, lawn 321a and background or the rest, which is not shown here. There, for example, buildings, roads and all other objects outside the lawn fall under it. It can be seen here that

[0110] Figure 3d 300d is shown, which corresponds to image 300a with semantic segmentation and orthographic projection. This can be obtained, for example, from image 300b by semantic segmentation or from image 300c by orthographic projection. The results are at least approximately the same. It goes without saying that various subtle differences may occur.

[0111] In any case, the orthographic projection of the image, in particular with (semantic) segmentation, allows particularly accurate determination of the edges of the (desired) working area, with which the boundaries can be particularly accurately determined. The edges can also be used to determine an accurate (visual) map of the environment, in which at least parts of the boundaries are displayed, for example. The boundaries can then be displayed to the user, for example on a smartphone.

[0112] Figure 4a An input device 440, such as a smartphone, is shown by way of example in FIG. Figure 1a In addition to controlling the mobile device during learning, a visual map 420 can also be displayed on it, as exemplarily shown in Figure 4a . There, for example, one can then see an area 421a of grass or lawn and an area 421b outside. In addition, a line 423 is shown in the map, which can correspond, for example, to the boundary created in the aforementioned first step; this can correspond, for example, to the boundary according to Figure 2a Boundary 223.

[0113] Furthermore, a line 433 is shown, which may represent a modified, in particular extended and / or improved boundary; this may correspond, for example, to Figure 2a As mentioned, such a changed or extended boundary can in particular be determined automatically and then displayed as a suggestion for the user in the map, as here in Figure 4a as shown in .

[0114] The user may then confirm that the user agrees with the suggestion of boundary 433, for example by a suitable input command at input device 440. Likewise, the user may reject the suggestion with boundary 433. Boundary 423 may then be retained, for example.

[0115] Likewise, users can suggest changes, for example. Figure 4b 424 is shown as an example, and a line 434 is used for the boundary of the change. To this end, the user can move the line 433 (for example at a suitable point of the line) by means of a touch display of an input device 440, for example, so that this produces a line 424.

[0116] Figure 5 The flow of a method for determining a boundary, which at least partially defines a mobile device (ie, for example, according to Figure 1a , 1b The working area of ​​the lawn mowing robot 100 of 2a, 2b, 2c is shown in FIG.

[0117] To this end, in step 500, a first step, environmental information 502 of the environment in which the mobile device moves or should move is provided. This environmental information may include, for example, position, orientation, and has been determined, for example, based on images acquired by means of a camera of the mobile device. For example, this can be done by means of SLAM, in particular visual SLAM. In this case, the mobile device is controlled from the outside, as described above, for example with reference to Figure 2a The mobile device can also be moved, in particular, outside the desired working area on another surface, for example with reference to Figure 2b Explained.

[0118] In step 504, the boundaries of the working area within the environment of the mobile device are then determined based on the environment information 502 from the first step, such as this, for example, with reference to Figure 2a This can be done, for example, using loop closure, especially when the start and end points are identical or close to each other in an externally controlled movement, for example at a docking station. This also provides a boundary.

[0119] In step 506, a second step, then environmental information 508, such as an image, of the environment is provided at least in the border area. The environmental information or image 508 is acquired when the mobile device moves again along the border and / or within the border, as described above, for example with reference to Figure 2a Explained.

[0120] In step 510, a map of the environment, in particular a visual map, may be determined based on or from the image 508. This may be done using orthographic projection and segmentation. An example of such an image is Figure 3a An example of such a map is Figure 4a Map 420 shown in .

[0121] In this case, in step 512, orthographic projection views 514 of images of the environment can be determined from the images 508, as shown, for example, with the image 300b in FIG. 3. In step 516, these orthographic projection images 514 or views thereof can then be segmented, as shown, for example, with Figure 3d As shown in the image 300d in FIG. A map can be created and also provided thereby.

[0122] Then, based on the map or just based on the segmented ortho-projected image, the existing boundary (from the first step) can be confirmed (or verified), step 518. However, if necessary, the existing boundary can be adjusted, step 520. Then, in step 522, the boundary obtained here (i.e., such as confirmed or adjusted) can be provided, such as for further use.

[0123] However, it can also be provided as an alternative that in step 500', the environmental information 502 and the image 508 acquired during the external controlled movement are provided. In step 510', a map, in particular a visual map, can be determined based on or according to the image 508, as also described above. In particular, steps 512 to 516 can also be provided here.

[0124] A boundary may then be determined based on the environmental information and the image at step 518'. The boundary may then be provided at step 522', for example, for further use.

[0125] Starting from the boundaries provided in step 522 or 522′, or also starting from the boundaries determined or provided in step 504, in step 524 a changed, in particular extended, boundary can be determined, and this is done either on the basis of the boundaries from the first step and / or the environmental information, in particular the image, from the second step, or generally on the basis of existing boundaries, such as, for example, with reference to Figure 2a Likewise, environmental information, in particular images, from further movement (eg during a processing procedure) can then be provided.

[0126] The determination of the changed boundaries is carried out in particular automatically. Then, in step 526, the changed boundaries are provided to the user, in particular as a map, for example with reference to Figure 4a Explained.

[0127] The user can then make changes or corrections, for example, as referenced Figure 4b Then, in step 528, correction information 530 is received from the outside (e.g., from a smart phone). Then, in step 532, the boundary confirmed in step 518 or obtained in step 518' can be adjusted. However, the adjusted map can also be adjusted again according to step 520.

[0128] Likewise, an adjusted map can be obtained in this way according to step 520, that is, only an adjusted map modified by the user is obtained. Then, the adjusted boundary is determined based on the boundary and based on the correction information. Alternatively, the changed boundary can also be determined without correction information.

[0129] Based on the boundaries (e.g., first acquired, adjusted, or changed boundaries), navigation information 536 for the mobile device can then be determined at step 534. Based on this, control information 540 for moving the mobile device can then be determined at step 538 and can then be provided.

[0130] After the boundary is acquired, whether it is an initial boundary (see step 522 or 522 ′), an adjusted or changed boundary (see step 532 ), additional, for example internal boundaries can also be determined based on this as an existing boundary.

[0131] To this end, in step 550, environmental information 552 (comparable to environmental information 502, such as position, orientation) and / or images 554 (comparable to image 508) of the environment in which the mobile device moves or should move is provided. Such environmental information or images are obtained when the mobile device moves, wherein the movement of the mobile device includes movement starting from a starting point, wherein the starting point corresponds to a point in the environment at which the mobile device stops or stops when moving (especially automatically) at an existing boundary. Reference Figure 2c Explain this in more detail.

[0132] Then, in step 556, additional boundaries of the working area of ​​the mobile device within the environment are determined based on the environment information and / or the image, such as, for example, also referring to Figure 2c This additional boundary can then be provided at step 558 and used again to move the mobile device, for example from step 534. A changed boundary can also be determined in the case of an additional boundary, as described above with respect to boundary 223 and modified boundary 233.

Claims

1. A method for determining a boundary (223, 233) which at least partially delimits a working area (122, 222) of a mobile device (100), in particular a vehicle or a robot which moves at least partially automatically, in particular a lawnmower robot, the method comprising: providing (500, 500') environmental information (502) and / or an image (508) of an environment in which a mobile device moves or should move, wherein the environmental information and / or the image is acquired when the mobile device moves or is acquired when the mobile device moves, wherein the movement of the mobile device (100) comprises at least one movement section (232) in which the mobile device is at least partially moved outside the working area on a base surface (221b) different from a base surface (221a) of the working area, and Boundaries (223) of a working area of ​​the mobile device within the environment are determined (504) based on the environment information and / or the image.

2. The method according to claim 1, wherein the base surface (221b) different from the base surface (221a) of the working area is adjacent to the base surface of the working area and / or has the same height within a predetermined tolerance.

3. The method according to claim 1 or 2, wherein the environmental information is acquired by means of SLAM, in particular visual SLAM. The method according to claim 3 , wherein the boundary is determined and / or refined based on the environment information using loop closure.

5. The method according to any of the preceding claims, wherein the movement of the mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a predetermined tolerance. 6 . The method according to claim 1 , wherein the environment information comprises information about reference points in the environment, in particular images, wherein the information about the reference points is used in determining boundaries and / or in determining a visual map.

7. The method according to any one of the preceding claims, further comprising: providing environmental information, in particular an image, of the environment at least in the region of the border, which is acquired during a further movement of the mobile device along and / or within the border or during a further movement of the mobile device along and / or within the border, during which the mobile device in particular performs a processing process, and A map of the boundary is determined from the further shifted image, the map including at least a portion of the boundary.

8. The method according to claim 7, further comprising: determining (524) a changed, in particular extended, boundary (233) based on the further moved boundary (223) and / or surrounding information, in particular an image, wherein the determination of the changed boundary is in particular performed automatically, and In particular, the changed boundaries are provided (526) to the user, in particular as a map.

9. The method according to claim 8, further comprising: obtaining (528) correction information (530) from an external source, and An adjusted boundary is determined based on the boundary and based on the correction information.

10. The method according to any one of claims 1 to 7, further comprising: determining a changed, in particular an expanded, boundary based on the boundary from the first step and / or surrounding information, in particular an image, from the second step, wherein the determination of the changed boundary is performed in particular automatically, and The changed boundary is determined as an adjusted boundary.

11. The method according to any one of the preceding claims, further comprising: determining (534) navigation information (536) for the mobile device based on the boundaries and / or map of the environment, determining (538) control information for moving the mobile device based on the navigation information (540), and Control information is provided and / or a mobile device is moved based on the control information.

12. A computing unit (108) comprising means for implementing the method according to any of the preceding claims.

13. A mobile device (100) which is configured to obtain control information, the control information being determined by means of a method according to claim 11 and / or which has a calculation unit according to claim 12, and A mobile device having a drive system (104) and a control or regulating unit (102) for controlling the drive system, and in particular a detection device (106) for detecting environmental information of an environment in which the mobile device is moving or is to move, The mobile device is in particular designed as an at least partially automatically movable vehicle, in particular a passenger or freight vehicle, and / or a robot, in particular a household robot, such as a cleaning robot, a floor or street cleaning device or a lawnmowing robot, and / or a drone.

14. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to claims 1 to 11.

15. A computer-readable storage medium having stored thereon the computer program according to claim 14.

Images