Method for determining visual map of mobile device

Abstract

The invention relates to a method for determining a visual map (420) of at least a part of an environment in which a mobile device, in particular an at least partially automatically moving vehicle or robot, in particular a mowing robot, moves or should move, comprising: providing an image of the environment, the method includes providing an image of an environment, the image being to be obtained when a mobile device moves in the environment or having been obtained when the mobile device moves in the environment, determining an orthographic projection view of the image of the environment, determining a visual map of the environment based on the orthographic projection view of the image of the environment (420), and providing the visual map of the environment.

Description

Technical Field

[0001] The invention relates to a method for determining a visual map of at least a part of an environment in which a mobile device, in particular an at least partially autonomous vehicle or a robot, in particular a lawnmower robot, moves, a computing unit and a computer program for executing the method, and a mobile device. Background Art

[0002] Mobile equipment or mobile working equipment (e.g. at least partially autonomous vehicles or robots) typically moves in an environment, in particular in an environment to be processed or a working area, for example in an apartment, in a garden, in a factory hall or on the street, in the air or in water. Summary of the invention

[0003] According to the invention, a method for determining a visual map, a computing unit for carrying out the method, a computer program for carrying out the method and a mobile device are proposed having the features of the independent claims. Advantageous embodiments are the subject matter of the dependent claims and the following description.

[0004] The invention generally relates to a mobile device that moves or at least can move in an environment or, for example, in a working area there. It can then also be referred to as a mobile working device. Examples of such mobile devices (or mobile working devices) are, for example, robots and / or drones and / or vehicles that move partially or (completely) automatically (on land, in water or in the air). Robots include, for example, household robots, such as cleaning robots (for example in the form of vacuum cleaning robots and / or mopping robots), floor or street cleaning devices, construction robots or lawn mowing robots, as well as other so-called service robots, and vehicles that move at least partially automatically include, for example, passenger vehicles or freight vehicles (also called industrial trucks, for example in warehouses), but also aircraft (for example, so-called drones) or ships.

[0005] Such a mobile device has in particular a control or regulation unit and a drive unit for moving the mobile device so that it can move in an environment, in particular along a movement path. To this end, navigation information, i.e., specific instructions, for example, as to the direction in which the mobile device should travel in order to follow the movement path, can be determined based on the movement path. These instructions can then be implemented by the control or regulation unit and the drive unit.

[0006] For example, a map of the environment can be used for navigation, which can be obtained or determined in particular with the aid of SLAM. SLAM (“Simultaneous Localization and Mapping”) is a method in robotics in which a mobile device such as a robot can or must simultaneously create a map of its environment and estimate its spatial position within this map. The method is thus used to identify obstacles and thus support autonomous navigation.

[0007] Furthermore, the mobile device or working device may have one or more sensors, by means of which the environment or information in the environment can be detected. These may be, for example, cameras, lidar sensors or inertial sensors, by means of which the environment and / or the movement of the mobile device can be detected, for example, in two or three dimensions.

[0008] Furthermore, such a mobile device can be configured to receive data and / or send data by means of a communication connection, i.e. to communicate or exchange data in general. In this way, communication with the mobile device can, for example, issue commands to the mobile device, transmit other data to the mobile device, or receive data or information from the mobile device. Wireless communication connections are particularly contemplated here. For this purpose, the mobile device can, for example, have corresponding (if necessary different) modules for wireless communication, which can also, for example, be integrated into the computing unit.

[0009] It is also common for such mobile devices to have functions that can be performed by the mobile device, i.e. the mobile device is configured to perform one or more preferably different functions. Instead of functions, they can also be referred to as applications. Such functions can be, for example, working functions or teaching functions.

[0010] The working function includes, for example, the automatic movement of the mobile device in the environment and the execution of a working process in the environment during the movement in the environment at least temporarily. Thus, in the case of a lawn mower robot, the working function may include, for example, mowing the lawn (during the movement). For this purpose, for example, a start command may be issued to the mobile device to execute the working function, for example by sending corresponding data via a wireless communication connection.

[0011] The teaching function comprises, for example, manually controlled movement of the mobile device in the environment, in particular in the case where no working process is being performed in the environment. Thus, in the case of a lawn mower robot, the teaching function may, for example, comprise (manually) moving or controlling along the boundaries of the working area in order to teach the lawn mower robot this boundary. For this purpose, for example, control commands may be (continuously) issued to the mobile device to navigate or move in the environment, for example by sending corresponding data via a wireless communication connection. This is also referred to as teaching or so-called "teaching".

[0012] One aspect of this teaching is that a boundary (or limit) enclosing a working area, such as a garden, should be found or predetermined. Then, for the working function, this boundary can be used to limit the automatic movement of the mobile device.

[0013] For a mobile device, a possibility is proposed to determine such a boundary which at least partially limits the working area of ​​the mobile device.

[0014] For this purpose, in a first step, environmental information of the environment in which the mobile device moves or should move is provided. These environmental information can be, for example, position, direction, etc., which can be used for example for SLAM. However, in particular, the environmental information can also include, for example, images or environmental images that will be detected by the camera of the mobile device or have been detected by the camera of the mobile device. Here, these environmental information are obtained when the movement of the mobile device is controlled from the outside (in the first step), and in particular when the mobile device is moved manually, as mentioned above with respect to the teaching function. Therefore, the mobile device can be controlled here, for example, via a smart phone and a Bluetooth connection. Therefore, the user can, for example, control the lawn mower robot at least approximately or as far as possible along the edge of the desired working area and follow the lawn mower robot in the process. The movement of the first step of the mobile device can also be referred to as the first movement.

[0015] Based on the environment information from the first step, the boundaries of the working area of ​​the mobile device within the environment are then determined. This can be a temporary or rough boundary, but in principle the boundary can be used for working functions. Since the boundary is based on manual control of the mobile device, the boundary may not be very accurate.

[0016] The environmental information from the first step can be obtained here in particular by means of SLAM, in particular visual SLAM. For this purpose, for example, the detected image can be processed accordingly. Preferably, loop closure is used to determine and / or refine the boundary based on the environmental information from the first step.

[0017] The so-called loop closure is an aspect related to SLAM. In this case, it is based on the fact that the mobile device reaches a specific position in the environment where it was previously located during movement in the environment or in the work area. Based on environmental information such as images or distance information seen from the corresponding position of the mobile device, and by comparing such environmental information with previous environmental information, it can be determined whether the mobile device is already in the previous position or the mobile device is already in the previous position. If this is determined, the map created within the scope of SLAM or the path marked in the map as being traveled by the mobile device must also indicate that the mobile device is in the same position again. However, due to possible deviations, this is often not the case. Instead, the current position in the created map is different from the previous position. Here, the map can now be adapted and the boundaries can be adapted so that the two separated positions are consistent again. This is called the so-called loop closure.

[0018] In this context, it is also particularly expedient if the movement of the externally controlled mobile device comprises a movement from a starting point to an end point, wherein the starting point and the end point are identical at least within a predetermined tolerance. For example, the starting point and the end point can be at a connecting station. This allows the loop to be closed and the boundaries to be particularly well determined in the first step.

[0019] In a second step, environmental information about the environment, in particular an image of the environment, is then provided at least in the region of the border. This environmental information is obtained when the mobile device is moved again along the border and / or within the border. This renewed movement is in particular an automatic movement, i.e., for example within the scope of the mentioned working functions, i.e., processing can be performed in the process. This can, for example, include a first mowing of (if necessary the entire) working area. This (re)movement of the mobile device in the second step can also be referred to as further movement.

[0020] Then, based on the environmental information from the second step, the boundary (obtained in the first step) is confirmed and / or adapted. If the boundary is confirmed, the boundary obtained in the first step can be further used, which can be called verification. If the boundary is adapted, the boundary obtained in the first step can be adapted, for example, based on the environmental information obtained in the second step. Thus, for example, the following information can be obtained based on the environmental information (preferably an image) from the second step: a lawn still exists at a specific location outside the boundary of the first step, so that the boundary there can be adapted accordingly. It can be understood that some parts of the boundary may be confirmed, while other parts of the boundary are adapted.

[0021] As described above, the environmental information from the second step may include an image of the environment. Then in one embodiment, confirming and / or adapting the boundary includes determining a map of at least a portion of the environment including the boundary based on the image from the second step. This can be done, for example, using an orthographic projection.

[0022] In this way, the first step does not require, or at most only requires, the detection and storage of very few images in order to create a map therefrom later. Rather, the boundaries are determined solely by (visual) SLAM. This saves computing resources. The environmental information (in particular the images) from the second step can then be used for verification or adaptation. Since the second step can be carried out within the scope of the working process, the mobile device can be used more quickly with the actual working function than, for example, taking an additional step for verification (before the working function).

[0023] However, as an alternative to these two steps, environmental information and images of the environment in which the mobile device moves or should move can also be provided, which will be obtained or have been obtained when the movement of the mobile device is controlled externally. Based on the environmental information and images, the boundaries of the working area of ​​the mobile device within the environment are then determined. Therefore, in this case, the image is also detected and, if necessary, stored and processed. This makes it possible to determine particularly accurate boundaries during only one movement of the mobile device, although more computing resources may be required here. Compared to the above-mentioned process with two steps, only the first step is required here, but the image is also detected there.

[0024] In one embodiment, in this case, environmental information of the environment, in particular an image of the environment, can also be provided at least in the region of the border, which environmental information will be obtained during a further (in particular automatic) movement of the mobile device along the border and / or within the border or has been obtained during a further (in particular automatic) movement of the mobile device along the border and / or within the border. During the further movement, the mobile device in particular performs a processing process. This can correspond to the second step of the previously described variant. It is then not necessary to confirm and / or adapt the border using this environmental information or image (although this may be the case). Instead, a map of the environment can be determined from this, which includes at least a part of the border; as mentioned above, this can be done, for example, using an orthographic projection.

[0025] As has already been shown, a particularly accurate boundary can also be obtained by targeted movement of the mobile device, depending on whether the boundary is determined using two steps or only one step, for example.

[0026] In this case, environmental information and / or images of the environment in which the mobile device moves or should move are provided. Here, the environmental information and / or images are obtained when the mobile device moves: this movement can be carried out externally, manually or automatically.

[0027] However, the movement of the mobile device here includes at least one movement section in which the mobile device is moved at least partially outside the working area on a ground surface that is different from the ground surface of the working area (or the expected working area). For example, this can be a road or a sidewalk, while the (expected) working area has, for example, grass as the ground surface. In other words, the mobile device is thus moved in a targeted manner outside the (actual or expected) working area, whereas this is usually only done within the (actual or expected) working area.

[0028] The boundaries of the working area of ​​the mobile device within the environment are then determined based on the environment information and / or the image. By means of this targeted movement of the mobile device outside the (actual or desired) working area, the boundaries of the (desired) working area can be detected more accurately, so that the boundaries (e.g. preliminary or final boundaries) can also be determined significantly more accurately. It should be pointed out in this context that the boundaries do not have to be based directly on the movement of the mobile device, e.g. an externally controlled movement, but can be adapted if necessary to limit the boundaries to the (desired) working area.

[0029] It is advantageous if a ground different from the ground of the working area adjoins the ground of the working area and / or has the same height within a predetermined tolerance. The height in this case is particularly dependent on the direction of gravity. In this way, the mobile device can be moved without problems from one ground to another, even on two grounds at the same time.

[0030] It is pointed out that usually only a portion of the boundaries can be determined in this way, since for example ground sections outside the (desired) working area may not be accessible; for example, fences, walls or water may, for example, locally limit the (desired) working area.

[0031] As mentioned above, an orthographic projection can be used to determine a map of the environment, and this is a visual map. In this context, a visual map is to be understood in particular as a map that displays the environment in an image, rather than a SLAM map with nodes and edges etc. This can also be called a structural map, i.e. it displays the structure of the environment, e.g. with areas with grass and areas without grass.

[0032] For this purpose, an image of the environment is provided, wherein an image of the movement of the mobile device in the environment is to be obtained or has been obtained. The movement can be performed externally, manually or automatically. An orthographic projection view of the image of the environment is determined. A visual map of the environment is then provided based on the orthographic projection view of the image of the environment, and then the visual map is provided; the visual map can then be used as explained above.

[0033] In one embodiment, a computer vision algorithm is used to determine a visual map of the environment, such as in particular a segmentation, for example a semantic segmentation. By applying the computer vision algorithm, the image can be preprocessed. In particular, coherent regions in the image can be determined here, especially in the case of segmentation. Here, the image of the environment can be segmented before determining the orthographic projection view, or in general processed by means of a computer vision algorithm to obtain a segmented image of the environment, or in general a preprocessed image. The orthographic projection view of the image of the environment is then determined based on the segmented image of the environment or the preprocessed image.

[0034] However, the orthographic view of the environment image may also be determined before segmentation or application of the computer vision algorithm. The orthographic view of the environment image is then segmented or pre-processed.

[0035] In one embodiment, the segmentation is performed with the aid of a machine learning algorithm, such as an artificial neural network. However, the classical application of computer vision algorithms is also possible.

[0036] If the mobile device is a lawn mowing robot and the environment image includes an image of grass (or lawn), the boundaries of the grass may be determined during segmentation or application of a computer vision algorithm.

[0037] Thus, in this way, the limits of the (desired) working area (i.e., for example, the lawn) can be determined particularly accurately, so that the visual map can display the working area particularly accurately. In particular, boundaries, i.e., for example, the boundary between the lawn and other ground surfaces, can be determined particularly accurately. As mentioned above, this allows a particularly accurate determination or adaptation of boundaries.

[0038] The orthographic projection of an image (or a preprocessed or segmented image) is to be understood here in particular as the projection of the image onto the ground, i.e. from top to bottom. In this context, it should be borne in mind that the camera of a mobile device, such as a lawn mower, is usually oriented at least approximately horizontally. As a result, in images captured or recorded with such a camera, the ground is only displayed at the bottom edge and is distorted. Taking into account the camera parameters (e.g. the specific arrangement of the camera on the mobile device) and the field of view, etc., the part of the image showing the ground can be corrected and displayed in a top view. In this display, i.e. in the orthographic projection, the specific and exact boundaries of the working area can be clearly better recognized.

[0039] In segmentation, in particular so-called semantic segmentation, each image point (pixel) in the image can be assigned one of several classes, i.e. the image point can be labeled or classified. For this purpose, typically features of the image point are first determined or extracted, and the class is then assigned based on these features. Such features can be, for example, shape, color, background, pattern, light changes and / or image background. Thus, for example, a pixel can be assigned the color green, and the surrounding pixels can also all be green. This indicates a lawn.

[0040] Different features are considered as categories here. In the example of a lawnmower robot, for example, two categories can be used in a simple case, namely "lawn" and "non-lawn", i.e. for each image point in the image it is determined whether this image point shows lawn. It is understood that more than two categories can also be used, for example to also recognize roads, streets, buildings, vehicles or people and assign them to image points if necessary. It is also conceivable - based on the "non-lawn" category - a "background" category, to which everything that is not assigned to another category is assigned. Compared to pure object recognition, which (only) recognizes objects in the image, more information is added by all these labelings.

[0041] In principle, the classification does not have to be performed for each individual pixel, but can also be performed for segments or parts of the image, each of which contains a plurality of pixels. Such segments can be fixedly predetermined or can also be formed within the scope of semantic segmentation, optionally also with different sizes within the image. For this purpose, one or more artificial neural networks or, in general, pattern recognition methods based on artificial intelligence are preferably used.

[0042] Thus, the segmented image allows for a simpler and faster determination of where the boundaries between different ground surfaces are.

[0043] In one embodiment, a modified, in particular extended, boundary is also determined, and based on the boundary from the first step and / or the environmental information (in particular the image) from the second step. The determination of the modified boundary is here in particular automatic. The modified boundary is then provided to the user, in particular as a map. Thus, for example, a suggestion of the modified boundary can be provided. For this purpose, the modified boundary can be displayed, for example, on a smartphone or other input device.

[0044] In one embodiment, it is provided that a modified, in particular extended, boundary is determined based on the boundary from the first step and / or the environmental information (in particular the image) from the second step. The determination of the modified boundary is here in particular automatic. The modified boundary is here determined as an adapted boundary. Thus, the boundary can be improved automatically here and in particular without the need for a user to make a change.

[0045] In particular, the modified boundaries can be determined regardless of how accurately the existing boundaries were obtained. For this purpose, the existing boundaries of the working area within the environment in which the mobile device is moving or is to be moving can be provided. The existing boundaries are determined at least based on environmental information of the environment obtained during the first movement of the mobile device.

[0046] Environmental information of the environment, in particular an image of the environment, is then provided at least in the area of ​​the existing border. This environmental information is then obtained when the mobile device is moved again, in particular automatically, along the existing border and / or within the existing border. The modified border is then automatically determined, and is based on the existing border and the environmental information, in particular the image, obtained during the re-movement of the mobile device. The modified border is provided, in particular as a map (for example, the mentioned visual map), in particular for the user. This can be done, for example, on a smartphone or other input or display device. The user can thus receive suggestions for improving the border.

[0047] The modified boundary can be in particular an extended boundary or an improved boundary. Thus, a better boundary can be obtained particularly simply and quickly (fast because automatically), which allows the desired working area to be covered more accurately.

[0048] It can also be determined whether the deviation between the modified boundary and the existing boundary meets a predetermined criterion. For example, the average distance or cumulative distance between the two boundaries can be determined and compared with a threshold value. The modified boundary is only offered if the deviation meets the predetermined criterion. Thus, for example, a suggestion is only made to the user if it seems to make sense.

[0049] Furthermore, it can be provided that correction information is obtained from the outside. For this purpose, the user can, for example, make and transmit a change to the suggestion based on the suggestion. The adapted boundary is then determined based on the boundary (depending on which boundary exists, this can also be the existing boundary and the proposed modified boundary) and based on the correction information.

[0050] In one embodiment, the environment information from the first step comprises information about reference points in the environment, in particular images. Information about reference points can then be used, in particular when determining boundaries and / or when determining a visual map. Such reference points can be, for example, checkpoints, in particular only a few checkpoints, not along the entire route, but for example at specific intervals and / or during specific maneuvers (90° bends, etc.). This makes it possible to verify or adapt boundaries or maps more accurately, while only a small number of images need to be stored in the first step.

[0051] In addition to the above-mentioned possibility of determining a boundary that at least partially limits the working area (this relates in particular to an outer boundary), it is also possible to determine an additional boundary based on the existing boundary (the existing boundary is then determined, for example, as described above), in particular an inner boundary, i.e. a boundary that is, for example, surrounded by the existing boundary. Such an additional boundary can, for example, exclude existing trees or other obstacles in the lawn from the working area to be processed.

[0052] For this purpose, environmental information (e.g. position, direction) and / or images of the environment in which the mobile device moves or should move are provided. This environmental information or images are obtained when the mobile device moves. In this case, the movement of the mobile device includes a movement starting from a starting point, wherein the starting point corresponds to the following point in the environment: the mobile device will stop or has stopped at this point when moving on or along an existing boundary (this can be a boundary, as described above, whether it has been adapted or changed). The movement of the mobile device is in this case particularly automatic. The stopping of the mobile device is in this case actively and / or manually (i.e. consciously).

[0053] Then, based on the environment information and / or the image, and if necessary also based on existing boundaries, additional boundaries of the working area of ​​the mobile device within the environment are determined. In other words, the mobile device is stopped during its (automatic) movement along the already existing boundaries or at least (temporarily) on such existing boundaries, and from there the mobile device is controlled, for example around obstacles. In this case, the mobile device is at least partially controlled externally (for example manually), for example as described above for the first step. In this way, it is known how the additional boundary is positioned relative to the existing boundaries.

[0054] Preferably, the movement of the mobile device also includes a movement to an end point, wherein the start point and the end point are identical at least within a predefined tolerance. This allows the additional boundaries to be optimized, for example by means of loop closure.

[0055] The movement from the starting point to the intermediate point preferably comprises an externally controlled (e.g. manual) movement of the mobile device, while the movement from the intermediate point to the end point comprises, for example, an automatic movement of the mobile device. In particular, the intermediate point corresponds, at least within a predetermined tolerance, to a point that the mobile device has already passed during the movement from the starting point to the intermediate point. In this way, it is ensured that the mobile device completely avoids obstacles.

[0056] The remaining aspects, which have been performed for the other boundaries, apply equally to the additional boundaries, ie the additional boundaries can be determined, for example, with the aid of SLAM.

[0057] Navigation information for the mobile device can then be determined based on the boundaries (if necessary also based on additional boundaries and / or modified boundaries or adapted boundaries) and / or a (if necessary visual) map of the environment. Based on the navigation information, control information for moving the mobile device can in turn be determined. The control information can then be provided and / or the mobile device can be moved based on the control information.

[0058] A computing unit according to the invention, ie a system for data processing in general, such as a control unit of a control device or mobile device, or a central server or another computing system, is configured, in particular in terms of programming, to carry out the method according to the invention.

[0059] The invention also relates to a mobile device, which is configured to obtain control information determined as described above. Additionally or alternatively, the mobile device has a computing unit according to the invention. Furthermore, the mobile device has a control or regulation unit and a drive unit for moving the mobile device. Furthermore, the mobile device can have a detection device for detecting environmental information of the environment in which the mobile device moves or should move, such as the mentioned camera or cameras.

[0060] The mobile device is preferably configured as an at least partially automatically moving vehicle, in particular a passenger vehicle or a freight vehicle, and / or as a robot, in particular a household robot, such as a cleaning robot, a floor or street cleaning device or a lawn mowing robot, and / or as a drone.

[0061] It is also advantageous to implement the method according to the invention in the form of a computer program or computer program product with a program code for executing all method steps, because this leads to particularly low costs, especially when the execution control device is also used for other tasks and therefore exists anyway. Finally, a machine-readable storage medium is provided on which a computer program as described above is stored. Suitable storage media or data carriers for providing computer programs are particularly magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. The program can also be downloaded via a computer network (Internet, intranet, etc.). Such downloading can be done wired or wirelessly (e.g., via a WLAN network, a 3G connection, a 4G connection, a 5G connection or a 6G connection, etc.).

[0062] Further advantages and embodiments of the invention are apparent from the description and the accompanying drawings.

[0063] The present invention is schematically shown in the drawings based on embodiments and is described below with reference to the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1a A mobile device is schematically shown for explaining the present invention.

[0065] Figure 1b The different views schematically show Figure 1a Mobile devices in .

[0066] Figure 2a , Figure 2b , Figure 2c An environment with a mobile device for explaining the present invention is schematically shown.

[0067] Figure 3a , Figure 3b , Figure 3c , Figure 3d Schematic images are shown for explaining embodiments of the present invention.

[0068] Figure 4a , Figure 4b A map on an input device is schematically shown.

[0069] Figure 5 The flow of a method for explaining an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0070] exist Figure 1a In FIG. 1 , a mobile device 100, in particular a working device, is schematically and exemplarily shown for explaining the present invention. Figure 1b In the figure, the mobile device 100 is shown in different views and different aspects. Figure 1a and Figure 1b .

[0071] The mobile working device 100 is, for example, a mowing robot having a control or regulating unit 102 and a drive unit 104 (with wheels), which is used to navigate or move the mowing robot 100 in an environment 120 and in particular in or on a working area 122 (e.g. a lawn or a garden) of the environment 120. The mowing robot 100 can move or be moved, for example, along a movement path or trajectory 130. In addition, the mowing robot 100 has, by way of example, a sensor 106 configured as a camera. With the aid of the camera 106, images of the environment can be detected, which in turn can be used for navigation. In addition, a connection station 110 is provided by way of example, at which the mowing robot can be charged.

[0072] exist Figure 1b In the environment, buildings 124, trees 126 and people 128 are also shown by way of example, which may be considered as objects or obstacles when navigating the lawn mowing robot.

[0073] Furthermore, the mowing robot 100 has a computing unit 108, for example a control device, by means of which data can be received and / or sent. As already mentioned, this can be done, for example, via various types of wireless communication connections. These types of wireless communication connections are Figure 1a 170, 172, 174. By way of example, these are a mobile radio connection 170, a Bluetooth connection 172 and a WLAN connection or WiFi connection 174. For this purpose, the computing unit 108 can, for example, have corresponding radio modules or be connected to radio modules, which then become part of the mowing robot 100.

[0074] In addition, Figure 1a In the example shown in FIG. 1 , in a building 162 (which may be different from or the same as the building 124), there is a mobile input device 140 (e.g. a smartphone), a central computing system 150 (or a server, which may represent a so-called cloud) and a WLAN router 160. The types of wireless communication connections 170, 172, 174 already mentioned are provided between the mowing robot 100 or its computing unit 108, the mobile input device 140, the computing system 150 and the WLAN router 160. It should be noted that the mobile radio connection 170 is established via a mobile radio transmitter 152, which in turn is connected to the computing system 150.

[0075] The Bluetooth connection 172 is provided between the mowing robot 100 or its computing unit 108 and the mobile input device 140, ie a Bluetooth connection can be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140. It is understood that the mobile input device 140 has a suitable radio module for this purpose.

[0076] The WLAN connection 174 is provided between the mowing robot 100 or its computing unit 108 and a WLAN router, i.e., a WLAN connection can be established between the mowing robot 100 or its computing unit 108 and a WLAN router 160. It is understood that the WLAN router 160 has a suitable radio module for this purpose. The WLAN router 160 can in turn be connected to the Internet via, for example, a wired connection.

[0077] A mobile radio connection 170 is provided between the mowing robot 100 or its computing unit 108 and the computing system 150, i.e. a mobile radio connection can be established between the mowing robot 100 or its computing unit 108 and the computing system 150. It should be mentioned here that such a mobile radio connection 170 is established by the mowing robot 100 or its computing unit 108 or a radio module therein, for example, with a mobile radio transmitter 152 or a corresponding mobile radio station, to which the computing system 150 is in turn connected, for example, via mobile radio and / or by wire.

[0078] Furthermore, a mobile radio connection 170 is provided between the mobile input device 140 and the computing system 150, i.e. a mobile radio connection can be established between the mobile input device 140 and the computing system 150. It also applies here that such a mobile radio connection 170 is established by the mobile input device 140 or a radio module there, for example, with a mobile radio transmitter or a corresponding mobile radio station, to which the computing system 150 is in turn connected. It should be mentioned that the mobile input device 140 includes a radio module for WLAN for this purpose, which is usually the case in the case of a typical smartphone.

[0079] Likewise, a WLAN connection 174 may be provided between the mobile input device 140 and the computing system 150, i.e., a WLAN connection may be established between the mobile input device 140 and the computing system 150. To this end, a WLAN connection 174 may be provided between the mobile input device 140 and a WLAN router. As already mentioned, the WLAN router 160 in turn may be connected to the Internet, for example, via a wired connection, and to the computing system 150 via a wired connection.

[0080] Thus, the mowing robot 100 or its computing unit 108 can send and receive data, i.e. exchange data, via each of the mentioned wireless communication connections 170, 172, 174. With the mobile input device 140, data can be exchanged here not only via a Bluetooth connection, but also via a mobile radio connection 170 or a WLAN connection 174, i.e. indirectly via the computing system 150. It should be mentioned that in principle a WLAN connection can also be established between the mowing robot 100 or its computing unit 108 and the mobile input device 140.

[0081] The mobile working device or lawn mowing robot 100 can be configured to perform one or more functions. In one embodiment, these functions include a working function (eg, a lawn mowing function) and a teaching function.

[0082] The working functions include, for example, the automatic movement of the mowing robot 100 in the environment 120 and the execution of a working process (or processing process) in the environment during at least temporary movement in the environment, i.e., for example, mowing the lawn (depending on the situation, the mowing robot can also first drive to a specific location without mowing the lawn and then start or continue mowing there).

[0083] The mobile radio connection 170 can be assigned to the work function. This means that the mowing robot 100 or its computing unit 108 obtains and / or sends the data required for performing the work function at least partially via the mobile radio connection. This can include, for example, a start command for starting a mowing process, which is sent to the mowing robot 100 from the mobile input device (if necessary via the computing system 150).

[0084] This can also include, for example, obtaining a trajectory (or navigation information in general) from the computing system 150, which trajectory (or navigation information in general) is determined in the computing system 150 and which the mowing robot should subsequently follow for the mowing process. However, it can also be provided that the trajectory (or navigation information in general) is determined in the mowing robot 100 or its computing unit 108 itself or that the trajectory (or navigation information in general) is obtained there in another way. However, instead of the trajectory, the computing unit 108 can also (only) obtain control information that has been determined based on the trajectory (which in turn can be determined in the computing system 150), and based on this control information, the control or regulating unit 102 can move the mowing robot 100 via the drive unit 104 to follow a trajectory, such as the trajectory 130 shown here only by way of example.

[0085] The mowing robot 100 can then move independently on the working area 122 or in the environment 120 or navigate there and mow the lawn in the process. As already mentioned, various objects can be detected by the mowing robot 100 or its camera 106 (thus visible in the images recorded by means of the camera) and then taken into account during navigation (i.e., when determining navigation information). For this purpose, the images detected in the process or the corresponding data can be transmitted to the computing system 150, for example, via the mobile radio connection 170.

[0086] The teaching function includes, for example, manually controlled movements of the mowing robot in the environment, in particular movements along the boundaries of the working area 122, in order to teach the mowing robot the boundaries. In this case, the working function, i.e. mowing, does not have to be used (or should not be used). For this purpose, control commands (or steering commands) for navigation or movement can be transmitted to the mowing robot 100 (continuously), for example. Such control commands or corresponding data are sent from the mobile input device 140 to the mowing robot via the Bluetooth connection 172. Likewise, the mowing robot 100 or its computing unit 108 can send data back to the mobile input device 140 via the Bluetooth connection. This is also called teaching or so-called "teaching".

[0087] In order to explain the present invention, Figure 2a An environment 220 with a building 224 and a tree 226 is shown, and Figure 1a , Figure 1b In addition, the desired working area, such as a lawn to be mowed, is exemplarily shown with 221. Exemplarily, the mobile device or mowing robot 200 and the connection station 210 are also shown again.

[0088] It is then typically desired that the desired working area 221 is covered completely and as accurately as possible by the mobile device during treatment, so that the entire lawn is mowed as completely as possible.

[0089] For this purpose, the boundaries of the (actual) working area should now be determined, which can then be processed by the mobile device. In other words, the boundaries or limits within which the mobile device should move and within which processing should be performed should be determined.

[0090] Within the scope of the aforementioned teaching function, mobile device 200 can now be controlled by the user, for example by means of a smartphone, from docking station 210 along the edge of desired working area 221 until mobile device 200 reaches docking station 210 again.

[0091] Based on this movement and the environmental information (eg images) obtained during this, a boundary 223 of the (actual) working area 222 can be determined in a first step, as shown in FIG2 . In this case, it can be seen that this boundary 223 deviates from the actual edge of the desired working area 221 .

[0092] This may be due in part to the user moving the mobile device 200 too far from the actual edge of the desired working area 221. This is shown in FIG. 2 in the lower left and upper right corners, where the border 223 is particularly far from the actual edge of the desired working area 221.

[0093] At this point it should be mentioned that when the mobile device 200 is moved, the position of the mobile device can be determined relative to a center point if necessary, while the geometric dimensions of the mobile device or its processing tool should be taken into account for the boundaries.

[0094] However, in a second step, environmental information of the environment, in particular an image of the environment, can now be obtained again, and when the mobile device 200 is moved again along the boundary 223 and / or within the boundary 223. This can be the case, for example, within the scope of the mentioned working function, in which the mobile device moves according to the trajectory 330.

[0095] The boundary 223 can then be confirmed and / or adapted based on the environmental information or images obtained during this process. Figure 2a In the figure, it is shown by way of example in the lower left and upper right corners, where, as described above, the boundary 223 is particularly far from the actual edge of the desired working area 221. The adapted boundary is denoted by 233 there.

[0096] If the mobile device 200 moves, for example, along the trajectory 330 , an image is also detected from the lower left location. By analyzing these images, it can be seen that the actual edge of the desired working area 221 deviates from the boundary 223 .

[0097] Based on these images, a map of the environment including at least a portion of the boundary may also be determined.

[0098] As already mentioned, instead of these two steps, it is also possible to use only the step of externally controlling the mobile device, but in which case an image of the surroundings is also subsequently acquired.

[0099] exist Figure 2b In the Figure 2a 2. Here, a road or a movement path along which the mobile device 200 moves (e.g. during a manual movement controlled externally) is now shown with 231. It can be seen here that the mobile device comprises a movement section or a movement path section 232, in which the mobile device moves at least partially outside the (desired) working area 221 and on a ground surface 221b that is different from the ground surface 221a of the working area 221. For example, the ground surface 221a can be a lawn, while the ground surface 221b can be a road or a sidewalk.

[0100] By this movement, the boundary indicated here by 223' can be determined more accurately, i.e. without having to perform the movement outside the working area 221. It should be mentioned at this point that the boundary 223' shown here is slightly spaced from the edge of the (desired) working area 221, although in practice this can be significantly more consistent.

[0101] exist Figure 2c In the Figure 2a 220 in the environment. Now, how to determine the additional boundary will be explained here. Here, the boundary 223 determined as described above, for example, can be assumed as an existing boundary.

[0102] When the mobile device 200 is moving along the boundary 223, and in particular when moving automatically, the mobile device can stop, and for example stop at point P1. From there, i.e. starting from point P1 as the starting point, the mobile device 200 can then be manually controlled around the tree 226 (i.e. the obstacle), for example via the mentioned Bluetooth connection to the smartphone; this is shown by roads 235 and 236. Suitably, this manual movement is carried out up to point P3, i.e. the intermediate point, which is located at least within a predetermined tolerance on the previously traveled point (here point P2) (i.e. on roads 235, 236). From there, the mobile device can then, for example, be driven automatically again to the end point, here point P4, which coincides with the starting point at least within a predetermined tolerance. This is shown by road 237.

[0103] In this way, an additional boundary can be determined, and in particular the road portion located between points P2 and P3; this is an at least approximately closed loop; Figure 2c , the additional boundary corresponds to road 236 .

[0104] While boundary 223 or 233 is particularly an outer boundary, boundary 236 may be referred to as an inner boundary.

[0105] exist Figure 3a , Figure 3b , Figure 3c , Figure 3d In FIG. 1 , an image for explaining the mentioned orthographic projection and segmentation, which may be used in embodiments of the present invention, is schematically shown as an example of applying a computer vision algorithm.

[0106] Figure 3a An image 300a is shown that can be detected by a mobile device or its camera. For example, the image 300a can be approximately Figure 2a or Figure 2b In the case shown, it is detected. In particular, in the front view, a building 324 can be seen, as well as a desired working area 322 at the bottom edge, for example with a lawn as ground 321a. A road with other ground 321b extends beside the working area 322.

[0107] As can be seen based on the image 300 a , it may be difficult to determine directly based on this image a boundary which should correspond as accurately as possible to the desired working area 322 or an edge thereof.

[0108] Figure 3bNow, image 300b is shown, which is an orthographic projection of image 300a. Here, it can be seen that a desired working area 322 with lawn as ground 321a and a road with other ground 321b are shown in a top view (or a top-down view). There, the geometrical relationship and the size or course of the edge of the desired working area 322 (i.e., the line denoted here by 322) at least approximately correspond to the actual relationship.

[0109] Figure 3c An image 300c is shown, which is a semantic segmentation of the image 300a. Here, only two classes are distinguished, namely, for example, the lawn 321a and the background or the rest, which is not represented here. Thus, for example, buildings, roads and everything else except the lawn fall there. It can be seen here that

[0110] Figure 3d An image 300d corresponding to image 300a with semantic segmentation and orthographic projection is shown. This can be obtained from image 300b by semantic segmentation or from image 300c by orthographic projection, for example. The results are at least roughly the same. It will be appreciated that various subtle differences may occur.

[0111] In any case, the orthographic projection of the image, in particular using (semantic) segmentation, allows particularly accurate determination of the edges of the (desired) working area, using which edges the border can be particularly accurately determined. Also, an accurate (visual) map of the environment can be determined thereby, in which at least a part of the border is shown, for example. This map can then be displayed to the user, for example on a smartphone.

[0112] Figure 4a An input device 440 is shown by way of example, such as a smartphone. This can be, for example, based on Figure 1a In addition to controlling the mobile device during teaching, the visual map 420 can also be displayed on the input device 440, such as exemplarily in Figure 4a . Then one can see there, for example, an area 421a of grass or lawn and an area 421b outside. In addition, a line 423 is shown in the map, which line can correspond to, for example, a boundary created in the first step described above; this can correspond, for example, to a boundary according to Figure 2a Boundary 223.

[0113] Furthermore, a line 433 is shown, which can represent a modified, in particular extended and / or improved boundary; this can correspond, for example, to Figure 2a As mentioned, such modified or extended boundaries can in particular be determined automatically and then displayed to the user as a suggestion in a map, as shown here Figure 4a as shown in .

[0114] The user may then confirm his agreement with the suggestion of boundary 433, for example via a suitable input command on input device 440. Likewise, the user may reject the suggestion with boundary 433. Boundary 423 may then be retained, for example.

[0115] Likewise, users can suggest changes, for example. Figure 4b 424 is shown as an example, and a line 434 is used for the boundary of the change. To this end, the user can, for example, shift the line 433 (for example, at a suitable point on the line) by means of a touch display of an input device 440, so that this produces a line 424.

[0116] Figure 5 Schematically shows a method for determining at least partially restricting a mobile device (ie, for example, based on Figure 1a , Figure 1b or Figure 2a , Figure 2b , Figure 2c The process of the method for determining the boundary of the working area of ​​the lawn mowing robot 100). The process should also be explained in more detail with reference to the figures explained above.

[0117] To this end, in step 500 (i.e. the first step), environmental information 502 of the environment in which the mobile device moves or should move is provided. The environmental information may include, for example, position, direction, and is determined, for example, based on images detected by means of a camera of the mobile device. This can be done, for example, by means of SLAM, in particular visual SLAM. In this case, the mobile device is controlled from the outside, as described above, for example with reference to Figure 2a The mobile device can in particular also be moved on other surfaces outside the desired working area, for example Figure 2b Explained.

[0118] Then in step 504, based on the environment information 502 from the first step, the boundaries of the working area of ​​the mobile device within the environment are determined, such as referring to Figure 2a This can be done, for example, using loop closure, in particular when the start and end point are identical or close to each other in a movement controlled externally, for example at a connection station. This also provides a boundary.

[0119] Then, in step 506, i.e., the second step, environmental information 508 of the environment, such as an image of the environment, is provided at least in the border area. This environmental information or image 508 is obtained when the mobile device moves along the border and / or within the border, as described above, for example, with reference to Figure 2a Explained.

[0120] In step 510, a map of the environment, in particular a visual map, may be determined based on image 508. This may be done using orthographic projection and segmentation. An example of such an image is Figure 3a An example of such a map is Figure 4a Map 420 shown in .

[0121] In this case, in step 512, orthographic projection views 514 of images of the environment may be determined from the images 508, such as shown by the image 300b in FIG. 3. Then, in step 516, these orthographic projection images 514 or views thereof may be segmented, such as by Figure 3d As shown in the image 300d in FIG. A map can be created and also provided.

[0122] The existing boundary (from the first step) can then be confirmed (or validated) based on the map or just based on the segmented ortho-projected image, step 518. However, the existing boundary can be adapted if necessary, i.e. step 520. The boundary obtained in this process (i.e., e.g., confirmed or adapted) can then be provided, e.g., for further use, in step 522.

[0123] Alternatively, however, it can also be provided that in step 500', environmental information 502 and image 508 obtained during the externally controlled movement are provided. In step 510', a map, in particular a visual map, can be determined based on image 508, as also described above. In particular, steps 512 to 516 can also be provided here.

[0124] A boundary may then be determined based on the environmental information and the image at step 518'. The boundary may then be provided at step 522', for example, for further use.

[0125] Starting from the boundaries provided in step 522 or 522', or also starting from the boundaries determined or provided in step 504, in step 524 a modified, in particular extended, boundary can be determined, and this can be based on the boundaries from the first step and / or on the environment information, in particular the image, from the second step, or generally from existing boundaries, for example with reference to Figure 2a Likewise, environmental information, in particular images, from further movement (eg during a processing procedure) can then be provided.

[0126] Determining the modified boundary is particularly automatic in this case. The modified boundary is then provided, in particular as a map, in particular to the user, i.e. step 526, for example with reference to Figure 4a Explained.

[0127] The user can then make changes or corrections, for example, as referenced Figure 4bThen, in step 528, correction information 530 is received from the outside (ie, for example, from a smartphone). Then, in step 532, the boundaries confirmed in step 518 or obtained in step 518' can be adapted. However, the adapted map can also be re-adapted according to step 520.

[0128] Likewise, in this way, an adapted map can be obtained according to step 520, i.e., an adapted map that has only been corrected by the user. That is, the adapted boundary is then determined based on the boundary and based on the correction information. Alternatively, however, the modified boundary can also be determined without correction information.

[0129] Based on the boundaries (e.g., first obtained, adapted, or modified boundaries), navigation information 536 for the mobile device can then be determined at step 534. Based on the navigation information, control information 540 for moving the mobile device can then be determined at step 538, and the control information 540 can then be provided.

[0130] After obtaining a boundary, whether an initial boundary (see step 522 or 522'), an adapted boundary or a modified boundary (see step 532), additional, for example, internal boundaries may also be determined based on the boundary as an existing boundary.

[0131] To this end, in step 550, environmental information 552 (similar to environmental information 502, such as position, direction) and / or images 554 (similar to images 508) of the environment in which the mobile device moves or should move are provided. These environmental information or images are obtained when the mobile device moves, wherein the movement of the mobile device includes movement starting from a starting point, wherein the starting point corresponds to the following point in the environment: the mobile device will stop at this point or has stopped at this point on the existing boundary during (especially automatic) movement. This has been referred to Figure 2c Explained in more detail.

[0132] Then, in step 556, additional boundaries of the operating area of ​​the mobile device within the environment are determined based on the environment information and / or the image, for example, also referring to Figure 2c The additional boundary can then be provided at step 558 and used again to move the mobile device, for example, starting from step 534. A modified boundary can also be determined for the additional boundary, as described above with reference to boundary 223 using modified boundary 233.

Claims

1. A method for determining a visual map (420) of at least a portion of an environment (120, 220), wherein a mobile device (100), in particular an at least partially autonomously moving vehicle or a robot, in particular a lawnmower robot, moves or is to move in the environment, the method comprising: providing (506) an image (508, 300a) of the environment, the image to be obtained or having been obtained while the mobile device (100) moves within the environment, determining (512) an orthographic projection view (514, 300b, 300d) of an image of the environment, determining a visual map (420) of the environment based on the orthographic projection view (514, 300b, 300d) of the image of the environment, and Provide a visual map of the environment.

2. The method according to claim 1, wherein a computer vision algorithm, in particular segmentation, is also used to determine a visual map of the environment (420).

3. The method according to claim 2, wherein the image of the environment is processed by means of the computer vision algorithm before determining the orthographic projection view to obtain a pre-processed image of the environment, and Wherein an orthographic projection view (300d) of an image of the environment is determined based on a pre-processed image (300c) of the environment.

4. The method according to claim 2, wherein an orthographic projection view (300b) of the image of the environment is determined before processing by means of the computer vision algorithm, and Therein an orthographic projection view (300b) of an image of the environment is processed with the aid of the computer vision algorithm. 5 . The method according to claim 2 , wherein the application, in particular the segmentation, of the computer vision algorithm is performed with the aid of a machine learning algorithm.

6. A method according to any of the preceding claims, wherein the mobile device (100) is a lawn mowing robot and wherein the image of the environment includes an image of grass, wherein, in particular with reference at least to claim 2, the boundaries of the grass are determined when applying the computer vision algorithm.

7. The method of any of the preceding claims, wherein the visual map (420) includes at least a portion of a boundary that at least partially limits an operating area of ​​the mobile device within the environment.

8. The method according to claim 7, further comprising: determining (524) a modified, in particular extended, boundary (233) based on the boundary and / or the image of the surroundings, wherein the determination of the modified boundary is performed in particular automatically, and The modified boundaries are provided (526), ​​in particular as a map, in particular to a user.

9. The method according to any one of the preceding claims, further comprising: determining (534) navigation information (536) for the mobile device based on the map of the boundary and / or the environment, determining (538) control information (540) for moving the mobile device based on the navigation information, and The control information is provided and / or the mobile device is moved based on the control information.

10. A computing unit (108) comprising means for performing the method according to any of the preceding claims.

11. A mobile device (100) configured to obtain control information determined by means of the method according to claim 9 and / or having a calculation unit according to claim 10, and A mobile device having a drive system (104) and a control or regulating unit (102) for controlling the drive system, and in particular a detection device (106) for detecting environmental information of an environment in which the mobile device is moving or is to move, The mobile device is in particular configured as an at least partially automatically movable vehicle, in particular as a passenger vehicle or a freight vehicle, and / or as a robot, in particular as a household robot, such as a cleaning robot, a floor or street cleaning device or a lawn mowing robot, and / or as a drone.

12. A computer program comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to claims 1 to 9.

13. A computer-readable storage medium having stored thereon the computer program according to claim 12.

Images