METHOD FOR DETERMINING A WORK AREA BOUND FOR A MOBILE DEVICE

Description

[0001] The present invention relates to a method for determining a boundary that at least partially limits a working area for a mobile device, in particular a vehicle or robot that moves at least partially automatically, especially a robotic lawnmower, a computing unit and a computer program for carrying it out, and a mobile device. Background of the invention

[0002] Mobile devices or work equipment, such as vehicles or robots that move at least semi-automatically, typically move in an environment, in particular an environment to be processed or a work area, such as an apartment, in a garden, in a factory hall or on the street, in the air or in water.

[0003] EP 4 086 722 A1 and US 2023 / 042867 A1 deal with robots and enclosures for work areas. Disclosure of the invention

[0004] According to the invention, a method for determining the boundary of a work area for a mobile device, a computing unit, and a computer program for carrying out this method, as well as a mobile device with the features of the independent claims, are proposed. Advantageous embodiments are the subject of the dependent claims and the following description.

[0005] The invention relates generally to mobile devices that move, or at least can move, within an environment, such as a work area. These can also be referred to as mobile work equipment. Examples of such mobile devices (or mobile work equipment) include robots and / or drones and / or vehicles that move semi-automatically or fully automatically (on land, water, or in the air). Examples of robots include household robots such as cleaning robots (e.g., vacuuming and / or mopping robots), floor or street cleaning equipment, construction robots, or robotic lawnmowers, as well as other so-called service robots. Examples of vehicles that move at least partially automatically include passenger transport vehicles or goods transport vehicles (also known as industrial trucks, e.g., in warehouses), but also aircraft such as drones or watercraft.

[0006] Such a mobile device includes, in particular, a control unit and a drive unit for moving the device, enabling it to be moved within its environment, especially along a path. Navigation information can be determined based on this path, such as specific instructions on the direction the device should travel to follow it. These instructions can then be implemented by the control unit and the drive unit.

[0007] For navigation, an environmental map can be used, which may have been obtained or determined using SLAM. SLAM (Simultaneous Localization and Mapping) is a robotics technique in which a mobile device, such as a robot, can simultaneously create a map of its environment and estimate its spatial position within that map. This facilitates obstacle detection and thus supports autonomous navigation.

[0008] Furthermore, a mobile device or work tool can have one or more sensors that can detect its surroundings or information within them. These can be, for example, cameras, lidar sensors, or inertial sensors, which can be used to capture the environment and / or the movement of the mobile device, for example, in two or three dimensions.

[0009] Furthermore, such a mobile device can be configured to receive and / or send data via a communication link, in general, to communicate and exchange data. This allows communication with the mobile device, for example, giving instructions to the mobile device, transmitting other data to the mobile device, or receiving data or information from the mobile device. Wireless communication links are particularly relevant in this context. For this purpose, the mobile device can, for example, have corresponding (possibly different) modules for wireless communication, which can also be integrated into a processing unit.

[0010] With such a mobile device, it is also common for it to have functions that it can perform; that is, the mobile device is configured to carry out one or more, preferably different, functions. Instead of functions, one can also speak of applications. Such functions could be, for example, a work function or a training function.

[0011] The work function includes, for example, the automated movement of the mobile device within an environment and the execution of a work process within that environment, at least temporarily while moving. In the case of a robotic lawnmower, the work function could therefore include, for example, mowing (while moving). For this purpose, the mobile device can be given a start command to perform the work function, for example, by sending corresponding data via the wireless communication link.

[0012] The learning function includes, for example, manually controlling the mobile device's movement within its environment, particularly without performing any work within that environment. In the case of a robotic lawnmower, the learning function can therefore include, for example, manually moving or controlling the device along the perimeter of the working area to teach the robotic lawnmower to follow the boundaries. For this purpose, the mobile device can be given (continuous) control commands to navigate or move within its environment, for example, by sending corresponding data via the wireless communication link. This process is also referred to as learning or a "teach-in."

[0013] One aspect of this training is that a boundary (or limit) needs to be found or defined to surround the work area, for example, a garden. During operation, this boundary can then be used to limit the automated movement of the mobile device.

[0014] A method is proposed for determining such a boundary for a mobile device, which at least partially limits a working area for the mobile device.

[0015] In a first step, environmental information about the environment in which the mobile device is moving or is intended to move is provided. This environmental information can include, for example, positions, orientations, and the like, which can be used for SLAM (Self-Assessment, Monitoring, and Control). In particular, the environmental information can also include images or environmental images that are captured, for example, by the mobile device's camera. This environmental information (in the first step) is obtained when the mobile device is moved externally, specifically during manual movement, as mentioned above in relation to the learning function. The mobile device can therefore be controlled, for example, via a smartphone and a Bluetooth connection. A user can thus, for example...Guide the robotic lawnmower along the edge of the desired working area, following its movements as closely as possible. This initial movement of the mobile device can also be considered a first step.

[0016] Based on the environmental information from the first step, a boundary of the workspace within the environment is then determined for the mobile device. This is a kind of preliminary or rough boundary, but it can generally be used for work purposes. Since this boundary is based on manual control of the mobile device, it may not be very precise.

[0017] The environmental information from the first step will be obtained using SLAM, specifically visual SLAM. For this purpose, the captured images can be processed accordingly. Preferably, the outline is determined and / or refined based on the environmental information from the first step using loop closure.

[0018] So-called loop closures are an aspect of SLAM (Swift Location Analysis and Management). The underlying assumption is that a mobile device, while moving within its environment or workspace, reaches a specific position it has previously occupied. Using environmental information such as images or distance data from the device's current location, and by comparing this information with previous locations, it can be determined whether the device has previously occupied that position. If this is confirmed, the map generated by SLAM, or the path marked on it, should also indicate that the device is back at the same location. However, due to potential discrepancies, this is often not the case. Instead, the current position shown on the map would differ from the previous position.Here, the map and therefore the border can be adjusted so that the two diverging positions align again. This is called a loop closure.

[0019] Therefore, it is particularly advantageous if the externally controlled movement of the mobile device involves movement from a starting point to an endpoint, where the starting point and the endpoint are identical, at least within predefined tolerances. For example, the starting point and the endpoint could be at the docking station. This allows for loop closure and enables particularly accurate determination of the boundary in the first step.

[0020] In a second step, environmental information, namely images of the surroundings, at least within the boundary area, is provided. This environmental information is acquired when the mobile device is moved again along and / or within the boundary. This subsequent movement is automated, occurring, for example, as part of the aforementioned work function; that is, processing is carried out. This could include, for example, the initial mowing of the (possibly entire) work area. This (renewed) movement of the mobile device for the second step can also be considered a further movement.

[0021] Based on the environmental information from the second step, the boundary (obtained in the first step) is then confirmed and / or adjusted. If the boundary is confirmed, it can be used as is; this can be referred to as verification. If the boundary is adjusted, the boundary obtained in the first step can be modified based on environmental information received in the second step. For example, the environmental information (preferably images) from the second step can reveal that there is a patch of grass outside the boundary from the first step, allowing the boundary to be adjusted accordingly. It goes without saying that parts of the boundary can be confirmed while other parts are adjusted.

[0022] As mentioned, the environmental information from the second step can include images of the environment. In one embodiment, confirming and / or adjusting the boundary then involves determining a map of the environment, encompassing at least part of the boundary, based on the images from the second step. This can be done, for example, using an orthographic projection.

[0023] In this way, few or no images need to be captured and stored in the first step to later create the map. Instead, the boundary is determined solely using (visual) SLAM. This saves computing resources. The environmental information, especially images, from the second step can then be used for verification or adjustment. Since the second step can be performed within a single workflow, the mobile device can be used for its actual work function much faster than if, for example, an additional verification step were performed (before the work function).

[0024] As an alternative to the two-step approach, it is also possible to provide environmental information and images of the environment in which the mobile device is moving or is intended to move. This information is obtained or has been obtained through externally controlled movement of the mobile device. Based on this environmental information and these images, a boundary of the workspace within the environment is then determined for the mobile device. In this case, images are also captured and, if necessary, stored and processed. This allows for a particularly precise boundary to be determined in a single movement of the mobile device, although this may require more computing resources. Compared to the two-step approach described above, only the first step is necessary here, but it also involves capturing the images.

[0025] In one embodiment, environmental information, particularly images, of the surroundings, at least within the boundary area, can also be provided in this case. This information is obtained, or has been obtained, during further (especially automated) movement of the mobile device along and / or within the boundary. During this further movement, the mobile device performs a processing operation. This can, in itself, correspond to the second step of the variant described above. With this environmental information or these images, the boundary does not then need to be confirmed and / or adjusted (although this may be necessary). Rather, the map of the surroundings, which includes at least part of the boundary, can be determined; this can be done, for example, using an orthographic projection, as mentioned above.

[0026] As has been shown, a particularly accurate outline can also be obtained by targeted movement of the mobile device, depending on whether, for example, two steps or only one step are used to determine the outline.

[0027] This involves providing environmental information and / or images of the environment in which the mobile device is moving or is intended to move. This environmental information and / or images are acquired when the mobile device moves: This movement can be external, manual, or automated.

[0028] The movement of the mobile device includes at least one phase in which it moves, at least partially, outside the work area on a surface different from the work area (or the desired work area). This could be, for example, a path or paving, while the (desired) work area has a surface such as grass. In other words, the mobile device is deliberately moved outside the (actual or desired) work area, whereas it normally only moves within the (actual or desired) work area.

[0029] Based on environmental information and / or images, the boundaries of the workspace within the environment are then determined for the mobile device. By deliberately moving the mobile device outside the (actual or desired) workspace, it is possible to capture the edge of the (desired) workspace more accurately, thus allowing for a significantly more precise determination of the boundaries (whether preliminary or final). It should be noted that the boundaries do not necessarily have to be directly based on the movement of the mobile device, e.g., externally controlled movement, but can be adjusted accordingly to limit the boundaries to the (desired) workspace.

[0030] It is advantageous if the surface, which differs from the surface of the work area, adjoins the surface of the work area and / or has the same height within specified tolerances. The height here refers specifically to a direction of gravity. In this way, the mobile device can be easily moved from one surface to another, even on both surfaces simultaneously.

[0031] It should be noted that often only part of the boundary can be determined in this way, since, for example, parts of the ground outside the (desired) work area may not be accessible by vehicle; for example, a fence, a wall or water may limit the (desired) work area in places.

[0032] As mentioned, the map of the environment can be determined using an ortho projection, specifically a visual map. A visual map in this context refers to a map that depicts the environment pictorially, not a SLAM map with nodes and edges or similar features. It can also be described as a structural map, showing the structure of the environment, for example, areas with grass and areas without.

[0033] For this purpose, images of the environment are provided, whereby the images are obtained or have been obtained from the movement of the mobile device within the environment. This movement can be external, manual, or automated. An ortho-projected view of the environment images is then determined. Based on this ortho-projected view, a visual map of the environment is then generated; this can then be used as described above.

[0034] In one embodiment, the visual map of the environment is determined using a computer vision algorithm, such as segmentation, e.g., semantic segmentation. Applying a computer vision algorithm allows for the preprocessing of images. In particular, it enables the identification of contiguous areas within an image, which is also the case with segmentation. The images of the environment can be segmented or, more generally, processed using the computer vision algorithm before determining the ortho-projected view, in order to obtain segmented, or more generally, preprocessed images of the environment. The ortho-projected view of the environment is then determined based on these segmented or preprocessed images.

[0035] However, the ortho-projected view of the images of the environment can also be determined before segmentation or application of the computer vision algorithm. The ortho-projected view of the images of the environment is then segmented or pre-processed.

[0036] In one embodiment, the segmentation is performed using a machine learning algorithm, e.g., an artificial neural network. However, a classic application of a computer vision algorithm is also possible.

[0037] If the mobile device is a robotic lawnmower, and the images of the surroundings include images of grass (or lawn), the boundaries of the grass can be determined during segmentation or application of the computer vision algorithm.

[0038] In this way, the boundaries of the (desired) work area, such as the lawn, can be defined very precisely, allowing the visual map to represent the work area with exceptional accuracy. In particular, this method allows for a very precise definition of the boundary, for example, the boundary between the grass and other surfaces. This enables a particularly accurate determination or adjustment of the boundary, as mentioned above.

[0039] Ortho projection of an image (or a pre-processed or segmented image) means, in particular, that the image is projected onto a surface, i.e., top-down. It's important to remember that cameras on mobile devices, such as lawnmowers, are generally oriented at least approximately horizontally. Therefore, in an image captured by such a camera, the surface is only visible at the bottom edge and is distorted. By considering camera parameters such as the specific camera placement on the mobile device and the field of view, the portion of the image showing the surface can be corrected and displayed as a top-down view. In this representation, the ortho projection, the precise boundaries of the work area can be identified much more clearly.

[0040] In segmentation, particularly semantic segmentation, each pixel in an image can be assigned to one of several classes; that is, the pixels are labeled or classified. Typically, features of the pixels are first determined or extracted, and the class is then assigned based on these features. Such features can include, for example, shape, color, context, pattern, light variation, and / or image context. For instance, a pixel could be assigned the color green, and the surrounding pixels could also all be green. This would suggest grass.

[0041] Various attributes can be used as classes. In the example of a robotic lawnmower, two classes can be used in a simple case: "lawn" and "not lawn." This means that for each pixel in the image, it is determined whether it shows lawn or not. It goes without saying that more than two classes can be used to recognize, for example, paths, roads, buildings, vehicles, or people and assign them to a pixel. Another possibility—similar to the "not lawn" class—is a "background" class, to which everything not assigned to another class is assigned. All these labels add more information compared to pure object recognition, which (only) detects objects in an image.

[0042] In principle, classification does not need to be performed for every single pixel; it can also be performed for segments or parts of the image, each comprising several pixels. Such segments can be predefined or created during semantic segmentation, possibly with varying sizes within the image. For this purpose, one or more artificial neural networks or, more generally, pattern recognition methods based on artificial intelligence are preferably used.

[0043] A segmented image thus allows for a simpler and faster determination of where a boundary lies between different substrates.

[0044] In one embodiment, a modified, and in particular extended, boundary is determined based on the boundary from the first step and / or the environmental information, especially images, from the second step. The determination of the modified boundary is performed automatically. The modified boundary is then made available, particularly as a map, especially to a user. For example, a suggestion for a modified boundary can be provided. For this purpose, the modified boundary can be displayed, for example, on a smartphone or other input device.

[0045] In one embodiment, a modified, and in particular extended, border is determined based on the border from the first step and / or the environmental information, especially images, from the second step. The determination of the modified border is performed automatically. The modified border is defined as the adapted border. Thus, an improvement to the border can be performed automatically, and in particular without any user intervention.

[0046] Determining the modified boundary can be done independently of how precisely the existing boundary was obtained. For this purpose, an existing boundary of the workspace within the environment in which the mobile device is moving or is intended to move can be provided. This existing boundary is determined at least based on environmental information obtained during the initial movement of the mobile device.

[0047] Environmental information, particularly images, of the surroundings, at least within the area of ​​the existing boundary, is then provided. This information is obtained during subsequent movement of the mobile device, particularly automated movement, along and / or within the existing boundary. The modified boundary is then automatically determined based on the existing boundary and the environmental information, particularly images, obtained during the subsequent movement of the mobile device. This modified boundary is then displayed, particularly as a map (e.g., The aforementioned visual map is provided, particularly for a user. This can be done, for example, on a smartphone or other input or display device. A user can therefore receive a suggestion for an improved outline.

[0048] The modified border can be, in particular, an extended border or an improved border. This makes it possible to obtain a better border very easily and quickly (since it is automated), allowing for more precise coverage of the desired work area.

[0049] It can also be determined whether a deviation between the modified border and the existing border meets a predefined criterion; for example, an average or cumulative distance between the two borders can be determined and compared to a threshold value. The modified border is only provided if the deviation meets the predefined criterion. Thus, the user is only presented with a suggestion if this appears relevant.

[0050] Furthermore, it may be possible to receive correction information from external sources. For this purpose, the user can, for example, make and submit changes to the proposal based on the initial suggestion. The adjusted border is then determined based on the existing border (whichever one exists; there may also be an existing border and a proposed modified border) and based on the correction information.

[0051] In one embodiment, the environmental information from the first step includes information, particularly images, of reference points in the environment. This reference point information can then be used, in particular, to determine the boundary and / or to create a visual map. Such reference points could be, for example, checkpoints, or even just a few, not along the entire route, but at specific intervals and / or at certain maneuvers (90° turns, etc.). This allows for more precise verification or adjustment of the boundary or the map, while requiring the storage of only a few images in the first step.

[0052] In addition to the aforementioned methods for defining a boundary that at least partially limits a work area (this applies particularly to outer boundaries), an additional boundary can also be defined based on an existing boundary (which has then been defined, for example, as above), especially an inner boundary, i.e., a boundary that is enclosed by an existing boundary. Such an additional boundary can, for example, exclude a tree or other obstacles in the lawn from the work area to be worked on.

[0053] For this purpose, environmental information (e.g., position, orientation) and / or images of the environment in which the mobile device is moving or is intended to move are provided. This environmental information or these images are acquired when the mobile device moves. The movement of the mobile device here refers to movement from a starting point, where the starting point corresponds to a point in the environment at which the mobile device stops or has stopped when moving along or at an existing boundary (this can be a boundary, as mentioned above, whether it has already been adjusted or changed or not). The movement of the mobile device is primarily automated. Stopping the mobile device is done actively and / or manually, i.e., deliberately.

[0054] Based on environmental information and / or images, and possibly also based on the existing boundary, the additional boundary of the workspace within the environment for the mobile device is then determined. In other words, the mobile device is stopped during its (automated) movement along an existing boundary, or at least (briefly) at such an existing boundary, and from there steered around an obstacle, for example. The mobile device is controlled at least partially externally (e.g., manually), as described above for the first step. In this way, the position of this additional boundary relative to the existing boundary is known.

[0055] Preferably, the movement of the mobile device also includes movement to an endpoint, where the starting point and the endpoint are identical, at least within predefined tolerances. This allows the additional boundary to be optimized, for example, using loop closure.

[0056] Preferably, the movement from the starting point to an intermediate point involves an externally controlled (e.g., manual) movement of the mobile device, while the movement from the intermediate point to the endpoint involves, for example, an automated movement of the mobile device. In particular, the intermediate point corresponds, at least within predefined tolerances, to a point that the mobile device has already passed during the movement from the starting point to the intermediate point. This ensures that the mobile device has completely navigated around the obstacle.

[0057] Furthermore, the same applies to the additional border as has already been stated for the other borders, i.e., it can be determined, for example, using SLAM.

[0058] Based on the outline (possibly including additional, modified, or adapted outlines) and / or the (possibly visual) map of the environment, navigation information for the mobile device can be determined. Based on this navigation information, control information for moving the mobile device can then be determined. This control information can then be provided, and / or the mobile device can be moved based on this information.

[0059] A computing unit according to the invention (i.e., generally a system for data processing), e.g., a control unit or a control unit of a mobile device, or a central server or other computing system, is, in particular in terms of programming, equipped to carry out a method according to the invention.

[0060] The invention also relates to a mobile device configured to receive control information as described above. Furthermore, or alternatively, the mobile device comprises a computing unit according to the invention. The mobile device also includes a control unit and a drive unit for moving the device. Additionally, the mobile device may include sensor means for acquiring environmental information from the environment in which the mobile device is moving or is intended to move, such as the aforementioned camera or multiple cameras.

[0061] Preferably, the mobile device is designed as a vehicle that moves at least partially automatically, in particular as a passenger transport vehicle or as a goods transport vehicle, and / or as a robot, in particular as a household robot, e.g. cleaning robot, floor or street cleaning device or lawn mowing robot, and / or as a drone.

[0062] Implementing a method according to the invention in the form of a computer program or computer program product with program code for carrying out all method steps is also advantageous, as this incurs particularly low costs, especially if an executing control unit is already available for other tasks. Finally, a machine-readable storage medium is provided with a computer program stored on it as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical, and electrical storage media, such as hard drives, flash memory, EEPROMs, DVDs, etc. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or wireless (e.g., via a WLAN network, a 3G, 4G, 5G, or 6G connection, etc.).

[0063] Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawing.

[0064] The invention is schematically illustrated in the drawing using an exemplary embodiment and is described below with reference to the drawing.

[0065] Brief description of the drawings Figure 1a schematically shows a mobile device to illustrate the invention. Figure 1b schematically shows the mobile device made of Figure 1a In another view. Figures 2a, 2b, 2c schematically show an environment with a mobile device to illustrate the invention. Figures 3a, 3b, 3c, 3d schematically show images to illustrate embodiments of the invention. Figures 4a, 4b schematically show maps on an input device. Figure 5 schematically shows a sequence of a method to illustrate embodiments of the invention. embodiment(s) of the invention

[0066] In Figure 1aA mobile device 100, in particular a work device, is shown schematically and by way of example to illustrate the invention. Figure 1b The mobile device is presented in a different view and with different aspects. The following are intended to... Figures 1a and 1b can be described comprehensively.

[0067] The mobile work device 100 is, for example, a robotic lawnmower with a control unit 102 and a drive unit 104 (with wheels) for navigating or moving the robotic lawnmower 100 in an environment 120, and in particular in or on a work area 122, e.g., a lawn or garden. The robotic lawnmower 100 can, for example, move or be moved along a movement path or trajectory 130. Furthermore, the robotic lawnmower 100 has, for example, a sensor 106 designed as a camera. Images of the environment can be captured by means of the camera 106 and used for navigation. In addition, a docking station 110 is provided, for example, where the robotic lawnmower can be charged.

[0068] In Figure 1bIn the surrounding area, examples of buildings (124), trees (126), and persons (128) are indicated, which may need to be considered as objects or obstacles in the navigation of the robotic lawnmower.

[0069] Furthermore, the robotic lawnmower 100 has a processing unit 108, e.g., a control unit, by means of which data can be received and / or sent. As already mentioned, this can be done via, for example, various types of wireless communication connections. These types of wireless communication connections are in Figure 1a Designated with 170, 172, and 174. Examples include a mobile network connection (170), a Bluetooth connection (172), and a WLAN or WiFi connection (174). The processing unit (108) can, for example, have corresponding radio modules or be connected to them, which are then part of the robotic lawnmower (100).

[0070] Furthermore, in Figure 1aA mobile input device 140, e.g., a smartphone, a central computing system 150 (or a server, which can represent the so-called cloud), and a WLAN router 160 in a building 162 (which can be a different building than building 124 or the same building). The aforementioned types of wireless communication connections 170, 172, and 174 are provided between the robotic lawnmower 100 or its computing unit 108, the mobile input device 140, the computing system 150, and the WLAN router 160. It should be noted that the mobile communication connection 170 is established via a mobile communication transmitter 152, which in turn is connected to the computing system 150.

[0071] A Bluetooth connection 172 is provided between the robotic lawnmower 100, or rather its processing unit 108, and the mobile input device 140. This means that a Bluetooth connection can be established between the robotic lawnmower 100, or rather its processing unit 108, and the mobile input device 140. It is understood that the mobile input device 140 has a suitable radio module for this purpose.

[0072] The WLAN connection 174 is provided between the robotic lawnmower 100 (or its processing unit 108) and the WLAN router. This means a WLAN connection can be established between the robotic lawnmower 100 (or its processing unit 108) and the WLAN router 160. It is understood that the WLAN router 160 has a suitable radio module for this purpose. The WLAN router 160 can, in turn, be connected to the internet via, for example, a wired connection.

[0073] A mobile communication connection 170 is provided between the robotic lawnmower 100, or rather its processing unit 108, and the computer system 150. This means that a mobile communication connection can be established between the robotic lawnmower 100, or rather its processing unit 108, and the computer system 150. It should be noted that this mobile communication connection 170 is established by the robotic lawnmower 100, or rather its processing unit 108, or a radio module there, for example, with the mobile transmitter 152 or a corresponding mobile base station, to which the computer system 150 is in turn connected, for example, via mobile and / or wired connection.

[0074] Furthermore, a mobile communication connection 170 is also provided between the mobile input device 140 and the computing system 150; that is, a mobile communication connection can be established between the mobile input device 140 and the computing system 150. Here, too, such a mobile communication connection 170 is established by the mobile input device 140, or by a radio module within it, for example, with a mobile communication transmitter or a corresponding mobile communication base station, to which the computing system 150 is in turn connected. It should be noted that the mobile input device 140 includes a radio module for WLAN, which is usually the case with a typical smartphone.

[0075] Similarly, a WLAN connection 174 can be established between the mobile input device 140 and the computer system 150. This means that a WLAN connection can be established between the mobile input device 140 and the computer system 150. For this purpose, a WLAN connection 174 can be established between the mobile input device 140 and the WLAN router. The WLAN router 160, as already mentioned, can in turn be connected to the internet via, for example, a wired connection, and thus to the computer system 150.

[0076] The robotic lawnmower 100, or rather its processing unit 108, can therefore send and receive data, i.e., exchange data, via any of the aforementioned wireless communication connections 170, 172, and 174. The mobile input device 140 can exchange data not only via Bluetooth, but also via the mobile network connection 170 or the WLAN connection 174, i.e., indirectly via the processing system 150. It should be noted that a WLAN connection can also be established between the robotic lawnmower 100, or rather its processing unit 108, and the mobile input device 140.

[0077] The mobile work device, or the robotic lawnmower 100, can be configured to perform one or more functions. In one embodiment, these functions include a work function, e.g., a mowing function, and a learning function.

[0078] The work function includes, for example, the automated movement of the robotic lawnmower 100 in the environment 120 and the performance of a work process (or processing operation) in the environment, such as mowing the lawn, at least temporarily while moving in the environment (depending on the situation, the robotic lawnmower may also initially drive to a specific location without mowing, in order to begin or continue mowing there).

[0079] This work function can be assigned to the mobile network connection 170. This means that the robotic lawnmower 100, or rather its processing unit 108, receives and / or sends data necessary for carrying out the work function, at least partially, via the mobile network connection. This can, for example, include a start command to initiate a mowing process, which is sent from the mobile input device (possibly via the processing system 150) to the robotic lawnmower 100.

[0080] This can also include receiving, for example, trajectories (or navigation information in general) from the computer system 150, which trajectories (or navigation information) are determined in the computer system 150 and which the robotic lawnmower is then supposed to follow during the mowing process. However, it is also possible that a trajectory (or navigation information in general) is determined or otherwise obtained within the robotic lawnmower 100 itself or by its computer unit 108. Instead of a trajectory, the computer unit 108 can, for example, also receive (only) control information that has been determined based on a trajectory (which in turn may have been determined in the computer system 150), and according to which the control unit 102 can move the robotic lawnmower 100 via the drive unit 104 to follow a trajectory, for example, the trajectory 130, which is only mentioned here as an example.

[0081] The robotic lawnmower 100 can then, for example, move independently within the working area 122 or the surrounding area 120, navigating and mowing the lawn. As already mentioned, various objects can be detected by the robotic lawnmower 100 or its camera 106 (i.e., they are visible in images captured by the camera) and then taken into account during navigation, i.e., when determining navigation information. For this purpose, the captured images or corresponding data can be transmitted to the computer system 150, for example, via the mobile network connection 170.

[0082] The learning function includes, for example, manually moving the robotic lawnmower around its surroundings, particularly along the perimeter of the working area 122, to teach the robotic lawnmower the boundaries. The actual mowing function does not need to be (or should not be) used during this process. For this purpose, the robotic lawnmower 100 can be continuously sent control commands (or commands) for navigation and movement. These control commands and corresponding data are sent from the mobile input device 140 to the robotic lawnmower via the Bluetooth connection 172. Likewise, the robotic lawnmower 100, or rather its processing unit 108, can send data back to the mobile input device 140 via the Bluetooth connection. This process is also referred to as learning or "teach-in."

[0083] In Figure 2aFor the purpose of illustrating the invention, an environment 220 with a building 224 and a tree 226 is shown, comparable to the environment according to Figures 1a, 1b Furthermore, an example of a desired work area, e.g., a lawn to be mowed, is shown in figure 221. Also shown again as examples are a mobile device or robotic lawnmower (200) and a docking station (210).

[0084] Typically, the desired working area 221 is fully and as accurately as possible covered by the mobile device during processing, so that as much of the lawn as possible is mowed.

[0085] For this purpose, a boundary must now be defined for an (actual) work area that can then be edited by the mobile device. In other words, a boundary or limit must be defined within which the mobile device should move and within which editing should take place.

[0086] Within the framework of the aforementioned learning function, the mobile device 200 can now be controlled, for example, by a user via smartphone from the docking station 210 along the edge of the desired work area 221 until the mobile device 200 reaches the docking station 210 again.

[0087] Based on this movement and the environmental information obtained, e.g., images, a boundary 223 of an (actual) work area 222 can be determined in a first step, as in Figure 2 shown. Here it can be seen that this outline 223 differs from the actual edge of the desired work area 221.

[0088] This may be due in particular to the user moving the mobile device 200 too far away from the actual edge of the desired work area 221. This is exemplified in the bottom left and top right of the image. Figure 2indicated where the outline 223 is particularly far from the actual edge of the desired work area 221.

[0089] It should be mentioned here that when the mobile device is moved, its position may be determined in relation to a center point, while the geometric dimensions of the mobile device or its processing tool must be taken into account for the boundary.

[0090] In a second step, environmental information, especially images, can be obtained again when the mobile device 200 is moved along the boundary 223 and / or within the boundary 223. This can be the case, for example, in the context of the aforementioned work function, where the mobile device moves according to a trajectory 330.

[0091] Based on the environmental information and / or images obtained, the border 223 can then be confirmed and / or adjusted. This is shown as an example in the bottom left and top right of the image. Figure 2a It is indicated where the border 223, as mentioned, is particularly far from the actual edge of the desired work area 221. An adjusted border is designated 233 there.

[0092] If the mobile device 200 moves along trajectory 330, images of the lower left corner are also captured. By analyzing these images, it can be determined that the actual edge of the desired work area 221 differs from the outline 223.

[0093] Based on these images, a map of the surroundings, which includes at least part of the perimeter, can also be determined.

[0094] As mentioned previously, instead of the two steps, only the step in which the mobile device is controlled externally can be used, in which case images of the surroundings are also captured.

[0095] In Figure 2b is part of the surroundings 220 from Figure 2a shown again. Here, a path or movement path is shown with 231, along which the mobile device 200 is moved, for example, during externally controlled, manual movement. It can be seen that the mobile device includes a movement segment or movement path segment 232 in which the mobile device moves at least partially outside the (desired) work area 221, specifically on a surface 221b that differs from a surface 221a of the work area 221. For example, the surface 221a could be grass, while the surface 221b could be a path or paving.

[0096] This movement allows the boundary, here labeled 223', to be determined more precisely than without this movement outside the working area 221. It should be noted that the boundary 223' is shown here somewhat spaced away from the edge of the (desired) working area 221, although in practice this may correspond much better.

[0097] In Figure 2c is part of the surroundings 220 from Figure 2a shown again. Here, the determination of an additional border will be explained. Border 223, as determined, for example, as explained above, can be used as the starting point for this explanation.

[0098] While the mobile device 200 moves along the boundary 223, particularly in an automated manner, it can be stopped, for example, at a point P1. From there, i.e., point P1 as the starting point, the mobile device 200 can then be manually controlled, e.g., via the aforementioned Bluetooth connection with the smartphone, around the tree 226, an obstacle; this is shown with paths 235 and 236. Conveniently, this manual movement continues to a point P3, an intermediate point that lies, at least within specified tolerances, on a point (here point P2) that has already been traversed (i.e., on paths 235 and 236). From there, the mobile device can then, for example, move automatically again to an endpoint, here point P4, which coincides with the starting point, at least within specified tolerances. This is shown with path 237.

[0099] In this way, the additional boundary can be determined, specifically as that part of the path that lies between points P2 and P3; this is at least an approximately closed loop; in Figure 2c Does this additional border correspond to route 236?

[0100] While border 223 or 233 is primarily an outer border, border 236 can be described as an inner border.

[0101] In Figures 3a, 3b , 3c, 3d Schematic images are shown to illustrate the aforementioned ortho projection and segmentation as an example of an application of a computer vision algorithm, as it can be used in embodiments of the invention.

[0102] In Figure 3a Image 300a is shown, which can be captured by a mobile device or its camera. Image 300a can be used as an example in a situation similar to that described in Figure 2a or 2b The image shows the area to be recorded. In particular, a building 324 is shown in front view, as well as a section of the desired work area 322 at the bottom, with, for example, grass as a surface 321a. A path with a different surface 321b runs alongside the work area 322.

[0103] As can be seen from image 300a, it can be difficult to directly determine a border based on this that should correspond as closely as possible to the desired work area 322 or its edge.

[0104] In Figure 3bImage 300b is now shown, which is an orthophoto projection of image 300a. It shows the section of the desired work area 322, with grass as its base 321a, and the path with a different base 321b in a top-down view. The geometric proportions and dimensions, or rather the contours, of the edge of the desired work area 322 (i.e., the line labeled 322 here) correspond at least approximately to the actual conditions.

[0105] In Figure 3c Image 300c is shown, which is a semantic segmentation of image 300a. It distinguishes between only two classes: for example, lawn 321a and background or the rest, which is not labeled here. This includes, for example, the building, the path, and everything else except the lawn. Here it can be seen that In Figure 3dImage 300d is shown, which corresponds to image 300a with semantic segmentation and ortho projection. This can be obtained, for example, from image 300b through semantic segmentation or from image 300c through ortho projection. The result will be at least approximately identical. It is understood that various minor differences may occur.

[0106] In any case, an orthographic projection of the image, especially with (semantic) segmentation, allows for a particularly precise determination of the boundary of the (desired) workspace, thus enabling a very accurate definition of the outline. Likewise, it allows for the creation of a precise (visual) map of the environment, which, for example, shows at least part of the boundary. This map can then be displayed to a user on a smartphone, for instance.

[0107] In Figure 4aAn input device 440 is shown as an example, e.g., a smartphone. This could, for example, be input device 140 according to... Figure 1a or a comparable input device. Besides controlling the mobile device during the learning process, it can also display, for example, a visual map 420, as exemplified in Figure 4a The map shows, for example, area 421a for grass or lawn and area 421b outside of it. Additionally, line 423 is shown on the map, which could correspond to a boundary created in the first step mentioned above; this could, for example, correspond to boundary 223 according to... Figure 2a are equivalent to.

[0108] Furthermore, line 433 is shown, which may represent a modified, in particular extended and / or improved, border; this may, for example, be the border 233 according to Figure 2aThis corresponds to such a modified or extended border, which, as mentioned, can be determined automatically and then suggested to the user on the map, as shown here. Figure 4a to be displayed.

[0109] The user could then, for example, confirm their agreement with the proposed border 433 by using a suitable input command on input device 440. Likewise, the user could reject the proposed border 433. In that case, border 423 could be retained.

[0110] Similarly, the user can, for example, suggest a change. This is exemplified in Figure 4b This is shown, specifically with line 434 for a modified border. The user can, for example, use a touch display of the input device 440 to move line 433 (e.g., at suitable points on the line), resulting in line 424.

[0111] In Figure 5The diagram schematically shows a procedure for determining a boundary that defines a working area for a mobile device, e.g., the robotic lawnmower 100, according to... Figures 1a, 1b or 2a , 2b , 2c , at least partially limited. This process will be explained in more detail below, also with reference to the figures discussed above.

[0112] In step 500, a first step, environmental information 502 of an environment in which the mobile device is moving or is intended to move is provided. This environmental information can include, for example, positions and orientations and may have been determined, for example, based on images captured by the mobile device's camera. This can be done, for example, using SLAM, in particular visual SLAM. The mobile device has been controlled externally, as described above, for example, with reference to... Figure 2aexplained. In particular, the mobile device may have been moved outside the desired work area onto a different surface, such as with regard to... Figure 2b explained.

[0113] In step 504, based on the environment information 502 from the first step, a boundary of the workspace within the environment is determined for the mobile device, as is the case, for example, with reference to Figure 2a This has been explained. This can be achieved, for example, using loop closure, especially if the start and end points of the externally controlled movement are the same or close together, e.g., at the docking station. This also provides a boundary.

[0114] In step 506, a second step, environmental information 508, e.g., images, of the environment, at least within the perimeter, is then provided. This environmental information or images 508 are obtained when the mobile device is moved again along and / or within the perimeter, as described above, e.g., with reference to Figure 2a explained.

[0115] In step 510, a map, specifically a visual map, of the surroundings can be determined based on the images 508. This can be done using ortho projection and segmentation. An example of such images is the one shown in Figure 3a Figure 300a shows an example of such a map. Figure 4a Map shown: 420.

[0116] In step 512, an ortho-projected view 514 of the surrounding images can be determined from the images 508, as for example with image 300b in Figure 3shown. In step 516, these ortho-projected images 514 or their views can then be segmented, as is done, for example, with image 300d in Figure 3d This is shown. This allows the map to be created and made available.

[0117] Based on the map or even just the segmented, ortho-projected images, either the existing outline (from the first step) can be confirmed (or verified) in step 518. Alternatively, the existing outline can be adjusted in step 520. Then, in step 522, the resulting outline (e.g., the confirmed or adjusted one) can be made available, for example, for further use.

[0118] Alternatively, it may also be provided that, in step 500', the environmental information 502 and images 508, obtained during the externally controlled movement, are made available. In step 510', a map, in particular a visual map, can be determined based on the images 508, as described above. In particular, steps 512 to 516 may also be provided.

[0119] Then, in step 518, the outline can be determined based on the environmental information and the images. Then, in step 522, this outline can be provided, for example, for further use.

[0120] Starting from the border provided in step 522 or 522' - or also from the border determined or provided in step 504, a modified, in particular extended, border can be determined in step 524, either based on the border from the first step and / or the environment information, in particular images, from the second step, or generally from an existing border, such as with reference to Figure 2a explained. Similarly, environmental information, especially images, can then be provided by a further movement (e.g., during an editing process).

[0121] The determination of the modified boundary is carried out primarily automatically. The modified boundary is then made available, particularly as a map, especially for a user, step 526, as e.g. with reference to Figure 4a explained.

[0122] The user can then, for example, make a change or correction, such as with regard to Fig. 4b This is explained. In step 528, correction information 530 is received from an external source (e.g., a smartphone). Then, in step 532, the border confirmed in step 518 or received in step 518' can be adjusted. Likewise, the adjusted map can be adjusted again according to step 520.

[0123] Similarly, the adjusted map can be obtained in this way according to step 520; that is, the adjusted map is only obtained after the user makes a correction. The adjusted border is then determined based on the border and the correction information. Alternatively, the modified border can also be determined without the correction information.

[0124] Based on the outline (e.g., the initially obtained, the adapted, or the modified outline), navigation information 536 for the mobile device can then be determined in step 534. Based on this, control information 540 for moving the mobile device can then be determined in step 538 and subsequently provided.

[0125] After obtaining the border, be it an initial border (see step 522 or 522'), an adapted or modified border (see step 532), an additional border, e.g. an inner border, can be determined based on this existing border.

[0126] For this purpose, in step 550, environmental information 552 (comparable to environmental information 502, e.g., position, orientation) and / or images 554 (comparable to images 508) of an environment are provided, in which the mobile device is moving or is intended to move. This environmental information or images are obtained when the mobile device moves, whereby the movement of the mobile device involves movement from a starting point, where the starting point corresponds to a point in the environment at which the mobile device is or has been stopped at an existing boundary during (especially automated) movement. This is with reference to Figure 2c explained in more detail.

[0127] Then, in step 556, based on the environmental information and / or the images, an additional boundary of the workspace within the environment for the mobile device is determined, such as also with reference to Figure 2cThis is explained in more detail below. This additional border can then be provided in step 558 and, for example, used again from step 534 onwards to move the mobile device. A modified border can also be defined for the additional border, as described above with regard to border 223 and the modified border 233.

Claims

1. Method for determining a perimeter (223, 233) that delimits at least part of a working area (122, 222) for a mobile device (100), in particular a vehicle or robot that moves in an at least partially automated manner, in particular a robot lawnmower, comprising: providing (500), in a first step, surroundings information (502) relating to surroundings (120) in which the mobile device moves or is meant to move, which information is or has been obtained during externally controlled movement of the mobile device, the surroundings information from the first step being obtained by means of one or more sensors of the mobile device and by means of SLAM, determining (504), on the basis of the surroundings information (502) from the first step, a perimeter (223) of the working area within the surroundings, for the mobile device, the perimeter being a provisional perimeter that can be used for a processing operation of a work function of the mobile device, providing (506), in a second step, surroundings information (508), namely images, relating to the surroundings at least in the region of the perimeter (223), which information is or has been obtained by means of the one or more sensors of the mobile device during renewed and automated movement of the mobile device along the perimeter and / or within the perimeter, the mobile device (100) performing a processing operation during the renewed movement of the mobile device along the perimeter and / or within the perimeter, and confirming (518) and / or adjusting (520) the perimeter (223) on the basis of the surroundings information from the second step.

2. Method according to Claim 1, wherein the surroundings information from the first step is obtained by means of visual SLAM.

3. Method according to Claim 2, wherein the perimeter is determined and / or refined on the basis of the surroundings information from the first step using loop closure.

4. Method according to one of the preceding claims, wherein the externally controlled movement of the mobile device comprises a movement from a starting point to an end point, the starting point and the end point being identical at least within specified tolerances.

5. Method according to one of the preceding claims, wherein the surroundings information from the first step comprises information, in particular images, concerning reference points in the surroundings, the information concerning the reference points in particular being used for determining the perimeter and / or for determining a visual map.

6. Method according to one of the preceding claims, wherein the surroundings information (508) from the second step comprises images of the surroundings, and wherein confirming and / or adjusting the perimeter comprises: determining (510) a map (420) of the surroundings, which comprises at least part of the perimeter, on the basis of the images from the second step.

7. Method according to one of the preceding claims, additionally comprising: determining (524) an altered, in particular extended, perimeter (233), on the basis of the perimeter (223) from the first step and / or the surroundings information, in particular images, from the second step, the altered perimeter being determined in particular in an automated manner, and providing (526) the altered perimeter, in particular as a map, in particular for a user.

8. Method according to Claim 7, additionally comprising: obtaining (528) correction information (530) from outside, and determining (532) the adjusted perimeter on the basis of the perimeter and on the basis of the correction information.

9. Method according to one of Claims 1 to 6, additionally comprising: determining an altered, in particular extended, perimeter, on the basis of the perimeter from the first step and / or the surroundings information, in particular images, from the second step, the altered perimeter being determined in particular in an automated manner, and determining the altered perimeter as the adjusted perimeter.

10. Method according to one of the preceding claims, additionally comprising: determining (534), on the basis of the perimeter and / or the map of the surroundings, navigation information (536) for the mobile device, determining (538), on the basis of the navigation information, control information (540) for moving the mobile device, and providing the control information and / or moving the mobile device on the basis of the control information.

11. Computing unit (108) comprising means for carrying out the method according to one of the preceding claims.

12. Mobile device (100) configured to obtain control information that has been determined by means of a method according to Claim 10, and / or having a computing unit according to Claim 11, and having a drive system (104) and an open-loop or closed-loop control unit (102) for controlling the drive system, and in particular having a capture means (106) for capturing surroundings information relating to surroundings in which the mobile device moves or is meant to move, the mobile device being in particular in the form of a vehicle moving in an at least partially automated manner, in particular in the form of a passenger transport vehicle or in the form of a goods transport vehicle, and / or in the form of a robot, in particular in the form of a household robot, e.g. a robot cleaner, a floor or road cleaning device or a robot lawnmower, and / or in the form of a drone.

13. Computer program comprising instructions that, when the program is executed by a computer, cause said computer to carry out the method according to Claims 1 to 10.

14. Computer-readable storage medium on which the computer program according to Claim 13 is stored.