Improved navigation for a robotic work tool system
Reinforcement learning with multi-scale maps and adaptive rewards enhances robotic work tool navigation in dynamic environments, addressing inefficiencies and collisions by enabling dynamic adaptation and improved coverage path planning.
Patent Information
- Application Number
- PCT/SE2024/050184
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-02-28
- Publication Date
- 2024-11-21
AI Technical Summary
Robotic work tools face challenges in navigating complex and dynamic outdoor environments with varying obstacles and surfaces, as existing navigation systems struggle with unknown obstacles and changing conditions, leading to inefficiencies and potential collisions.
Employing reinforcement learning for online coverage path planning, using multi-scale maps and sensor data to adaptively determine navigational actions, with rewards based on total variation and collision avoidance, enabling dynamic and efficient area coverage.
This approach allows robotic work tools to dynamically adapt to unknown obstacles and changing conditions, improving coverage path planning efficiency and reducing collisions, resulting in more effective and homogeneous area coverage.
Smart Images

Figure SE2024050184_21112024_PF_FP_ABST
Abstract
Description
[0001] IMPROVED NAVIGATION FOR A ROBOTIC WORK TOOL SYSTEM
[0002] TECHNICAL FIELD
[0003] This application relates to a robotic work tool and in particular to a system and a method for providing an improved navigation for robotic work tools, such as lawnmowers, in such a system.
[0004] BACKGROUND
[0005] Automated or robotic work tools such as robotic lawnmowers are becoming increasingly more popular and so is the use of the robotic work tool(s) in more and more complicated operational areas as well as larger and larger areas.
[0006] Furthermore, an operational area, such as a garden to be mowed, may have several obstacles that are to be avoided. Not only static or known obstacles, such as trees, flowerbeds, or stones, but also dynamic or unknown obstacles, for example garden furniture being moved, a barbeque or any other object that may be moved around or put in the garden. Such an operational area may also include surfaces of varying type, which makes straight-forward planning less than optimum as it comes to providing coverage path planning. Some examples of such surfaces are slopes where a robotic lawnmower may slide - even only slightly - whereby the robotic lawnmower may drift off the planned path. Furthermore, the obstacles may change or new obstacles may occur as the season or weather changes and an area that is easily covered in summer may not be easily covered during rain, such as early autumn, where the grass or ground may become slippery which will lead to the robotic lawnmower to diverge or drift from a planned trajectory.
[0007] Even if the problems discussed are mainly focused on a robotic lawnmower, it should be noted that similar problems also exist for other types of robotic work tools, especially robotic work tools adapted to operate in an outdoor environment, such as robotic watering robotics, robotic gold collectors, robotic snow removers, and so on.
[0008] Thus, there is a need for an improved manner of navigating an operational area to provide coverage path planning. SUMMARY
[0009] The inventors have realized that by employing or utilizing reinforcement learning while performing online coverage path planning (i.e. when the environment is not completely known beforehand) for a robotic work tool, a dynamic and adaptable coverage path planning may be achieved which reacts to unknown obstacles.
[0010] It is therefore an object of the teachings of this application to overcome or at least reduce those problems by providing by providing a robotic work tool robotic work tool configured to operate in an operational area, wherein the robotic work tool comprises a memory configured to store a map of the operational area, one or a plurality of sensors and a controller configured to control the navigation of the robotic work tool to cover the operational area, wherein the controller is configured to: perform coverage path planning based on reinforcement learning.
[0011] In some embodiments the controller is further configured to generate a local map of a subsection of the operational area, rotate the local map to align with the robotic work tool and determine a next navigational action based on the local map.
[0012] In some embodiments the local map indicates an aspect of the operation of the robotic work tool.
[0013] In some embodiments the aspect relates to local coverage.
[0014] In some embodiments the aspect relates to obstacles.
[0015] In some embodiments the controller is further configured to receive sensor input from the one or plurality of sensors and to determine a navigational action based on the sensor input according to the coverage path planning thereby combining the sensor data into the reinforcement learning.
[0016] In some embodiments a reward of the reinforcement learning is based on a total variation of the coverage of the robotic work tool.
[0017] In some embodiments the controller is further configured to determine a current total variation based on the covered area and compare the current total variation to a previous total variation and if the current total variation is larger than the previous total variation, a negative reward is determined, and conversely, if the current total variation is smaller than the previous total variation, a positive reward is determined. In some embodiments a reward of the reinforcement learning is based on a total covered area, wherein the controller is further configured to determine a current total covered area and compare the current total covered area to a previous covered area.
[0018] In some embodiments a reward of the reinforcement learning is based on a detected collision, wherein a negative reward is determined if a collision is detected.
[0019] In some embodiments the robotic work tool is configured to be provided with a trained reinforcement model and to perform the coverage path planning based on the trained reinforcement learning model.
[0020] In some embodiments the robotic work tool is adapted to operate in an outdoor environment.
[0021] In some embodiments the robotic work tool is a robotic lawnmower.
[0022] It is also an object of the teachings of this application to overcome the problems by providing a method for training a reinforcement model to be used in a robotic work tool, wherein the method comprises initialize weights, repeat for each time step: determine if a terminal state is reached, and if so reset a training environment, determine a current state, feed the current state to an agent, whereby the agent computes an action based on the state, compute a reward is then computed, whereby neural network parameters are updated based on a Reinforcement Learning method.
[0023] In some embodiments the method further comprises recording the state, the action and the reward in a replay buffer.
[0024] In some embodiments resetting the training environment includes setting a coverage to zero and randomize obstacles.
[0025] In some embodiments determining the current state includes determining local maps and sensor data.
[0026] In some embodiments the terminal state is a state where an obstacle collision is detected.
[0027] In some embodiments the terminal state is a state where an end of operation has been reached. As will be discussed in the below, a specific example where the use of this invention is investigated in detail will also be given. Task automation is an ever growing field, and as automated systems become more intelligent, they are expected to solve increasingly challenging problems. One such problem is coverage path planning (CPP), where the shortest path is sought that covers the entire free space of a given confined area. It is related to the traveling salesman and covering salesman problems, and known to be NP-hard. Applications include area maintenance tasks, such as robotic lawn mowing, snow removal, road sweeping, automated harvesting, vacuum cleaning, window cleaning, but also tasks such as automated spray painting, autonomous underwater exploration, machine milling, manufacturing inspection, aerial terrain coverage, surveillance, as well as public safety-related applications, such as robotic demining and search-and-rescue tasks. Each application presents different variations of the CPP problem, with varying objectives based on obstacle avoidance, movement and time constraints, as well as what is considered as covered. For example, the area covered by a lawn mower is defined by its cutting blade, while for search-and-rescue, the covered area is defined by what can be observed from the vehicle. Thus, the space of optimal paths differ between these two cases. The CPP problem has a rich past, where methods to solve it have been developed from as far back as 1986. Early works focus on offline CPP, where a path is planned before it is navigated by a robot. This requires full access to obstacle locations and to the geometry of the environment at planning time. Online methods, on the other hand, jointly consider path planning and navigation, and can 37 cover unknown environments. A common approach is to perform cellular decomposition to subdivide the area into simpler subareas which are easy to construct coverage paths for. Subsequently, these paths are joined together using graph-search techniques.
[0028] However, the resulting cell decompositions can limit path optimality, as the transport paths connecting two subareas may overlap with the coverage path. Recently, reinforcement learning-based approaches have been investigated. These have the potential to find paths that surpass the optimality limitations of classical online methods. In this work, we formulate the CPP problem as a reinforcement learning problem with continuous state and action spaces. We consider the online case, where the agent directly predicts continuous control signals for a robot. We apply our method on two variations of the CPP problem. (1) The lawn mowing problem, where the covered area is defined by the spatial extent of the agent, and requires precise navigation close to obstacles. (2) The exploration problem, where the covered area is defined by the reach of a light detection and ranging (lidar) sensor, and the task is to map the free space of an initially unknown environment. Our contributions can be summarized as follows:
[0029] There is proposed an end-to-end reinforcement learning method for the continuous online CPP problem.
[0030] There is proposed to use multi-scale maps for a scalable input representation.
[0031] There is introduced a novel total variation reward term for eliminating thin strips of uncovered space.
[0032] Finally, there have been conducted extensive experiments validating the effectiveness of the approach.
[0033] Further embodiments and aspects are as in the attached patent claims and as discussed in the detailed description.
[0034] Other features and advantages of the disclosed embodiments will appear from the following detailed disclosure, from the attached dependent claims as well as from the drawings. Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the [element, device, component, means, step, etc.]" are to be interpreted openly as referring to at least one instance of the element, device, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.
[0035] BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The invention will be described in further detail under reference to the accompanying drawings in which:
[0037] Figure 1 shows a schematic view of the components of an example of a robotic work tool being a robotic lawnmower according to some example embodiments of the teachings herein; Figure 2 shows a schematic view of a robotic work tool system according to some example embodiments of the teachings herein;
[0038] Figure 3 shows a schematic view of a robotic work tool system according to some example embodiments of the teachings herein;
[0039] Figure 4 shows a schematic view of a local maps according to some example embodiments of the teachings herein;
[0040] Figure 5 shows a flowchart for a method for training a reinforcement model according to some example embodiments of the teachings herein;
[0041] Figure 6 shows a schematic view of an agent-environment interaction. The observation consists of a multi-scale coverage map (green / white), obstacle map (blue / white), and lidar range detections (orange vector), based on which, the agent predicts control signals directly;
[0042] Figure 7 shows a schematic view of how an area is represented as separate coverage and known obstacle maps in multiple scales. This example shows m = 4 scales with a scale factor of s = 2. All scales are centered at the agent, and discretized into the same pixel resolution, which is 8 * 8 in this case, resulting in the two multi-scale map representations Me and Mo, both of size 8 ^ 8 x 4;
[0043] Figure 8 shows a schematic view of.how a reward is determined based on the teachings herein;
[0044] Figure 9 shows a schematic view of a convolutional agent architecture consisting of convolution layers with ReLU (conv) and fully connected layers with ReLU (fc). The scales of the multi-scale maps are convolved separately as their spatial positions are not aligned in the grid;
[0045] Figure 10 shows examples of randomly generated training maps;
[0046] Figure 11 shows maps used for training, in addition to the randomly generated maps; and
[0047] Figure 12 shows maps used for evaluation.
[0048] DETAILED DESCRIPTION
[0049] The disclosed embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Like reference numbers refer to like elements throughout.
[0050] It should be noted that even though the description given herein will be focused on robotic lawnmowers, the teachings herein may also be applied to other robotic work tools adapted to operate in an outdoor environment. In some embodiments the robotic work tool is a robotic lawnmower, robotic ball collector, robotic mine sweeper, robotic watering tool, robotic mulcher, robotic snow remover or other robotic work tool. In some embodiments the robotic work tool is a robotic grinder.
[0051] It should also be noted that for the purposes of the teachings herein a robotic work tool is taken to be an autonomous implementation of a work tool, a work tool being an implement used by a human in performing or completing a specific job. The teachings herein are aimed at work tools of a relatively small size compared to a human, and not related to other larger types of automated vehicles, which cannot be seen as work tools, especially since such machinery are exposed to completely different level of risks and also regulated by different safety standards.
[0052] Figure 1 shows a schematic overview of a robotic work tool 100, here exemplified by a robotic lawnmower 100. The robotic work tool 100 may be a multichassis type or a mono-chassis type (as in figure 1). A multi-chassis type comprises more than one main body parts that are movable with respect to one another. A monochassis type comprises only one main body part.
[0053] It should be noted that robotic lawnmower may be of different sizes, where the size ranges from merely a few decimetres for small garden robots, to even more than 1 meter for large robots arranged to service for example airfields. Again, the robot thus being of a size and exposed to risks completely different from those of large autonomous vehicles.
[0054] It should also be noted that the robotic work tool is a self-propelled robotic work tool, capable of autonomous navigation within a work area, where the robotic work tool propels itself across or around the work area in a pattern (random or predetermined). The robotic work tool 100, exemplified as a robotic lawnmower 100, has a main body part 140 and a plurality of wheels 130 (in this example four wheels 130, but other number of wheels are also possible, such as three or six).
[0055] The main body part 140 substantially houses all components of the robotic lawnmower 100. At least some of the wheels 130 are drivably connected to at least one electric motor 155 powered by a battery 150. It should be noted that even if the description herein is focused on electric motors, combustion engines may alternatively be used, possibly in combination with an electric motor. In the example of figure 1, each of the wheels 130 is connected to a respective electric motor 155 for driving the wheels 130 to navigate the robotic lawnmower 100 in different manners. The wheels, the motor 155 and possibly the battery 150 are thus examples of components making up a propulsion device. By controlling the motors 155, the propulsion device may be controlled to propel the robotic lawnmower 100 in a desired manner, and the propulsion device will therefore be seen as synonymous with the motor(s) 150.
[0056] It should be noted that wheels 130 driven by electric motors is only one example of a propulsion system and other variants are possible such as caterpillar tracks. As a skilled person would also understand, the robotic lawnmower 100 may be powered by a common motor whereby some or all of the wheels are driven by the motor through a system of axles and gears (not shown). Such a system is also seen as an example of a propulsion system.
[0057] The robotic lawnmower 100 also comprises a controller 110 and a computer readable storage medium or memory 120. The controller 110 may be implemented using instructions that enable hardware functionality, for example, by using executable computer program instructions in a general -purpose or special-purpose processor that may be stored on the memory 120 to be executed by such a processor. The controller 110 is configured to read instructions from the memory 120 and execute these instructions to control the operation of the robotic lawnmower 100 including, but not being limited to, the propulsion and navigation of the robotic lawnmower.
[0058] The controller 110 in combination with the electric motor 155 and the wheels 130 forms the base of a navigation system (possibly comprising further components) for the robotic lawnmower, enabling it to be self-propelled and to perform autonomous navigation as discussed herein.
[0059] The controller 110 may be implemented using any suitable, available processor or Programmable Logic Circuit (PLC). The memory 120 may be implemented using any commonly known technology for computer-readable memories such as ROM, FLASH, DDR, or some other memory technology.
[0060] The robotic lawnmower 100 is further arranged with a wireless communication interface 115 for communicating with other devices, such as a server, a personal computer, a smartphone, the charging station, and / or other robotic work tools. Examples of such wireless communication devices are Bluetooth®, WiFi® (IEEE802.1 lb), Global System Mobile (GSM) and LTE (Long Term Evolution), to name a few. The robotic lawnmower 100 may be arranged to communicate with a user equipment (as an example of a connected device) as discussed in relation to figure 2 below for providing information regarding status, location, and progress of operation to the user equipment as well as receiving commands or settings from the user equipment. Alternatively or additionally, the robotic lawnmower 100 may be arranged to communicate with a server (referenced 240 in figure 2) for providing information regarding status, location, and progress of operation as well as receiving commands or settings.
[0061] The robotic lawnmower 100 also comprises a grass cutting device 160, such as a rotating blade 160 driven by a cutter motor 165. The grass cutting device being an example of a work tool 160 for a robotic work tool 100. In embodiments where the robotic work tool 100 is exemplified as a robotic grinder, the work tool 160 is a rotating grinding disc.
[0062] The robotic lawnmower 100 may further comprise at least one signal navigation sensor 185 configured to provide navigational information (such as position) based on receiving one or more navigation signals.
[0063] In some embodiments the navigation sensor 185 is a beacon navigation sensor, such as a Radio Frequency receiver, such as an Ultra Wide Band (UWB) receiver or sensor, configured to receive signals from a Radio Frequency beacon, such as a UWB beacon. Alternatively or additionally, the beacon navigation sensor may be an optical receiver configured to receive signals from an optical beacon. In some embodiments the navigation sensor 185 is a satellite navigation sensor such as a GPS (Global Positioning System) device or other Global Navigation Satellite System (GNSS) device. In some embodiments the navigation sensor is a combination of one or more of the examples given above, for example a Real-Time Kinetics (RTK) sensor 185.
[0064] The robotic lawnmower 100 may also or alternatively comprise deduced reckoning sensors 180. The deduced reckoning sensors may be odometers, accelerometer or other deduced reckoning sensors. In some embodiments, the deduced reckoning sensors are comprised in the propulsion device, wherein a deduced reckoning navigation may be provided by knowing the current supplied to a motor and the time the current is supplied, which will give an indication of the speed and thereby distance for the corresponding wheel. In some embodiments the deduced reckoning sensors 180 comprises an optical navigation sensor, such as a camera-based sensor and / or a laserbased sensor. The robotic work tool 100 may thus also comprise additional deduced reckoning sensors 180, such as visual sensors (for example a camera) for enabling camera-based navigation and / or for enabling object detection, ranging sensors for enabling SLAM-based navigation (Simultaneous Localization and Mapping), moisture sensors, collision sensors, distance detecting sensors (for example camera, laser or radar), wheel load sensors to mention a few examples of sensors. Such deduced reckoning sensors 180 will provide the robotic lawnmower 100 with information about the surroundings that the robotic lawnmower can take into account while navigating the robotic lawnmower 100. The sensors are thus referred to as deduced reckoning sensors even if - traditionally - not being deduced reckoning sensors.
[0065] Optionally, for enabling the robotic lawnmower 100 to navigate with reference to a wire 220, such as a boundary wire or guide wire, emitting a magnetic field caused by a control signal transmitted through the wire, the robotic lawnmower 100 is in some embodiments configured to have at least one magnetic field sensor 170 arranged to detect the magnetic field and for detecting the wire. In some embodiments, the sensors 170 may be connected to the controller 110, possibly via filters and an amplifier, and the controller 110 may be configured to process and evaluate any signals received from the sensors 170. The magnetic sensor signals are caused by the magnetic field being generated by the control signal being transmitted through the wire. This enables the controller 110 to determine whether the robotic lawnmower 100 is close to or crossing the boundary wire 220, or inside or outside an area enclosed by the boundary wire 220.
[0066] The robotic lawnmower 100 is in some embodiments arranged to operate according to a map application representing one or more work areas (and possibly the surroundings of the work area(s)) stored in the memory 120 of the robotic lawnmower 100. The map application comprises indicators (such as coordinates) for a boundary - also referenced 220. The map application may be generated or supplemented as the robotic lawnmower 100 operates or otherwise moves around in the work area 205. In some embodiments, the map application includes one or more start regions and one or more goal regions for each work area. In some embodiments, the map application also includes one or more transport areas.
[0067] As discussed in the above, the map application is in some embodiments stored in the memory 120 of the robotic work tool(s) 100. However, alternatively or additionally, in some embodiments, the map application is stored in the server (referenced 240 in figure 2). In some embodiments, maps are stored both in the memory 120 of the robotic work tool 100 and in the server 240, wherein the maps may be the same maps or show subsets of features of an area.
[0068] Figure 2 shows a robotic work tool system 200 in some embodiments. The schematic view is not to scale. The robotic work tool system 200 comprises one or more robotic work tools 100 according to the teachings herein. It should be noted that the operational area 205 shown in figure 2 is simplified for illustrative purposes.
[0069] The robotic work tool system 200 further comprises a station 210 possibly at a station location. A station location may alternatively or additionally indicate a service station, a parking area, a charging station or a safe area where the robotic work tool may remain for a time period between or during operation session.
[0070] The robotic work tool system comprises a boundary 220 that may be electro mechanical (such as a boundary wire) and / or virtual (such as defeined by map coordinates or other coordinates). An example of an electro mechanical boundary is a magnetic field generated by a control signal being transmitted through a boundary wire, and which magnetic field is sensed by sensor in the robotic work tool 100. In some embodiments the control signal is generated by a signal generator comprised in the station. An example of a virtual boundary is a set of coordinates representing or defining a geofence that can be navigated using the signal navigation sensor 180.
[0071] As with figure 1, the robotic work tool(s) is exemplified by a robotic lawnmower, whereby the robotic work tool system may be a robotic lawnmower system or a system comprising a combinations of robotic work tools, one being a robotic lawnmower, but the teachings herein may also be applied to other robotic work tools adapted to operate within a work area.
[0072] The one or more robotic work tools 100 of the robotic work tool system 200 are arranged to operate in an operational area 205. However, it should be noted that an operational area may comprise a single work area or one or more work areas, possibly arranged adjacent for easy transition between the work areas, or connected by one or more transport paths or areas, also referred to as corridors. In the following work areas and operational areas will be referred to interchangeably, unless specifically indicated.
[0073] The operational area 205 is in this application exemplified as an outdoor environment, such as a garden, but can also be other work areas as would be understood, such as an airfield. As discussed above, the garden may contain a number of obstacles, for example a number of trees (T), rocks (R), slopes (S) and houses (H) or other structures.
[0074] In some embodiments the robotic work tool is arranged or configured to traverse and operate in work areas that are not essentially flat, but contain terrain that is of varying altitude, such as undulating, comprising hills or slopes or such. The ground of such terrain is not flat and it is not straightforward how to determine an angle between a sensor mounted on the robotic work tool and the ground. The robotic work tool is also or alternatively arranged or configured to traverse and operate in a work area that contains obstacles that are not easily discerned from the ground. Examples of such are grass- or moss-covered rocks, roots or other obstacles that are close to ground and of a similar colour or texture as the ground. The robotic work tool is also or alternatively arranged or configured to traverse and operate in a work area that contains obstacles that are overhanging, i.e. obstacles that may not be detectable from the ground up, such as low hanging branches of trees or bushes. Such a garden is thus not simply a flat lawn to be mowed or similar, but a work area of unpredictable structure and characteristics. The work area 205 exemplified with referenced to figure 2, may thus be such a non-uniform work area as disclosed in this paragraph that the robotic work tool is arranged to traverse and / or operate in.
[0075] The robotic work tool system 200 may alternatively or additionally comprise or be arranged to be connected to a server 240, such as a cloud service, a cloud server application or a dedicated server 240. The connection to the server 240 may be direct from the robotic work tool 100, direct from a user equipment 250, indirect from the robotic work tool 100 via the service station, and / or indirect from the robotic work tool 100 via the user equipment.
[0076] A skilled person would understand that a server, a cloud server or a cloud service may be implemented in a number of ways utilizing one or more controllers 240A and one or more memories 240B that may be grouped in the same server or over a plurality of servers. The server 240 also comprises a communication interface 250C for communicating with the robotic work tool.
[0077] The user equipment 250 may be a smartphone, a tablet computer or a remote control panel, and comprises one or more controllers 250A and one or more memories 250B for storing instructions that when executed by the controller 250A controls the operation of the user equipment. The user equipment 250 also comprises a communication interface 250C for communicating with the robotic work tool.
[0078] In the below, several embodiments of how the robotic work tool may be adapted will be disclosed. It should be noted that all embodiments may be combined in any combination providing a combined adaptation of the robotic work tool.
[0079] In general, the inventors have realized that an improved area coverage may be achieved by utilizing reinforcement learning when performing coverage path planning, and especially for online coverage path planning.
[0080] Coverage path planning (CPP) is the problem of finding the shortest path that visits every point in an operational area. There are two main aspects of coverage path planning, namely the offline and online planning. In offline coverage path planning, the complete environment is known beforehand, and a path is planned prior to starting the task. This approach, however, does not take into account unknown or dynamic obstacles. Online coverage path planning on the other hand does. In online coverage path planning all the obstacles are not assumed to be known beforehand.
[0081] Reinforcement Learning (RL) is a machine learning technique. It consists of an agent (e.g. a robotic lawn mower in the case of lawnmowing) and a system. The agent operates in the environment in order to solve a task at hand. In alternating turns, the agent performs an action, e.g. drive forward or turn left, and the system responds by providing a reward for how good the action was together with a new state, e.g. sensor readings. By optimizing the expected reward, the agent can solve the task.
[0082] Figure 3 shows a simplified view of a robotic work tool system 200 as in figure 2, where a coverage path is indicated by dashed arrows. However, due to environmental issues, even if the robotic lawnmower 100 knows the boundaries and even the obstacles of the operational area, the service provided in a subsection, such as in the slopes, may be of inferior quality or below expected results, for example, the robotic lawnmower 100 may experience a slight drift (indicated by bold dotted line referenced D) in the slope which results in uneven service for example resulting in stripes of uncut grass in the slope. Such results may then be indicated to the robotic lawnmower 100 as a negative result whereby negative feedback is provided in the reinforcement learning. As a result, the robotic lawnmower 100 will learn to compensate for the slope in the future. The slope may be determined through sensor input, for example through an inclinometer or through visual comparison of a horizon. The compensation for the slope is an example of a continuous action as the slope will or is likely to vary. The robotic lawnmower 100 has thereby learnt an improved manner of navigating in the operational area that will be beneficial in future operating turns. This has been achieved by the robotic lawnmower utilizing machine learning - also during operation - for enabling an improved coverage path planning utilizing reinforcement learning and by utilizing continuous actions. As would be understood, the actions of a robotic lawnmower 100 are inherently continuous, such as continuous throttle and steering inputs, or where continuous voltages are applied to the drive wheels.
[0083] Other examples of continuous actions include, but are not limited to braking signal, wheel torque, wheel speeds, linear and angular velocity of the robot, next waypoint that should be navigated to by a separate waypoint navigation algorithm (such as pure pursuit), in which case the navigation algorithm determines the control signals.
[0084] These actions are thus examples of navigational actions.
[0085] The robotic work tool is thus configured to perform a coverage path planning while operating in an operating area utilizing reinforcement learning based on sensor inputs and continuous actions determined by the controller 110.
[0086] The inventors have further realized that the reinforcement learning approach is highly beneficial if continued learning is provided, ie if the reinforcement learning also is continued as the robotic lawnmower is set to operate. So, even if a basic training is provided through for example simulations, the robotic work tool is also configured to continue its training -during training sessions in the operational area where it will be set to operate, and / or during operation. To enable such continuing learning, it is advantageous if the learning is performed in the robotic work tool 100, as opposed to in a server 240. For this purpose the inventors are proposing a manner of reducing the computational requirements of the training. It should be noted that a system where the training is also or alternatively performed in the server 240, the system will also benefit from the reduced computations provided by the teachings herein, for example in that les data need be provided between the robotic work tool 100 and the server 240 and in that the server 240 will be able to train multiple robotic work tools 100 simultaneously.
[0087] To enable for improved reinforcement learning, the inventors are therefore proposing to reduce the map that is being operated on by replacing the map 120A of the operational area stored in the memory 120 with a smaller or local map 121 showing only the local vicinity of the robotic work tool 100. To further reduce the computational complexity, the local map 121 Is in some embodiments further divided into local aspect maps 121-1, 121-2, each local aspect map 121-1, 121-2 only showing (one or more) specific aspect(s) of the local vicinity. One example is an obstacle map 121-1 showing (only) the (known) obstacles in the local vicinity. Another example is a local border map 121-2 showing (only) the borders 220 in the local vicinity. In some embodiments a local aspect map 121 is a copy of the global map 120A, but at a reduced resolution.
[0088] Figure 4 shows one example of such a local map 121 comprising two local aspect maps 121-1, 121-2. In the example of figure 4 a first local aspect map 121-1 shows the obstacles local to the current position of the robotic lawnmower 100. In this example the slope S and a tree T are indicated in the first local aspect map 121-1. Also in the example of figure 4 a second local aspect map 121-2 shows the borders local to the current position of the robotic lawnmower 100. In this example two border segments 220A, 220B are indicated in the second local aspect map 121-2. Also in the example of figure 4 a third local aspect map 121-3 shows the coverage (i.e operated area) local to the current position of the robotic lawnmower 100. In this example the area to the left of the robotic lawnmower 100 and directly behind the robotic lawnmower 100 have been covered.
[0089] To further reduce the computational requirements, the local maps are scaled and transformed to align with the direction of the robotic work tool 100. In the example of figure 4, this is indicated by the slopes and the border segments being at different (absolute) angles than in figure 3. In figure 4, they are shown relative the robotic lawnmower 100.
[0090] The inventors are thus proposing to use multi-scale maps for representing the local area. Using a single map, such as the global map 120A, would be computationally infeasible for large operational areas as the computational cost scales quadratically. The local maps are translated and rotated such that the center of each map corresponds to the position of the robotic work tool, and the forward direction is always in the same direction in the local maps. The robotic work tool 100 is thus also, in some embodiments, configured to operate based on local maps that are translated and rotated.
[0091] In some embodiments one local map represents the global map of the operating area. In some such embodiments, that map is at a lower resolution. Furthermore in such embodiments one local map represents the local sensory inputs. This allows for planning both in a long-term perspective and in a short term perspective. More on this later on.
[0092] To accommodate for the robotic work tool 100 moving the local maps 121 are updated regularly. In some embodiments where the reinforcement learning is based on time steps, the local maps 121 are updated at each time step. In some embodiments, the local maps are updated at time steps where the time steps are determined to correspond to a movement of the robotic work tool relative the area of the local map 121. In some such embodiments the time step Is determined based on the size of the local map 121, and / or based on the speed of the robotic work tool 100. Figure 4 shows two rows of local maps, where different instances of the local maps are shown. In the example of figure 4 two rows are shown (an upper and a lower row). In the upper row, a first instance of the local maps 121-1A, 121-2A, 121-3A is shown, and in the lower row a second instance of the local maps 121-1B, 121-2B, 121-3B is shown. As can be seen, the maps have been updated so that the robotic work tool 100 has moved relative the obstacles S, T, and the borders 220, while the local maps 121 are still centered on the robotic work tool 100.
[0093] The local maps are thus updated, for example in each time step. Separate local maps 121-1, 121-2, 121-3 are used for the coverage and known obstacles and possibly other aspects. The coverage map 121-3 keeps track of which positions have been visited. The known obstacle map 121-1 keeps track of known static obstacles or inaccessible areas, e.g. houses or ponds. In this example, two scales are used, i.e. a global scale and a local scale. However, any number of scales can be used, depending on the maximum considered size of the area.
[0094] The robotic work tool 100 is thus allowed or set to operate in the operational area and continuously (re)train using the local maps and input received from the various sensors 180 and 185. The training is performed on matching sensor input and comparing to the local maps and basing the rewards of the reinforcement learning on these sensor inputs (and the comparison. For example a positive reward is given if the navigational sensors indicate that an area to be covered has been covered, and a negative reward is given if an obstacle is collided with. The sensor input from the deduced reckoning sensors 185 will over time be matched to the actions and the success of the actions. For example, f the odometers indicate a wheel slip, an action of countersteering will result in a better coverage which in turn will result in a positive reward, whereby the robotic work tool 100 will learn to counter-steer if a slip is determined.
[0095] To enable the robotic work tool 100 to handle unknown obstacles, the robotic work tool 100 is enabled to detect obstacles. To detect such unknown obstacles, (raw) sensor data (from camera, lidar, radar, ultrasonic sensors or other of the sensors 185) is in some embodiment added as an input to the learning. This will lead to the robotic work tool 100 being trained to take evasive action based on certain sensor input as if not, a collision may occur, whereby a negative reward is given.
[0096] The robotic work tool is thus configured to fuse or combine sensor data into a learning model. The sensor input may be from (as discussed in the above) sensors 185 such as a camera, a lidar, a radar, an ultrasonic sensor etc. The robotic work tool 100 uses all this data to compute the next action. The inventors have thus provided a system wherein local and global maps are fused together with sensor data, for a self-learning method to optimize a path.
[0097] The inventors are proposing to base the reward in some embodiments on total variation.
[0098] The total variation is the variation between adjacent points, summed over the spatial dimensions. In the reward term, the total variation reward is determined based on the total variation of the coverage map. More details on this in the below.
[0099] In some embodiments a reward term is thus based on total variation.
[0100] In some embodiments the total variation reward is determined for each time step, wherein the (current) total variation is determined based on the covered area and compared to the (previous) total variation in the previous time step. If the current total variation is larger than the previous total variation, a negative reward is given, and conversely, if the current total variation is smaller than the previous total variation, a positive reward is given.
[0101] In some embodiments the total variation reward is determined based on the global variation, i.e. the variation for the whole operating area.
[0102] This encourages the robotic work tool 100 to minimize the total variation which results in a more homogeneous coverage, without stripes of uncovered parts left behind.
[0103] Additionally or alternatively, other reward terms are also used in some embodiments. In some embodiments the reward is based on a total covered area, where the total covered area in the current time step is compared to the covered area in the previous time step. In some embodiments the reward is based on detecting edges (or other discontinuous sections). In such embodiments, an edge detector may be utilized and the edge detections are summed up and compared to previous edge detections In some embodiments the reward is based on obstacle collisions, where a large negative reward is given if the robotic work tool 100 collides with an obstacle.
[0104] The robotic work tool 100, and especially the controller 110 of the robotic work tool 100, is thus configured to execute a machine-learning based on reinforcement learning for providing coverage path planning in an improved manner. In some embodiments the controller 110 executes the machine-learning based on a deep-learning model such as a deep-learning neural network model.
[0105] In some embodiments the robotic work tool (or a model that is later to be distributed to the robotic work tool 100) is trained in the following manner. Such training is based on simulated operating environments. Figure 5 shows a flowchart for such a training.
[0106] Firstly, neural network weights are initialized 510. Then the following loop is repeated for each time step. Determine 520 if a terminal state is reached. A terminal state is in some embodiments a state where an obstacle collision is detected. A terminal state is in some embodiments a state where an end of operation has been reached, such as operation for a given operating length or time has been performed. A terminal state is in some embodiments a state where no new operating surface has been detected in a time period, such as in 5, 10 or 100 seconds. A terminal state is in some embodiments a state where an operating goal has been reached, such as a total coverage of 90, 95, 99 or 100 % coverage of the total operating area has been achieved.
[0107] If a terminal state has been reached the training environment is reset 530, whereby the coverage is set to zero and obstacles are randomized and so on.
[0108] Then the current state is determined 540. This includes determining the multiscale maps (i.e. the local maps) and the sensor data. The state is fed to the agent (representing the robotic work tool 100), whereby the agent computes 550 its action based on the state. The reward is then computed 560 and the state, the action and the reward is recorded in a replay buffer, whereby the neural network parameters are updated 570 based on existing Reinforcement Learning method.
[0109] After training, the agent has learned to cover the area while avoiding obstacles. To improve generalization, the environments during training should be as varied possible. The trained agent can then be deployed in a robotic work tool 100 set to operate in a previously unseen area and still perform the task at hand in a satisfactory manner.
[0110] Alternatively, if an agent is to operate only at a single site, a robotic work tool 100 with an untrained agent could be deployed directly. Its performance would suffer in the beginning but it would learn over time. In this case, the training could be done indefinitely, where it can learn the site in-and-out, and find the optimal path over time.
[0111] If a certain pattern is desired, one could construct a map containing the desired travel direction at each position. A directional reward term is then included during training, where a negative reward is given if the current travel direction deviates from the desired travel direction at the current position. The travel direction map would be represented in the same multi-scale manner in the local maps and included in the state input to the agent.
[0112] An additional map can be added which contains detected obstacles based on the sensor data, and updated at each detection. This gives the agent the chance to remember previously seen obstacles and plan accordingly.
[0113] Finally, the total variation reward term could also be used for solving other problems based on reinforcement learning, such as controlling a motor or servo, where the total variation reward would prevent unwanted sporadic behaviors.
[0114] In the below a discussion on a specific and optimized approach to implement the teachings herein will be given. The agents are - as a skilled person would understand - implemented as the robots and in particular the robotic lawnmowers discussed herein.
[0115] In this section, we present our approach for solving the continuous online CPP problem using reinforcement learning end-to-end. Formulating the online CPP problem as an end-to-end reinforcement learning problem in continuous space presents several challenges, including path optimality, computational scalability, and loss convergence. In this section, we describe how we approach these challenges at each component of the reinforcement learning framework, as illustrated in Figure 6. We construct a suitable state representation based on coverage and obstacle maps, which are condensed into a scalable input representation for the observation space, to which sensory inputs are appended. We define separate map and sensor feature extractors for this multi-modal input space. In order not to limit path optimality by the use of discrete actions, we consider a continuous action space, where the agent predicts control signals directly. Finally, we introduce a total variation reward term for reducing thin strips of uncovered space, in addition to area coverage and obstacle collision rewards.
[0116] We formulate the CPP problem as a partially observable Markov decision process (POMDP) with discrete time steps t G [1..T], continuous actions at ~ p(at|ot) made by the agent, observations and rewards provided by the environment, and new state nt is to maximize the expected discounted reward
[0117] Observation space
[0118] The full state representation consists of three maps, namely a coverage map, a known obstacle map, and an unknown obstacle map. The coverage map keeps track of visited locations. It is empty initially, and is updated at each time step to reflect which parts of the free space that have been covered. The known obstacle map is static, and shows which parts of the confined area are known to be non-free space. The last map contains information about unknown obstacles, and further defines parts of the area that are non-free space. To mimic a realistic setting, the agent can only observe the coverage and known obstacle maps directly, while the unknown obstacles can be observed through sensors. In this work, we consider a lidar sensor for this purpose.
[0119] Multi-scale maps
[0120] Since our agent is a deep neural network, we intend to represent the maps as images of a fixed image pixel resolution. However, this presents a scalability challenge. If the entire area is to be observed at once, and we consider a fixed map resolution in pixels per meter, then for a square area with side length d, the input image pixel resolution is O(d2). This is infeasible for large-scale environments. Another option is to define the input space as a local crop, both with fixed map resolution and fixed image pixel resolution. However, this limits the long-term planning potential as the space beyond the reach of the local crop can not be observed, and thus can not be accounted for in the path planning. On the contrary, a previous approach kept the pixel resolution fixed, while adapting the map resolution based on the size of the area. This, however, means that the pixel information varies between environments, and can hinder learning and generalization for differently sized environments.
[0121] We propose to use maps in multiple scales to solve the scalability challenge. We start with a local square cropM(l) for the smallest scale, where both the map resolution and the image pixel resolution are fixed. Let dl be its side length in meters. The full multi-scale map representation {M(i)}mi=l with m scales is constructed by cropping increasingly larger areas based on a fixed scale factor s between scales. Concretely, the side length of map M(i+1) is di+1 = sdi, where di is the side length of map M(i). To keep memory consumption under control, the image pixel resolution w x h is kept fixed for all scales, meaning that the map resolution decreases as the scale increases. Furthermore, each map M(i) is aligned such that the center of the map corresponds to the position of the agent, and the forward direction of the agent corresponds to the upward direction in the map. The map resolution is chosen sufficiently high for the smallest scale such that the desired level of detail is attained in the nearest vicinity of the agent. Thus, the level of detail is highest in the closest neighborhood of the agent, allowing it to perform precise local navigation, while simultaneously allowing it to perform long-term planning based on the large-scale maps where a high map resolution is less important. This multi-scale map representation can completely contain an area of size d x d in O(log d) number of scales. Thus, the total number of pixels is 0(wh log d), which is a significant improvement over a single map with a fixed map resolution, containing O(d2) pixels, especially since w and h are fixed. For the observation space, we use a multi-scale map Me for the coverage, as well as a multi-scale map Mo for the known obstacles. This is illustrated in Figure 7.
[0122] Sensor observations
[0123] Since unknown obstacles cannot be mapped beforehand, we include sensor data in the observation space. We consider a lidar sensor, which measures the distance from the agent to the first encountered known or unknown obstacle for a set of angles relative to the agent pose. The distance measurements are normalized to [0, 1] based on the maximum range, and concatenated into a vector S, where the i :th element Si contains the distance measurement for the i :th angle.
[0124] To keep track of the unknown obstacles over a longer time period, distance measurements are recorded in a spatial detection map D. This map is empty initially, and updated in each time step, where a point is added at the corresponding position of the detection. Furthermore, if an obstacle is no longer detected, previous detection points are removed from the map if a lidar ray passes through them without triggering a new detection. The detections in D are combined with the known obstacles when constructing the multi-scale known obstacle map Mo, and thus, dynamic obstacles can be accounted for by the agent.
[0125] Action space
[0126] We consider a continuous action space, where the agent predicts actions that are directly related to the control signals of the robot. The control signals are the linear velocity v and the angular velocity co.
[0127] To keep a fixed output range for the agent, the actions are normalized control signals av, aco G [-1, 1],
[0128] The actions are scaled by the maximum linear and angular velocities, i.e. v = avvmax m / s and co = acocomax rad / s. The sign of the linear velocity determines the direction of travel, i.e. forward or backward, while the sign of the angular velocity determines the direction of rotation, i.e. clockwise or counterclockwise.
[0129] Reward function
[0130] The ultimate goal is to train an agent to cover as much of an area as possible in the shortest possible time, while avoiding obstacle collisions. To accomplish this task, we formulate several reward terms that coincide with our objectives. For the main goal of covering as much of the area as possible, we use a reward term Rarea based on the size of the area Anew that was covered in the current time step, and that was previously not covered. To control the reward contribution, we first normalize it to [0, 1] by dividing the area with the maximum area that can be covered in a time step, based on the maximum speed vmax. We subsequently multiply it with the scale factor Rarea, max, which reflects the highest possible reward for this term, resulting in the reward where r is the radius of the agent and At is the time step size. See Figure 8 for an illustration of this reward term.
[0131] Furthermore, the agent is rewarded with a positive reward Rgoal when achieving a predefined goal coverage, at which point the episode is ended. The episode is also ended whenever T consecutive time steps have passed without the agent having covered any new space. At this point, a negative truncation reward is given, based on the available free space not yet covered, which is expressed as where Afiree is the area of the free space that can be covered, Acovered is the area of the covered space, and Rtrunc, scale is a reward scaling parameter.
[0132] In initial experiments, it was found that the agent left behind small stripes of uncovered space, as it would leave a small gap when driving adjacent to a previously covered part. This is reasonable as this behaviour maximizes the short-term reward by not overlapping the previous path. Covering the thin stripes afterward only yields a minor reward, resulting in slow convergence towards an optimal path. To improve the convergence rate and the quality of the learned path, we propose a reward term based on minimizing the discrete 2D isotropic total variation (TV), which has been effectively used in image denoising. Given a 2D signal x, the isotropic total variation is expressed
[0133] We consider two versions of the TV reward term, a global and an incremental. For the global TV reward RTV,g, the agent is given a reward based on the TV of the global coverage map C. To avoid an unbounded TV for large environments, the total variation is scaled by the square root of the covered area Acovered, as this results in a constant TV for a given shape of the coverage map, independent of scale. The global TV given at each time step is where RTV, scale is a TV reward scaling parameter to make sure that |RTV| < |Rarea|. Otherwise, the optimal behaviour is simply to stand still. The incremental TV reward RTV,i is based on the difference in TV between the current and the previous time step. A positive reward is given if the TV is decreased, while a negative reward is given if the TV is increased. The incremental TV reward is given by where C(t) is the coverage map at time step t.
[0134] To avoid obstacle collisions, a constant negative reward Rcoll is given each time the agent collides with an obstacle. Finally, a small constant negative reward Rconst is given in each time step to encourage fast execution. Thus, our final reward function is written as
[0135] Agent architecture
[0136] The reinforcement learning loop is depicted in Figure 9. The agent predicts control signals (v, co) = f(Mc,Mo, S) based on the multi-scale coverage map Me, obstacle map Mo, and lidar detection vector S, where f is a deep neural network with parameters 9. Since the different inputs consist of significant structural differences, we construct f from a map feature extractor gm, a sensor feature extractor gs, and a fusing module gf, with parameters 9m, 9s, and 9f respectively. The map and sensor features are fused, resulting in the control signal prediction
[0137] We consider two network architectures, a simple multilayer perceptron (MLP), and a convolutional neural network (CNN). For the MLP, the feature extractors, gm and gs, are identity functions that simply flatten the inputs. The fusing module consists of two fully connected layers with ReLU, of 256 units each. For the CNN architecture, since the pixel positions between different scales do not correspond to the same world coordinate, we group the maps in Me and Mo by scale, and convolve each scale separately using grouped convolutions. This makes sure that pixels are always compared to pixels that correspond to the same spatial position. The map feature extractor additionally includes a final fully connected layer. The sensor feature extractor, gs, is a single fully connected layer, and the fusing module consists of three fully connected layers. ReLU is used as the activation function. The CNN architecture is depicted in Figure 9, and full details can be found in the supplementary material.
[0138] Environments
[0139] To increase the variation of scenarios encountered during training, we use both randomly generated maps (see figure 10), as well as a fixed set of training maps (see figure 11). The fixed maps provide some simple, and some more challenging scenarios that a CPP method is expected to solve. The randomly generated maps provide orders of magnitude more variation, and avoid the problem of overfitting to the fixed maps. They are created by randomizing grid-like floor plans, together with randomly placed obstacles. Aspects that are randomly chosen include the size of the map, whether to use obstacles, the number of obstacles, whether to use random floor plans, wall thickness, and number of rooms. For evaluation, we used separate fixed maps (see figure 12) for mowing and exploration that are different from the fixed training maps. The purpose of the evaluation maps is to evaluate the agent on scenarios that are of interest for the two tasks.
[0140] Herein is thus presented a method for solving the online coverage path planning (CPP) problem in unknown environments, based on an end-to-end reinforcement learning (RL) approach. The agent has to implicitly solve three tasks simultaneously, namely mapping of the environment, planning a coverage path, and navigating the planned path while avoiding obstacles. For a scalable solution, there is proposed to use multi-scale coverage and obstacle maps. Furthermore, there is proposed a novel reward term based on total variation, to reduce thin strips of uncovered space. This results in a general real-time method for CPP that can be applied to different applications, as has been demonstrated for the lawn mowing and exploration problems. Experiments show that the method herein compares favorably to prior art RL-based approaches.
Claims
CLAIMS1. A robotic lawnmower (100) configured to operate in an operational area (205), wherein the robotic lawnmower (100) comprises a memory (120) configured to store a map (120A) of the operational area (205), one or a plurality of sensors (170, 180, 185) and a controller (110) configured to control the navigation of the robotic lawnmower (100) to cover the operational area (205), wherein the controller (110) is configured to: perform coverage path planning based on reinforcement learning.
2. The robotic lawnmower (100) according to claim 1, wherein the controller (110) is further configured to generate a local map (121) of a subsection of the operational area (205), rotate the local map to align with the robotic lawnmower (100) and determine a next navigational action based on the local map (121).
3. The robotic lawnmower (100) according to claim 2, wherein the local map indicates an aspect of the operation of the robotic lawnmower (100).
4. The robotic lawnmower (100) according to claim 3, wherein the aspect relates to local coverage and / or obstacles.
5. The robotic lawnmower (100) according to any preceding claim, wherein the controller (110) is further configured to receive sensor input from the one or plurality of sensors (170, 180, 185) and to determine a navigational action based on the sensor input according to the coverage path planning thereby combining the sensor data into the reinforcement learning.
6. The robotic lawnmower (100) according to any preceding claim, wherein a reward of the reinforcement learning is based on a total variation of the coverage of the robotic lawnmower (100).
7. The robotic lawnmower (100) according to claim 6, wherein the total variation is an incremental total variation, wherein the total variation reward is based on the difference in total variation between a current and a previous time step.
8. The robotic lawnmower (100) according to claim 7, wherein the controller (110) is further configured to determine a current total variation based on the covered area and compare the current total variation to a previous total variation and if the current total variation is larger than the previous total variation, a negative reward is determined, and conversely, if the current total variation is smaller than the previous total variation, a positive reward is determined.
9. The robotic lawnmower (100) according to claim 6, wherein the total variation is a global total variation, wherein the total variation reward is based on the total variation of a global coverage map.
10. The robotic lawnmower (100) according to any preceding claim, wherein a reward of the reinforcement learning is based on edge detection, and / or wherein a reward of the reinforcement learning is based on a total covered area, wherein the controller (110) is further configured to determine a current total covered area and compare the current total covered area to a previous covered area, and / or wherein a reward of the reinforcement learning is based on a detected collision, wherein a negative reward is determined if a collision is detected.
11. A method for training a reinforcement model to be used in a robotic lawnmower (100), wherein the method comprises initialize weights, repeat for each time step: determine if a terminal state is reached, and if so reset a training environment, determine a current state,feed the current state to an agent, whereby the agent computes an action based on the state, compute a reward is then computed, whereby neural network parameters are updated based on a Reinforcement Learning method, wherein resetting the training environment includes setting a coverage to zero and randomize obstacles.
12. The method according to claim 11, wherein the method further comprises recording the state, the action and the reward in a replay buffer.
13. The method according to claim 11, or 12, wherein determining the current state includes determining local maps and sensor data.
14. The method according to any of claims 11 to 13 wherein a terminal state is a state where an obstacle collision is detected and / or a state where an end of operation has been reached.
Citation Information
Patent Citations
Semiconductor device and manufacturing method of the same
KR1020230147330A
Cited By
Complex curved surface turning machining path planning method based on survey map
CN120722837A
Improved scheduling of operation for a robotic lawnmower
EP4793730A1
Improved scheduling of operation for a robotic lawnmower
SE2550152A1
Improved scheduling of operation for a robotic lawnmower
SE548445C2
