Local region mapping for robotic lawnmower

Abstract

A robotic lawn mower and a method of controlling the robotic lawn mower based on a generated local map. The method includes receiving an image from an imaging sensor on the robotic lawnmower, the image including a ground surface in an imminent path; applying a semantic segmentation algorithm to generate a segmented image from the received image, the segmented image including a region corresponding to a feature in the image; applying perspective transformation to the segmented image to obtain an overlook transformed image, determining the position of the region relative to the current position of the robotic mower from the transformed image, and drawing a local map of the robotic mower environment according to the position of the region relative to the current position of the robotic mower; and controlling the robot mower to navigate the lawn area by using the local map.

Description

Technical Field

[0001] The present application relates to an apparatus, system and method for autonomously controlling a robotic lawn mower. Background Art

[0002] A robot or robotic lawn mower is an autonomous robot that is configured to cut grass on a lawn or field. With a robotic lawn mower, there is no need for humans to personally mow the grass, which can be a mundane and tedious task.

[0003] Figure 1 An example of a lawn area in which the robot lawn mower can operate is shown. The robot lawn mower 1 mows within a lawn area 3 designated by boundary lines or guide lines 4. Before starting the robot lawn mower, the boundary lines or guide lines 4 need to be physically installed. 1. After the robot lawn mower 1 detects the boundary lines 4, it usually changes direction randomly to avoid the boundary lines.

[0004] The robotic lawn mower may include multiple sensors for detecting obstacles. For example, these sensors may be laser or radar based sensors. Likewise, the robotic lawn mower may rely on satellite communications to navigate within the lawn area and may include radio sensors for navigating to a charging station. Figure 1 In the embodiment, the charging station 2 is located outside the lawn area 3 .

[0005] However, there is a need to strike a balance between the complexity of a robotic lawn mower and its relative capabilities. Robotic lawn mowers that include different types of sensors or that include more complex systems are generally more expensive. Therefore, there is a need to provide a robotic lawn mower that can work efficiently without requiring overly complex systems and sensors. Summary of the invention

[0006] This summary introduces some concepts in a simplified form that are further described in the detailed description below.

[0007] In a first aspect, the present disclosure provides a computer-implemented method for controlling a robotic lawn mower, the method comprising: receiving an image from an imaging sensor located on the robotic lawn mower, the image comprising a ground area in a forward path of the robotic lawn mower; applying a semantic segmentation algorithm to generate a segmented image from the received image, the segmented image comprising areas corresponding to features in the image, applying a perspective transformation to the segmented image to obtain a top-down transformed image, wherein the areas are retained in the transformed image; determining a position of the area relative to a current position of the robotic lawn mower from the transformed image, drawing a local map of an environment in which the robotic lawn mower is located based on the position of the area relative to the current position of the robotic lawn mower; and controlling the robotic lawn mower to navigate a lawn area using the local map.

[0008] The local map provides the robotic mower with data about areas and features in the physical environment relative to the robotic mower's location, so that the robotic mower can be controlled to move toward or away from those areas and features.

[0009] The area may include one or more areas containing grass features and one or more areas containing non-grass features. By dividing the area into grass and non-grass, the robot mower may be controlled to avoid the non-grass area and cut the grass area.

[0010] The non-grass features may include one or more of obstacles, hazards, and / or lawn area boundaries. Thus, the non-grass areas may be further classified to determine the types of non-grass features. The control of the robotic lawn mower may differ depending on its proximity to different types of non-grass features.

[0011] Further, the local map may be limited in size based on a threshold distance from the current location of the robotic lawn mower, such that the local map includes areas with a distance from the current location of the robotic lawn mower below the threshold distance. The areas within the threshold distance may include one or more areas including grass, and one or more areas including non-grass features, the non-grass features may include one or more of obstacles, hazards, and / or boundaries of lawn areas.

[0012] Applying a threshold distance to the size of the local map limits the memory requirements for storing the local map in the robot mower's memory, while maintaining maps of areas and features that are closest to the robot mower's current location. This is more efficient than saving and storing larger maps.

[0013] Rendering the local map may further include obtaining a previous iteration of the local map, rendering it using data from a previously received image, and adding one or more regions to the previous iteration of the local map, the regions being from a transformed image corresponding to the received image.

[0014] Therefore, the method is iterative, that is, each time a new image is captured from the imaging sensor of the robotic lawn mower, the local map is updated with the regions segmented from the new image. The previous iteration of the local map is also updated with the latest position of the region contained therein, based on the latest position of the robotic lawn mower. This has the benefit of keeping the local map synchronized with the most recently received images, thereby better forming real-time or near real-time control of the robotic lawn mower.

[0015] In at least one embodiment, the drawing of the local map may consider limiting the size of the local map according to a threshold distance from the current position of the robotic lawn mower, so that the local map includes an area whose distance from the current position of the robotic lawn mower is less than the threshold distance. The area within the threshold distance may include one or more areas including grass, and one or more areas including non-grass features, and the non-grass features may include one or more of obstacles, hazards and / or boundaries of lawn areas.

[0016] Drawing the local map may further include: deleting a portion of the local map corresponding to one or more regions in a previous iteration, which are greater than a threshold distance from the current position of the robotic lawn mower; and adding one or more regions from the transformed image corresponding to the received image in the previous iteration of the local map, which are less than the threshold distance from the current position of the robotic lawn mower. This deleting step may be used in combination with any other method steps described herein.

[0017] Applying a distance threshold to features or regions included in the previous iteration of the local map limits the memory requirements for storing the local map in the robot mower's memory while retaining the map of regions and features that are closest to the robot mower's current location. This is more efficient than saving and storing larger maps. Some parts of the local map may have been historically close to the robot mower, but are now farther away, exceeding the distance threshold, and these parts may be considered less important to the control of the robot mower. Therefore, removing regions or features in the local map that are more than the distance threshold away from the robot mower's relative position can free up memory to add more regions and features from more recently received images.

[0018] Drawing the local map may further include: recording the number of previous images used for drawing the previous iteration of the local map, comparing the recorded number of images with the maximum number of images; and when the recorded number of images used for drawing the previous iteration of the local map is equal to the maximum number of images: when receiving an image from the imaging sensor, deleting a portion of the previous iteration of the local map corresponding to the oldest image in the previous image; and adding one or more regions in the previous iteration of the local map, which are from the converted image corresponding to the received image, so as not to exceed the maximum number of images of the local map. These steps may be further used in combination with any other method steps described herein.

[0019] Thus, by using a distance threshold either instead or in combination, the local map can be iteratively updated to include data from only a limited number of consecutively captured images. The local map is limited by a maximum number of images, such that when the maximum number is reached, any new image and its corresponding data effectively replaces the oldest image and its corresponding data in the local map. Thus, the local map can be considered to be formed by a sliding window of captured image data, where the sliding window extends from the most recently captured image to the image in the sequence of consecutive images that is at a maximum threshold number of distances from it.

[0020] Limiting the number of images in this way again limits the memory requirements for storing local maps in the robot mower's memory, while retaining maps of areas and features that are closest to the robot mower's current location. This is more efficient than saving and storing larger maps.

[0021] Determining the position of the region relative to the current position of the robotic lawn mower from the transformed image may further include: processing the transformed image using a VSLAM algorithm to obtain the position of the region in the transformed image relative to the current position of the robotic lawn mower. This determination step may be used in combination with any other method steps described herein.

[0022] The method may further include receiving one or more additional images from an imaging sensor on the robotic mower and updating the local map based on the one or more additional images.

[0023] The local map can be generated from an initial image captured at the beginning of each session, where a session corresponds to a period of time when the robotic mower is active. The initial image provides the first input for generating the local map. The local map can then be built up using further successively captured images while the robotic mower is moving. These images can be captured periodically based on the frame rate of the imaging sensor.

[0024] The method may further include obtaining additional data related to the movement of the robotic lawn mower; and updating the position of the area in the local map relative to the current position of the robotic lawn mower based on the additional data. This step of obtaining additional data may be used in combination with any other method steps described herein.

[0025] Thus, the local map can be updated from two or more information sources, one of which is the received image and another of which is the additional data. Using multiple information sources in this way can make the drawing of the local map more reliable.

[0026] The additional data may include additional sensor data from one or more additional sensors.

[0027] The additional sensor data may further include one or more of the following data: odometry data, IMU data, and GPS data. Such data may be received from a speed sensor, an inertial measurement unit (IMU), a GPS sensor, or similar devices. Optical sensors may also be used to detect parameters such as distance, speed, and position of the robotic lawn mower, either absolutely or relative to certain areas or features.

[0028] Updating the positions of regions in the local map relative to the current position of the robotic lawn mower based on the additional data may include: for each region in the local map: obtaining a previous position of the region from the local map; and modifying the previous position of the region based on the additional sensor data to determine an updated position of the region.

[0029] The additional data may include region tracking data for one or more regions present in the received image.

[0030] The method may further include generating region tracking data for one or more regions present in the received image by: identifying one or more regions in one or more previously received images; tracking the one or more regions through the one or more previously received images and the received image to determine a tracking path for each of the one or more regions.

[0031] The method may further include inferring a tracking path for the one or more regions when the one or more regions no longer appear in a subsequently received image; and updating a position of the one or more regions in the local map relative to a current position of the robotic mower based on the inferred tracking path.

[0032] Tracking regions over successively received images provides additional data to the regions obtained through image segmentation. In particular, a region tracked over multiple images may exhibit apparent motion in the images, depending on the motion of the robotic mower. This apparent motion may be tracked using a feature tracking algorithm, resulting in an inferred tracking path. The inferred tracking path may be used to relocate one or more regions being tracked in the local map, based on parameters such as the capture time of the received images and the speed of the robotic mower. This allows the location of a region in the local map to be updated even if the region leaves the field of view of the robotic mower's imaging sensor.

[0033] The frequency of updating the location of the area in the local map relative to the current location of the robot mower based on the additional data may be higher than the frequency of updating the location of the area in the local map relative to the current location of the robot mower based on the received images.

[0034] This means that additional data (e.g., additional sensor data and / or region tracking data) can be used to update the local map more regularly than if the local map were updated continuously by receiving images. This reduces the computational burden of image processing while still allowing the robotic mower to safely navigate within the lawn area, avoiding obstacles, etc., based on the regularly updated local map. Furthermore, the use of additional data enables the local map to be updated even if the area is not within the field of view of the imaging sensor. Therefore, even if areas such as obstacles or hazards near the robotic mower are not within the field of view of the imaging sensor, the location of these areas can still be accurately updated.

[0035] Acquiring and using additional sensor data (e.g., odometry and IMU data) to update the locations of regions or features in a local map is computationally less expensive than repeatedly capturing, receiving, and processing images to update a local map. Therefore, to maximize computational efficiency and reduce the need for complex hardware, the method works best when images are updated relatively slowly and sensors are updated relatively quickly. This means that the frequency of updating the local map with images is less frequent than the frequency of updating the local map with sensor data. When updating the local map with sensor data, the local map may be initially updated with the most recently received image, and then the relative locations of regions and features in the local map may be predicted or otherwise updated based on the detection parameters of the robotic mower (e.g., speed, direction, acceleration, etc.) obtained from the additional sensors. Each time an update is made with a received image, one or more updates may be made with the additional sensors. Once a new image is received, the local map may be updated with the new image, thereby repeating the above method.

[0036] The method of the first aspect may be repeated periodically for a plurality of received images.This step of regularly updating the local map may be used in combination with any other method steps described herein.

[0037] The method may include capturing an image.

[0038] Controlling the robotic mower to navigate the lawn area using the local map may include accessing the local map; and controlling the robotic mower to navigate the lawn area according to the location of the area indicated by the local map. This step of controlling the robotic mower to navigate the lawn area may be used in combination with any other method steps described herein.

[0039] Thus, the robotic mower can be controlled to steer toward and mow grass areas, and / or avoid non-grass areas, such as borders, hazards, and obstacles. Control of the robotic mower is based on the position of these features relative to the position of the robotic mower as indicated by the local map.

[0040] Controlling the robotic lawn mower to navigate the lawn area according to the area location indicated by the local map may include: controlling one or more drive mechanisms of the robotic lawn mower to move the robotic lawn mower to navigate the lawn area.

[0041] The drive mechanism may be any suitable mechanism including one or more motors, engines, wheels, etc.

[0042] In a second aspect, the present disclosure provides a robotic lawn mower, comprising one or more actuating mechanisms, an imaging sensor, a storage module and a control module, the control module comprising a processor communicatively coupled to the imaging sensor and the storage module, wherein the processor is configured to execute the method of the first aspect described above.

[0043] The imaging sensor may be a camera.

[0044] The camera can be configured to capture wide-angle images with a field of view exceeding 100 degrees.

[0045] The camera's field of view is 130 degrees.

[0046] The robotic mower may further include one or more additional sensors communicatively coupled to the processor, the one or more additional sensors configured to provide data related to at least one of: a speed of the robotic mower; an orientation of the robotic mower; a direction of the robotic mower; a distance traveled by the robotic mower; an acceleration of the robotic mower; and a distance from the robotic mower to one or more obstacles surrounding the robotic mower.

[0047] The control module may be configured to communicate with an external server to perform the method according to the first aspect. By communicating with the external server, the robotic lawn mower itself does not require complex hardware and memory. The robotic lawn mower may communicate with the external server by any suitable communication method. For example, the robotic lawn mower may communicate wirelessly with a server over the Internet via a wireless connection.

[0048] In a third aspect, the present disclosure includes a computer-readable medium having instructions stored thereon, which, when executed by a processor, can cause the processor to perform the method of the first aspect.

[0049] In a fourth aspect, the present disclosure comprises a computer program which, when executed by a processor, causes the processor to perform the method of the first aspect.

[0050] Portions of the methods described herein may be performed by software in machine-readable form on a tangible storage medium, for example in the form of a computer program comprising computer program code means adapted to perform all of the steps of any of the methods described herein when the program is run on a computer, wherein the computer program may be embodied on a computer-readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards, etc., but do not include propagating signals. The software may be adapted for execution on a parallel processor or a serial processor so that the method steps may be performed in any suitable order or simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Embodiments of the present invention will be described with reference to the following drawings:

[0052] Figure 1 is a schematic diagram illustrating an example of a lawn area where a robotic lawn mower can work;

[0053] Figure 2 is a schematic diagram illustrating an example of a lawn area in which a robot lawn mower according to an embodiment of the present invention can operate;

[0054] Figure 3 is a schematic diagram illustrating a robotic lawn mower according to an embodiment of the present invention;

[0055] Figure 4 is a flow chart illustrating a method for updating a local map of a robotic lawn mower's environment according to one embodiment of the present invention;

[0056] Figure 5 is a schematic diagram illustrating an example of a lawn area in which a robotic lawn mower according to an embodiment of the present invention may operate, and visually represents a local map of the environment of the robotic lawn mower; and

[0057] Common reference numerals are used in the Figures to indicate similar features. Specific embodiments

[0058] The embodiments of the present invention described below relate to an apparatus and method for controlling a small autonomous vehicle, such as a robotic lawn mower.

[0059] Figure 2A schematic overview of a robotic mower 101 in accordance with one embodiment of the present invention is shown within a typical lawn area 103. The lawn area 103 includes a lawn (grass) and may include obstacles / hazards 104 contained therein. The robotic mower 101 is configured to cut the lawn (grass) within the lawn area 103 without leaving the lawn area 103 and without striking / entering any obstacles / hazards 104. When the robotic mower 101 is inactive, it is configured to charge at a charging station 102. The lawn area 103 does not need to be delineated by guide lines or boundary lines, as the robotic mower 101 is able to navigate within the lawn area 103 without these features. The lawn area 103 is defined by a boundary 103a. The boundary 103a refers to the actual boundary of the lawn.

[0060] Figure 3 A schematic diagram of a robotic lawn mower 101 according to the present invention is shown. The robotic lawn mower 101 includes a body 1011, wheels 1012, a power source (not shown), a mowing implement (not shown), and a control module 1013. The robotic lawn mower is preferably a differential drive robot, which allows the robot to have a zero turning radius. This allows the robotic lawn mower to turn in any situation and helps prevent the robotic lawn mower 101 from getting stuck. The control module 1013 includes a computer, or a processor configured to perform calculations and execute software, and a memory. The control module 1013 includes or otherwise communicates with one or more sensors. One or more sensors are configured to provide information about mower parameters to the control module 1013, such as the absolute / relative position of the robot mower 101; the absolute / relative position of obstacles and / or lawn area boundaries; the absolute / relative position of the charging station 102; the absolute / relative position of the lawn in the lawn area 103 that was recently cut or trimmed by the robot mower 101; humidity; temperature; robot mower kinematics; robot mower power, etc. These sensors may include inertial measurement units (IMUs), speed sensors, acceleration sensors, GPS sensors, ultrasonic sensors, radars, infrared sensors, laser sensors, lidars, etc. It is understood that the control module may include any of these types of sensors or may be communicatively coupled to any of these types of sensors. The control module 1013 of the robot mower 101 may be further configured to include and / or communicate with an imaging sensor (such as a camera device). The imaging sensor is configured to capture images of the environment in which the robot mower is located, preferably on the path that the robot mower is about to move. It is not difficult to understand that the imaging sensor has parameters such as field of view angle and maximum frame rate. It is best to use a wide-angle camera device so that the field of view includes at least part of the side view and the view of the front or upcoming path. For example, the imaging sensor can be a Sony IMX327 or IMX390, with a field of view of 130 degrees.

[0061] The imaging sensor is configured to routinely capture images corresponding to a front view of the robotic mower 101, thereby capturing a stream of images illustrating the environment in front of the robotic mower 101 or in a path in front of the robotic mower 101. It should be understood that although a front view is referred to herein, the wide angle field of view may include images such as a side view or a rear view of the robotic mower.

[0062] The control module 1013 is configured to process each captured image using a semantic segmentation algorithm to segment the captured image into regions or areas and identify the regions or areas as including grass or not including grass. Further similar processing may also be performed to determine whether there are obstacles / hazards or boundaries in the captured image. The semantic segmentation algorithm is applied to each captured image in the captured image stream via software stored on a local memory connected to the control module 1013. However, it should be understood that any one or more of the computer-implemented processes discussed herein may be performed remotely via a network, server, or similar device.

[0063] It will be self-explanatory that the semantic segmentation algorithm can be trained based on various techniques. Once trained, the semantic segmentation algorithm can classify the segmented regions of the image between grass and non-grass areas. The training can further help in classifying the image areas as "obstacles / hazards".

[0064] Applying a semantic segmentation algorithm on the captured image produces a segmented image, which is segmented into different regions or areas. These regions are then classified. Thus, a stream of continuously captured images is fed through the semantic segmentation algorithm to obtain a series of segmented images, each with classified regions.

[0065] The control module 1013 is configured to further process the segmented images to construct a local map of the environment of the robotic lawn mower 101. The local map is generated by a visual simultaneous localization and mapping (VSLAM) algorithm, such as the Gmapping algorithm. In practice, a series of segmented images are stitched together after perspective transformation to draw a top-down map. Figure 4 Describes the process of drawing a local map.

[0066] Figure 4 A flow chart 400 is shown that details a method for local mapping. In a first step 401, an image is received from an imaging sensor. The image corresponds to a single frame in a stream of images routinely captured by the imaging sensor.

[0067] In a second step 402, the semantic segmentation algorithm process explained above is applied to the received image to generate a segmented image. The image is then classified to determine the features contained in each segment of the segmented image.

[0068] In the third step 403, a perspective transformation is performed on the segmented image, thereby transforming the segmented image into a transformed image showing a top view of the environment shown in the segmented image.

[0069] In a fourth step 404, one or more VSLAM algorithms are applied to the transformed image to obtain and track the position and orientation of classified features in the transformed image, such as grassy areas, grass-free areas, and obstacles / hazards.

[0070] In a fifth step 405, the local map in memory is updated with the position and orientation of the features identified in the fourth step 404. The transformed image obtained from steps 403 and 404 is effectively stitched into the existing local map. The existing local map includes the previously transformed images, which together make up the local map. If there is no existing local map, for example, no local map is currently stored in the local memory, the transformed image is used to create a first part of the local map.

[0071] The local map includes classified features derived from multiple continuously captured images. In one example, the generation and maintenance of the local map causes the robot mower 101 to always appear at the origin or center of the local map. In this example, the position and direction of the features are regularly or continuously updated in the local map relative to the current position of the robot mower 101. In this example, further, sensors can be used to detect and track the motion trajectory and / or kinematics of the robot mower 101, so that parameters such as the speed, distance and direction of the robot mower 101 are taken into account in the update of the local map. These sensors and the data they generate can also be used to help update the position of features in the local map relative to the updated position of the robot mower 101. The use of sensors other than imaging sensors to update the local map will be described in detail below.

[0072] This method 400 is repeated for more images in a continuously captured image stream received from the imaging sensor. In one example, each image is acquired at a frame rate of 3 Hz, but it will be appreciated that any suitable frame rate may be used.

[0073] As described above, specific areas / points in the consecutive segmented images are tracked in the corresponding transformed images to determine the orientation and position of environmental features (such as lawn, no lawn, and obstacles / hazards) relative to the robotic mower 101. In order to reduce the computational burden of making, maintaining and storing local maps in local memory, the local maps are limited to a predefined size. The local map is defined as a map that describes the environment around the robotic mower 101 in detail. Therefore, the local map can be limited to the environment within a limited distance or radius from the robotic mower 101, where the distance or radius is less than the size of the lawn area. When the robotic mower 101 moves, or when the process 400 provides a new transformed image, the local map or part of it stored in the memory can be overwritten, so that only the direct environment on the robotic mower 101 is stored in the local map. By deleting the areas in the local map that are greater than the distance threshold from the current position of the robotic mower 101, and adding newly acquired features to the local map, these features are within or less than the distance threshold from the position of the robotic mower 101, so as to maintain the local map showing only the direct environment of the robotic mower 101. One benefit of this is that large memories are no longer required.

[0074] In one embodiment, the size of the local map may be limited by the number of consecutive images stored so that the local map is generated at any given time. For example, the local map may include the N most recently converted images, where N is a number between 2 and 50. It will be appreciated that the maximum number of images used to generate the local map may be any number selected based on the memory and computing power of the robotic mower 101.

[0075] To further reduce the computational burden, the frame rate of the imaging device may be limited so that continuous images are captured at a predetermined frame rate, which is selected based on the desired mowing speed of the robotic mower 101. Likewise, the robotic mower 101 may also be configured such that the speed of the robotic mower 101 does not exceed the frame rate of the imaging sensor and the corresponding time required to run the VSLAM algorithm on each frame of the image stream to ensure that the local map is complete and does not include blank spots or map areas with no information. By maintaining a low robotic mower speed, there is no need for expensive imaging and processing equipment, which can reduce costs. For example, the speed of the robotic mower may be between 0.25 and 0.5 m / s.

[0076] Alternatively, the frame rate may be adjusted dynamically to compensate for changes in the speed of the robotic mower 101 .

[0077] Figure 5 Provides a visual representation of the local map. Figure 5 Shows the Figure 2 A similar schematic diagram. Figure 5In FIG. 1 , a robotic lawn mower 101 is located on a lawn area 103 . The local map is visually represented by a map area 110 , which is defined by a maximum distance radius 112 from the current location of the robotic lawn mower 101 .

[0078] Although the local map 110 is illustrated as a circle, the local map 110 can be any shape. In one example, the local map 110 is formed from a series of images captured by an imaging sensor, wherein the imaging sensor is positioned such that its field of view includes a wide angle view of the environment in front of the robot mower 101 relative to the speed of the robot mower 101. For example, the field of view can be 130 degrees. As the robot mower 101 moves forward and captures more images in the series of images, the local map becomes substantially rectangular as newly acquired transformed images are spliced ​​in front of previously acquired transformed images to form part of the local map 110. The wide field of view of the imaging sensor can provide images of the path that the robot mower is about to traverse, which are in an arc shape on both sides of the robot mower. This means that the robot mower can effectively perceive a cone view of the world in front of it. Although images of the environment on each side of the robot mower adjacent to the front are not captured, the wide field of view of the previously captured images ensures that when new images are received to update the local map, the local map includes grass and non-grass features, such as boundaries and hazards in the environment on both sides of the robot mower adjacent to the front. The local map 110 is continuously updated when receiving newly captured images. Figure 1 Partially converted images, the features contained in these images are no longer near the robotic lawn mower or within the distance threshold range. In this way, the size of the local map 110 will not exceed the maximum size specified by the distance threshold.

[0079] In one example, the size of the local map is 0.5 to 2 square meters, while the field of view of the imaging sensor is approximately 0.7 meters. As described above, by keeping the size of the local map small and deleting and replacing portions of the local map when the distance to the robotic mower exceeds a maximum distance threshold, there is no need to store a large map, thereby reducing the memory requirements of the robotic mower 101.

[0080] The control module 1013 of the robotic lawn mower 101 may further utilize other sensor data to supplement the above-described process of forming and maintaining the local map 110. For example, according to process 400, the local map 110 and features such as boundaries, obstacles / hazards, and lawns may be updated faster or more frequently than each iteration of capturing images, transforming images, and stitching newly transformed images into the local map. In particular, based on steps 401 to 405, the local map is updated by adding the transformed image to the local map 110. These steps of the process are limited by the computing devices and sensors used to perform these steps. For example, the imaging sensor is limited by the frame rate, and the performance of the semantic segmentation algorithm and the VSLAM algorithm in forming the local map is limited by the processing power of the control module 1013. These constraints such as the frame rate are maintained at a nominally low / non-intensive value so that the process 400 can be performed on less complex and smaller computers and computer processors. For example, the frame rate is maintained at one frame per second. This means that the semantic segmentation and process 400 will only be repeated once per second when a new image is received. This allows the semantic segmentation and processing 400 to be performed on low-cost and small computer processors (such as mobile phone processors, etc.).

[0081] Because process 400 is repeated at a low rate, the robotic mower may move between the capture and processing of consecutive images. This means that the robotic mower 101 may move between local map updates, which means that the local map becomes out of date in terms of the relative positions of features such as boundaries and obstacles. To address this issue, additional sensor data can be used to predict or update the local map and the positions of features such as boundaries, obstacles / hazards, and lawn relative to the robotic mower 101, starting from the most recent available version of the local map 110. In particular, odometer data from an odometer sensor or velocity / acceleration data from an IMU (inertial measurement unit) can be used to calculate the distance traveled by the robotic mower 101 since the last update of the local map 110. The positions of existing features in the local map 110 can then be recalculated based on the measured distance traveled by the robotic mower. Upon receiving an update to the local map 110, the positions of the features are adjusted according to process 400.

[0082] Additional sensor data may also be used to more accurately calculate the location of features such as boundaries, obstacles / hazards, and lawn relative to the robotic mower 101. For example, an IMU may be used in conjunction with odometry data to provide sensor redundancy. Use of IMU data may help eliminate drift that may occur in the odometry data, which may be caused by wheel rotation or the like. IMU data may also be used in conjunction with odometry data to determine the location of the robotic mower relative to features contained in the local map between local map updates.

[0083] While the above examples discuss the use of IMU and odometry data, the present invention is not limited to any particular type of additional sensor. As will be appreciated, other types of sensors such as infrared sensors, ultrasonic sensors, GPS systems, or similar devices may be used in conjunction with imaging sensors to develop and maintain local maps 110 between map updates of process 400.

[0084] Compared to the semantic segmentation of process 400 itself, the additional sensors (such as IMU) used to predict the position of the robot mower between updates of the local map by process 400 are relatively low cost and computationally efficient. By preferentially using these additional sensors to maintain the local map, updating the relative proximity of features in the local map to the current position of the robot mower, rather than using semantic segmentation and repeating process 400 more frequently, the local map can be maintained at a lower computational cost. This means that the control of the robot mower is more efficient and uses less complex hardware. Additional sensors can be used to maintain the local map in real time between updates of the local map by process 400.

[0085] In one embodiment, further processing is performed on portions of the local map 110 that no longer correspond to the field of view of the imaging sensor. In particular, the control module 1013 may perform further processing on areas of the local map 1013 that correspond to the environment near the left and right sides of the robot mower 101 relative to the direction in which the robot mower 101 moves forward. The left and right sides of the robot mower 101 may be outside the field of view of the imaging sensor. However, due to the wide field of view of the imaging sensor, when the robot mower moves forward, the environment near the left and right sides of the robot mower 101 will still be mapped into the local map based on one or more previously captured images. Therefore, it is important to maintain the local map 110, including the locations of obstacles determined from the field of view, so that the robot mower 101 knows the relative locations of features on the map (e.g., obstacles on the left and right sides of the robot mower 101) so that navigation can be performed by the robot mower 101. In other words, the local map retains the positions of potential obstacles relative to the robotic mower 101 so that the vehicle can use its motion control mechanism (e.g. based on odometry) to determine its position relative to each such feature on the map and navigate past the or each feature without colliding with any such feature.

[0086] In one example, an additional processing step includes a more detailed modeling of the location of objects in respective areas on the left and right sides of the robotic mower 101 by the control module 1013. The additional processing step includes iteratively running a proportional-integral-derivative (PID) loop using the Bresenham algorithm on features that appear in the successive segmented images. In this way, the control function generates a tracking curve for a feature when it is in the field of view of the imaging sensor, and the tracking curve for the feature is extrapolated once the feature is no longer in the field of view of the imaging sensor. Once the object is no longer in the field of view of the imaging sensor, the tracking profile can be used to determine an estimated distance to the object. This facilitates more accurate localization of the object on the local map. This additional processing step can be viewed as providing data similar to lidar-based object distance measurements, although without the need for additional hardware. Maintaining the location of the object on the local map allows the control module 1013 to monitor boundaries, hazards, or obstacles on one side of the robotic mower 101 without expensive hardware. Therefore, the location of the object on the local map can be used in conjunction with one or more additional sensors (such as an IMU) described above, or can be used alone. During operation, the robot mower 101 uses the local map to navigate, mow, and avoid obstacles / hazards and boundaries. The local map is used to measure one or more distances from the current location of the robot mower to features such as obstacles / hazards in order to control the robot mower to avoid or bypass these features. The local map can be used in conjunction with a path planning algorithm and other sensors such as a global positioning system (GPS) sensor and / or an odometry sensor.

[0087] When the robotic lawn mower 101 is deployed in a lawn area 103, where the lawn area 103 is unknown to the robotic lawn mower 101, the robotic lawn mower 101 can navigate according to a path planning algorithm. The path planning algorithm can be used to ensure that the robotic lawn mower can safely and efficiently bypass any obstacles / hazards or boundaries in order to cut the lawn in the lawn area 103.

[0088] In the embodiment of the present invention, the control module 1013 can be implemented as any form of computing and / or electronic device. It should be understood that although the control module 1013 of the robot mower 101 is mentioned above as being responsible for the image processing method, the method of forming a local map, the method of simulating a laser radar, and the path planning method, these methods can be executed or calculated through interaction with an external server, computer, or satellite that communicates with the robot mower 101.

[0089] The robotic mower 101 may form part of a system including a charging station 102 or any other entity that may communicate with the robotic mower 101 to control the robotic mower 101 or otherwise perform one or more of the method steps described above.

[0090] The control module 1013 may include a computing and / or electronic device that may include one or more processors, which may be microprocessors, controllers, or any other suitable type of processor, for processing computer executable instructions to control the operation of the device to collect and record routing information. In some examples, such as when a system-on-chip architecture is used, the processor may include one or more fixed function blocks (also referred to as accelerators) that may implement a portion of the method in hardware (rather than in software or firmware). Platform software consisting of an operating system or any other suitable platform software may be provided on the computing-based device to execute application software on the device.

[0091] The above method can be performed using computer executable instructions, which can be provided using any computer-readable medium accessible to a computing-based device. For example, a computer-readable medium may include a computer storage medium (such as a memory) and a communication medium. Computer storage media (such as a memory) include volatile and non-volatile, removable and non-removable media, which are implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computer device. In contrast, a communication medium can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal (such as a carrier wave) or other transmission mechanism. As defined herein, a computer storage medium does not include a communication medium.

[0092] The term "computer" as used herein refers to any device with processing capability that can execute instructions. Those skilled in the art will appreciate that such processing capability is integrated into many different devices, so the term "computer" This includes personal computers, servers, mobile phones, personal digital assistants, and many other devices.

[0093] Those skilled in the art will appreciate that storage devices for storing program instructions can be distributed in a network. For example, a remote computer can store examples of the above process. A local computer or a terminal computer can access the remote computer and download some or all of the software to run the program. Alternatively, the local computer can download part of the software as needed, or execute part of the software instructions at the local terminal and execute part of the software instructions at the remote computer (or computer network). Those familiar with the art will also appreciate that all or part of the software instructions can be executed by a dedicated circuit, such as a digital signal processor, a programmable logic array, etc., by utilizing conventional techniques known to those skilled in the art.

[0094] It is to be understood that the above benefits and advantages may be related to one embodiment or multiple embodiments, and these embodiments are not limited to those embodiments that can solve any or all of the above problems, nor are they limited to those embodiments that have any or all of the above benefits and advantages.

[0095] Any reference to "an" item refers to one or more of the items. As used herein, the term "comprising" means including the identified method steps or elements, but these steps or elements do not include an exclusive list and the method or apparatus may include additional steps or elements.

[0096] The order of steps of the methods described herein is exemplary, but the steps may be performed in any suitable order, or simultaneously where appropriate. In addition, steps may be added or replaced, or individual steps may be deleted from any method without departing from the scope of the subject matter described herein. Aspects of any of the above embodiments may be combined with aspects of any of the other embodiments described to form further embodiments without losing the effects sought.

[0097] It will be appreciated that the above description of the preferred embodiments is given by way of example only and that various modifications may be made by those skilled in the art. Although various embodiments have been described to a certain extent above, or with reference to one or more individual embodiments, those skilled in the art may make numerous modifications to the disclosed embodiments without departing from the spirit or scope of the invention.

Claims

1. A computer-implemented method for controlling a robotic lawn mower, the method comprising: receiving an image from an imaging sensor onboard a robotic lawn mower, the image including a ground surface in a path that the robotic lawn mower is about to traverse; Applying a semantic segmentation algorithm to generate a segmented image from the received image, the segmented image including regions corresponding to features in the image; Performing perspective transformation on the segmented image to obtain a top-down transformed image, wherein the region is retained in the transformed image; determining, based on the transformed image, the position of each of the regions relative to the current position of the robotic lawn mower; plotting a local map of the environment of the robotic lawn mower based on the position of each of the regions relative to the current position of the robotic lawn mower; as well as The local map is used to control the robotic lawn mower to navigate within a lawn area.

2. The method of claim 1, wherein the area includes one or more areas containing non-grass features, wherein: The non-grass features include one or more of obstacles, hazards, and / or turf area boundaries.

3. The method according to claim 1, wherein the size of the local map is limited according to a threshold distance from the current position of the robotic mower, so that the distance between the area included in the local map and the current position of the robotic mower is lower than the threshold distance.

4. The method according to claim 1, wherein drawing the local map comprises: A previous iteration of obtaining the local map is drawn using previously received image data; as well as One or more regions in the transformed image corresponding to the received image are added to a previous iteration of the local map.

5. The method according to claim 4, wherein drawing the local map comprises: deleting a portion of a previous iteration of the local map corresponding to one or more regions that are located at a distance greater than a threshold distance from a current position of the robotic lawn mower; as well as One or more regions in the transformed image corresponding to the received image are added to a previous iteration of the local map, the one or more regions being located less than a threshold distance from a current position of the robotic mower.

6. The method according to claim 4, wherein drawing the local map comprises: Recording the number of previous images used in a previous iteration of drawing the local map; Compare the number of recorded images with the maximum number of images; as well as When the number of recorded images used in the previous iteration for drawing the local map is equal to the maximum number of images: Upon receiving an image from the imaging sensor, deleting a portion of the local map from a previous iteration that corresponds to the oldest of the previous images; as well as One or more regions in the transformed image corresponding to the received image are added to the previous iteration of the local map so that the maximum number of images of the local map is not exceeded.

7. The method according to claim 1, wherein determining the position of each of the regions relative to the current position of the robotic lawn mower based on the transformed image comprises: The transformed image is processed using a VSLAM algorithm to obtain the position of each of the regions in the transformed image relative to the current position of the robotic lawn mower.

8. The method of claim 1 further comprising: acquiring additional data related to the motion of the robotic lawn mower; as well as Based on the additional data, the position of each area in the local map relative to the current position of the robotic lawn mower is updated, wherein the additional data includes sensor data from one or more additional sensors, and the sensor data includes one or more of the following: mileage data, IMU data and GPS data.

9. The method according to claim 8, wherein updating the position of each area in the local map relative to the current position of the robotic lawn mower according to the additional data comprises: For each region in the local map: obtaining a previous position of the area from the local map; as well as A previous position of the region is modified based on the additional sensor data to determine an updated position of the region.

10. The method of claim 8, wherein the additional data comprises region tracking data for one or more regions in the received image.

11. The method according to claim 10, further comprising: The region tracking data is generated for one or more regions present in the received image by: identifying one or more regions in one or more previously received images; One or more regions are tracked using one or more previously received images and the received image to determine a tracking path for each of the one or more regions.

12. The method of claim 11 further comprising: inferring a tracking path of the one or more regions when the one or more regions no longer appear in a subsequently received image; as well as Based on the inferred tracking path, the position of one or more regions in the local map relative to the current position of the robotic lawn mower is updated.

13. The method of claim 8, wherein a frequency of updating the position of an area in a local map relative to a current position of the robotic mower based on the additional data is higher than a frequency of updating the position of an area in a local map relative to a current position of the robotic mower based on a received image.

14. The method of claim 1, wherein controlling the robotic lawn mower to navigate within a lawn area using the local map comprises: accessing the local map; as well as The robot lawn mower is controlled to navigate within the lawn area according to the area position indicated by the local map.

15. The method according to claim 14, wherein controlling the robotic lawn mower to navigate within the lawn area according to the area location indicated by the local map comprises: One or more drive mechanisms of the robotic lawn mower are controlled to move the robotic lawn mower to navigate within a lawn area.

16. Robotic lawn mower, including: one or more drive mechanisms; Imaging sensors; Storage module; as well as A control module comprising a processor communicatively coupled to one or more of the drive mechanisms, the imaging sensor, and the storage module; the processor being configured to execute the method of claim 1.

17. The robotic lawn mower of claim 16, wherein the imaging sensor is a camera configured to capture wide-angle images with a field of view exceeding 100 degrees.

18. The robotic lawn mower of claim 16 further comprising: One or more additional sensors communicatively coupled to the processor, the one or more additional sensors configured to provide sensor data related to at least one of: the speed of the robotic lawn mower; the orientation of the robotic lawn mower; the orientation of the robotic lawn mower; the distance traveled by the robotic lawn mower; the acceleration of the robotic lawn mower; and The distance between the robotic lawn mower and one or more surrounding obstacles.

19. The robotic lawn mower of claim 16, wherein the control module is configured to communicate with an external server.

20. A computer readable medium having instructions stored thereon, which when executed by a processor, cause the processor to perform the method of claim 1.

Images