Local area mapping for robotic lawnmowers

Abstract

A robotic lawnmower and a method of controlling a robotic lawnmower based on generating a local map. The method comprises receiving an image from an imaging sensor on the robotic lawnmower, the image comprising ground in a path to be reached; applying a semantic segmentation algorithm to generate a segmented image from the received image, the segmented image comprising regions corresponding to features in the image; applying a perspective transform to the segmented image to obtain an overhead transformed image; determining a position of the regions relative to a current position of the robotic lawnmower from the transformed image; drawing a local map of an environment of the robotic lawnmower based on the position of the regions relative to the current position of the robotic lawnmower; and controlling the robotic lawnmower to navigate the lawn area using the local map.

Description

Technical Field

[0001] This application relates to an apparatus, system, and method for autonomously controlling a robotic lawnmower. Background Technology

[0002] A robot, or robotic lawnmower, is an autonomous robot configured to mow lawns or fields. With robotic lawnmowers, humans no longer need to mow the grass themselves, which can be a mundane and arduous task.

[0003] Figure 1 This shows an example of a lawn area where a robotic lawnmower can operate. The robotic lawnmower 1 mows within a lawn area 3 defined by boundary lines or guide lines 4. The boundary lines or guide lines 4 need to be actually installed before the robotic lawnmower can be started. 1. After detecting the boundary line 4, the robotic lawnmower 1 will typically change direction randomly to avoid the boundary line.

[0004] Robotic lawnmowers may include multiple sensors for detecting obstacles. These sensors could be laser- or radar-based. Similarly, robotic lawnmowers may rely on satellite communication to navigate within the lawn area and may include radio sensors for navigating to charging stations. Figure 1 In the middle, charging station 2 is located on the periphery of lawn area 3.

[0005] However, a balance needs to be struck between the complexity of robotic lawnmowers and their relative capabilities. Robotic lawnmowers that incorporate different types of sensors or more complex systems are generally more expensive. Therefore, there is a need to provide a robotic lawnmower that works efficiently without requiring overly complex systems and sensors. Summary of the Invention

[0006] This summary introduces some concepts in a simplified form, which will be further elaborated in the detailed descriptions below.

[0007] In a first aspect, this disclosure provides a computer-implemented method for controlling a robotic lawnmower, the method comprising: receiving an image from an imaging sensor located on the robotic lawnmower, the image including ground regions along the path of the robotic lawnmower; applying a semantic segmentation algorithm to generate a segmented image from the received image, the segmented image including regions corresponding to features in the image; applying a perspective transformation to the segmented image to obtain a top-view transformed image, wherein regions are preserved in the transformed image; determining the position of the regions relative to the current position of the robotic lawnmower from the transformed image; drawing a local map of the environment in which the robotic lawnmower is located based on the position of the regions relative to the current position of the robotic lawnmower; and using the local map to control the robotic lawnmower to navigate the lawn area.

[0008] Local maps provide the robotic lawnmower with relevant data on areas and features in the physical environment relative to its position, allowing the robotic lawnmower to be controlled to move toward or away from these areas and features.

[0009] The area can include one or more areas containing grass features, and one or more areas containing non-grass features. By dividing the area into grass and non-grass categories, the robotic lawnmower can be controlled to avoid non-grass areas and cut grass areas.

[0010] Non-grass features can include one or more of obstacles, hazards, and / or lawn area boundaries. Therefore, non-grass areas can be further classified to determine the type of non-grass feature. The control of a robotic lawnmower can vary depending on its proximity to different types of non-grass features.

[0011] Furthermore, the local map can be limited in size based on a threshold distance from the current position of the robotic lawnmower, such that the local map includes areas whose distance from the current position of the robotic lawnmower is less than the threshold distance. The areas within the threshold distance may include one or more areas comprising grass, and one or more areas comprising non-grass features, which may include one or more of obstacles, hazards, and / or the boundaries of lawn areas.

[0012] Applying a threshold distance to the size of the local map limits the memory requirements for storing the local map in the robot lawnmower's memory, while preserving the map of the region and features closest to the robot lawnmower's current location. This is more efficient than saving and storing a larger map.

[0013] Drawing a local map may further include: obtaining the previous iteration of the local map, drawing it using data from previously received images, and adding one or more regions from the previous iteration of the local map, these regions being derived from transformed images corresponding to the received images.

[0014] Therefore, this method is iterative; each time a new image is captured from the robotic lawnmower's imaging sensor, the local map is updated with regions segmented from the new image. For each previous iteration of the local map, it is also updated based on the latest position of the regions contained within it, according to the latest position of the robotic lawnmower. The advantage of this is that it keeps the local map synchronized with the most recently received images, thus enabling better real-time or near-real-time control of the robotic lawnmower.

[0015] In at least one embodiment, the local map drawing may consider limiting the size of the local map based on a threshold distance from the current position of the robotic lawnmower, such that the local map includes areas whose distance from the current position of the robotic lawnmower is less than the threshold distance. The areas within the threshold distance may include one or more areas comprising grass, and one or more areas comprising non-grass features, which may include one or more of obstacles, hazards, and / or the boundaries of lawn areas.

[0016] Drawing a local map may further include: deleting portions of the local map corresponding to one or more regions in a previous iteration, these regions being located at a distance greater than a threshold distance from the current position of the robotic lawnmower; and adding one or more regions from a transformed image corresponding to the received image in a previous iteration of the local map, these regions being located at a distance less than a threshold distance from the current position of the robotic lawnmower. This deletion step may be used in conjunction with any other method steps described herein.

[0017] Applying distance thresholds to features or regions included in the previous iteration of the local map limits the memory requirements for storing the local map in the robot mower's memory, while retaining the map of regions and features closest to the robot mower's current location. This is more efficient than saving and storing a larger map. Some parts of the local map may have historically been close to the robot mower but are now farther away, exceeding the distance threshold; these parts may be considered less important for the robot mower's control. Therefore, removing regions or features in the local map that are more than the distance threshold from the robot mower's relative position frees up memory to add more regions and features from recently received images.

[0018] Drawing a local map may further include: recording the number of previous images used in the previous iteration for drawing the local map, comparing the recorded number of images with a maximum number of images; and when the recorded number of images used in the previous iteration for drawing the local map equals the maximum number of images: upon receiving an image from the imaging sensor, deleting portions of the local map corresponding to the oldest image in the previous iteration; and adding one or more regions in the previous iteration of the local map, these regions being derived from transformed images corresponding to the received images, such that they do not exceed the maximum number of images for the local map. These steps may be further combined with any other method steps described herein.

[0019] Therefore, by using distance thresholds for substitution or combination, the local map can be iteratively updated to include only data from a finite number of consecutively captured images. The local map is limited by a maximum number of images, so that when the maximum number is reached, any new image and its corresponding data effectively replace the oldest image and its corresponding data in the local map. Thus, the local map can be considered to be formed by a sliding window of captured image data, where the sliding window extends from the most recently captured image to images in the consecutive image sequence that are at a maximum distance of a threshold number.

[0020] Limiting the number of images in this way further limits the memory requirements for storing local maps in the robotic lawnmower's memory, while preserving the map of the area and features closest to the robotic lawnmower's current location. This is more efficient than saving and storing larger maps.

[0021] Determining the position of a region relative to the current position of the robotic lawnmower from the transformed image may further include: processing the transformed image using a VSLAM algorithm to obtain the position of the region in the transformed image relative to the current position of the robotic lawnmower. This determination step may be used in conjunction with any other method steps described herein.

[0022] The method may further include: receiving one or more additional images from an imaging sensor on a robotic lawnmower, and updating a local map based on the one or more additional images.

[0023] Local maps can be generated from initial images captured at the start of each session, where a session corresponds to a period of time during which the robotic lawnmower is active. The initial images provide the first input for generating the local map. Then, as the robotic lawnmower moves, further continuously captured images can be used to build the local map. These images can be captured periodically according to the frame rate of the imaging sensor.

[0024] The method may further include acquiring additional data related to the movement of the robotic lawnmower; and updating the position of a region in a local map relative to the current position of the robotic lawnmower based on the additional data. This step of acquiring the additional data may be used in conjunction with any other method steps described herein.

[0025] Therefore, local maps can be updated using two or more information sources, one of which is the received image, and the other is supplementary data. Using multiple information sources in this way makes the local map creation more reliable.

[0026] Additional data may include additional sensor data from one or more additional sensors.

[0027] Additional sensor data may further include one or more of the following: odometer data, IMU data, and GPS data. This data can be received from a speed sensor, inertial measurement unit (IMU), GPS sensor, or similar device. Optical sensors can also be used to detect parameters such as distance, speed, and position of the robotic lawnmower, either absolutely or relative to certain areas or features.

[0028] Updating the position of each region in the local map relative to the current position of the robotic lawnmower based on additional data may include: for each region in the local map: obtaining the previous position of the region from the local map; and modifying the previous position of the region based on additional sensor data to determine the updated position of the region.

[0029] Additional data may include region tracking data for one or more areas present in the received image.

[0030] The method may further include generating region tracking data for one or more regions present in the received image by: identifying one or more regions in one or more previously received images; and tracking one or more regions using one or more previously received images and the received image to determine a tracking path for each of the one or more regions.

[0031] The method may further include inferring a tracking path for one or more regions when one or more regions no longer appear in subsequently received images; and updating the position of one or more regions in a local map relative to the current position of the robotic lawnmower based on the inferred tracking path.

[0032] Region tracking on continuously received images can provide additional data for regions obtained through image segmentation. In particular, regions tracked through multiple images may exhibit significant motion within the images, depending on the robot lawnmower's movement. Feature tracking algorithms can be used to track this apparent motion, thereby forming an inferred tracking path. Based on parameters such as the capture time of the received images and the robot lawnmower's speed, the inferred tracking path can be used to reposition one or more tracked regions in the local map. This allows for updating the region's position in the local map even if it leaves the robot lawnmower's imaging sensor's field of view.

[0033] The frequency of updating the area position in the local map relative to the current position of the robot lawnmower based on additional data can be higher than the frequency of updating the area position in the local map relative to the current position of the robot lawnmower based on received images.

[0034] This means that supplementary data (such as supplementary sensor data and / or area tracking data) can be used more regularly to update the local map compared to continuously receiving images to update the local map. This reduces the computational burden of image processing while still allowing the robotic lawnmower to navigate safely within the lawn area, avoiding obstacles, etc., based on regularly updated local maps. Furthermore, the local map can be updated using supplementary data even if the area is outside the imaging sensor's field of view. Therefore, the positions of obstacles or hazards near the robotic lawnmower can still be accurately updated even if they are outside the imaging sensor's field of view.

[0035] Compared to the process of repeatedly capturing, receiving, and processing images to update a local map, acquiring and using additional sensor data (such as odometry and IMU data) to update the location of regions or features in a local map involves less computation. Therefore, to maximize computational efficiency and minimize the need for complex hardware, this method ideally operates with relatively slow image updates and relatively fast sensor updates. This means that the local map is updated using images less frequently than it is updated using sensor data. When updating the local map using sensor data, the most recently received image can be used initially, and then the relative positions of regions and features in the local map are predicted or otherwise updated based on detection parameters of the robotic lawnmower (such as speed, orientation, acceleration, etc.) obtained from additional sensors. Each time an update is performed using a received image, one or more updates may be performed using additional sensors. Once a new image is received, the local map can be updated using the new image, thus repeating the above method.

[0036] The first approach can be repeated periodically for multiple received images. This step of periodically updating the local map can be used in conjunction with any other methodological steps described herein.

[0037] This method may include capturing images.

[0038] Using a local map to control a robotic lawnmower to navigate a lawn area may include accessing a local map and controlling the robotic lawnmower to navigate the lawn area based on the area location indicated by the local map. This step of controlling the robotic lawnmower to navigate the lawn area may be used in conjunction with any other method steps described herein.

[0039] Therefore, the robotic lawnmower can be controlled to turn towards grassy areas and perform / mowing, and / or avoid non-grassy areas, such as boundaries, hazards, and obstacles. The control of the robotic lawnmower is based on the position of these features relative to the location of the robotic lawnmower indicated by the local map.

[0040] Controlling a robotic lawnmower to navigate a lawn area based on the location indicated by a local map may include: controlling one or more drive mechanisms of the robotic lawnmower to move the robotic lawnmower to navigate the lawn area.

[0041] The drive mechanism can be any suitable mechanism, including one or more motors, engines, wheels, etc.

[0042] In a second aspect, this disclosure provides a robotic lawnmower including one or more actuation mechanisms, an imaging sensor, a storage module, and a control module, the control module including a processor communicatively coupled to the imaging sensor and the storage module, wherein the processor is configured to perform the method described in the first aspect.

[0043] The imaging sensor can be a camera.

[0044] The camera can be configured to capture wide-angle images with a field of view exceeding 100 degrees.

[0045] The camera's field of view is 130 degrees.

[0046] The robotic lawnmower may further include one or more additional sensors communicatively coupled to a processor, the one or more additional sensors being configured to provide data relating to at least one of the following: the speed of the robotic lawnmower; the orientation of the robotic lawnmower; the direction of the robotic lawnmower; the distance traveled by the robotic lawnmower; the acceleration of the robotic lawnmower; and the distance of the robotic lawnmower to one or more obstacles in the vicinity of the robotic lawnmower.

[0047] The control module can be configured to communicate with an external server to perform the method described according to the first aspect. By communicating with an external server, the robotic lawnmower itself does not require complex hardware and memory. The robotic lawnmower can communicate with the external server using any suitable communication method. For example, the robotic lawnmower can wirelessly communicate with the server over the Internet via a wireless connection.

[0048] In a third aspect, this disclosure includes a computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method of the first aspect.

[0049] In a fourth aspect, this disclosure includes a computer program that, when executed by a processor, causes the processor to perform the method of the first aspect.

[0050] Parts of the methods described herein can be executed by software in a machine-readable form on a tangible storage medium, for example, in the form of a computer program comprising computer program code means, which, when run on a computer, is adapted to perform all the steps of any of the methods described herein, wherein the computer program may be embodied on a computer-readable medium. Examples of tangible (or non-transitory) storage media include disks, thumb drives, memory cards, etc., but excluding those for propagating signals. The software may be adapted to execute on a parallel or serial processor, thereby allowing the method steps to be executed in any suitable order or simultaneously. Attached Figure Description

[0051] Embodiments of the present invention will be described with reference to the following drawings:

[0052] Figure 1 This is a diagram illustrating an example of a lawn area where a robotic lawnmower can operate;

[0053] Figure 2 This is a schematic diagram illustrating an example of a lawn area where a robotic lawnmower can operate according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating a robotic lawnmower according to an embodiment of the present invention;

[0055] Figure 4 It is a flowchart illustrating a method for updating a local map of a robotic lawnmower environment according to an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram illustrating an example of a lawn area in which a robotic lawnmower can operate according to an embodiment of the present invention, and visually representing a partial map of the environment of the robotic lawnmower; and

[0057] The figure uses general reference numbers to represent similar features. Specific Implementation

[0058] The embodiments of the present invention described below relate to an apparatus and method for controlling small autonomous vehicles, such as robotic lawnmowers.

[0059] Figure 2A schematic overview of a robotic lawnmower 101 within a typical lawn area 103 is shown, according to an embodiment of the present invention. The lawn area 103 includes a lawn (grassland) and may include obstacles / hazards 104 contained therein. The robotic lawnmower 101 is configured to cut the lawn (grass) within the lawn area 103 without leaving the lawn area 103 or impacting / entering any obstacle / hazard 104. When the robotic lawnmower 101 is inactive, it is configured to charge at a charging station 102. The lawn area 103 does not need to be defined by guide lines or boundary lines because the robotic lawnmower 101 is capable of navigating within the lawn area 103 without these features. The lawn area 103 is defined by a boundary 103a. Boundary 103a refers to the actual boundary of the lawn.

[0060] Figure 3 A schematic diagram of a robotic lawnmower 101 according to the present invention is shown. The robotic lawnmower 101 includes a body 1011, wheels 1012, a power source (not shown), mowing instruments (not shown), and a control module 1013. The robotic lawnmower is preferably a differential drive robot, which results in a zero turning radius. This allows the robotic lawnmower to turn in any situation and helps prevent the robotic lawnmower 101 from getting stuck. The control module 1013 includes a computer, or a processor configured to perform calculations and execute software, and a memory. The control module 1013 includes or otherwise communicates with one or more sensors. One or more sensors are configured to provide the control module 1013 with information about lawnmower parameters, such as the absolute / relative position of the robotic lawnmower 101; the absolute / relative position of obstacles and / or lawn area boundaries; the absolute / relative position of the charging station 102; the absolute / relative position of the lawn within the lawn area 103 that was recently cut or mowed by the robotic lawnmower 101; humidity; temperature; robotic lawnmower kinematics; robotic lawnmower power, etc. These sensors may include inertial measurement units (IMUs), velocity sensors, accelerometers, GPS sensors, ultrasonic sensors, radar, infrared sensors, laser sensors, lidar, etc. It is understood that the control module may include any of these types of sensors and may communicatively couple with any of these types of sensors. The control module 1013 of the robotic lawnmower 101 may be further configured to include and / or communicate with imaging sensors (such as camera devices). The imaging sensors are configured to capture images of the environment in which the robotic lawnmower is located, preferably along the path the robotic lawnmower is about to move. It is readily understood that the imaging sensors have parameters such as field of view and maximum frame rate. It is best to use a wide-angle camera so that the field of view includes at least a partial side view as well as a view of the path in front or ahead. For example, the imaging sensor could be a Sony IMX327 or IMX390 with a field of view of 130 degrees.

[0061] The imaging sensor is configured to routinely capture images corresponding to the front view of the robotic lawnmower 101, thereby capturing an image stream illustrating the environment in front of or along the path in front of the robotic lawnmower 101. It should be understood that while this refers to the front view, the wide field of view may include images such as side or rear views of the robotic lawnmower.

[0062] Control module 1013 is configured to process each captured image using a semantic segmentation algorithm, segmenting the captured image into regions or areas, and identifying these regions or areas as including or excluding grass. Further similar processing may also be performed to determine the presence of obstacles / hazards or boundaries in the captured image. The semantic segmentation algorithm is applied to each captured image in the captured image stream via software stored in local memory connected to control module 1013. However, it should be understood that any one or more computer implementations discussed herein can be executed remotely via a network, server, or similar device.

[0063] It goes without saying that semantic segmentation algorithms can be trained using a variety of techniques. After training, a semantic segmentation algorithm can classify segmented regions of an image between grassy and non-grassy areas. Training can further help classify image regions as "obstacles / hazards".

[0064] Applying semantic segmentation algorithms to captured images produces segmented images, which are divided into different regions or areas. These regions are then classified. Therefore, by feeding a continuously captured image stream using semantic segmentation algorithms, a series of segmented images is obtained, each containing a classified region.

[0065] Control module 1013 is configured to further process the segmented images to construct a local map of the environment of the robotic lawnmower 101. The local map is generated by a Visual Simultaneous Localization and Mapping (VSLAM) algorithm, such as Gmapping. In practice, a series of segmented images are stitched together after perspective transformation to create a top-down map. The following will refer to... Figure 4 This section introduces the process of creating a local map.

[0066] Figure 4 A flowchart 400 detailing the method for creating a local map is shown. Step 401: An image is received from the imaging sensor. This image corresponds to a single frame in the image stream routinely captured by the imaging sensor.

[0067] In step 402, the semantic segmentation algorithm process explained above is applied to the received image to generate a segmented image. The image is then classified to determine the features contained in each segment of the segmented image.

[0068] In step 303, a perspective transformation is performed on the segmented image to transform it into a transformed image that displays a top-down view of the environment shown in the segmented image.

[0069] In step 404, one or more VSLAM algorithms are applied to the transformed image to obtain and track the location and orientation of classification features in the transformed image. These features include grassy areas, grassless areas, and obstacles / hazards, etc.

[0070] In the fifth step 405, the local map in memory is updated using the location and orientation of the features identified in the fourth step 404. The transformed images obtained from steps 403 and 404 are effectively stitched into the existing local map. The existing local map includes the previously transformed images, which together constitute the local map. If there is no existing local map, for example, if no local map is currently stored in local memory, the transformed images are used to create the first part of the local map.

[0071] The local map comprises classification features derived from multiple consecutively captured images. In one example, the generation and maintenance of the local map ensures that the robotic lawnmower 101 always appears at the origin or center of the local map. In this example, the position and orientation of features in the local map are updated periodically or continuously relative to the current position of the robotic lawnmower 101. Further, in this example, sensors can be used to detect and track the motion trajectory and / or kinematics of the robotic lawnmower 101, thereby incorporating parameters such as the speed, distance, and orientation of the robotic lawnmower 101 into the updates of the local map. These sensors and the data they generate can also be used to help update the position of features in the local map relative to the updated position of the robotic lawnmower 101. The following section details the use of sensors other than imaging sensors to update the local map.

[0072] This method 400 is repeated for more images in the continuously captured image stream received from the imaging sensor. In one example, each image is acquired at a frame rate of 3 Hz, but it is understood that any suitable frame rate can be used.

[0073] As described above, specific regions / points in consecutively segmented images are tracked in the corresponding transformed images to determine the orientation and position of environmental features (such as lawn, no lawn, and obstacles / hazards) relative to the robotic lawnmower 101. To reduce the computational burden of creating, maintaining, and storing local maps in local memory, the local maps are limited to a predefined size. A local map is defined as a map that details the environment surrounding the robotic lawnmower 101. Therefore, the local map can be limited to the environment within a finite distance or radius from the robotic lawnmower 101, where this distance or radius is smaller than the size of the lawn area. When the robotic lawnmower 101 moves, or when process 400 provides a new transformed image, the local map stored in memory, or a portion thereof, can be overwritten, so that only the immediate environment on the robotic lawnmower 101 is stored in the local map. By deleting regions in the local map that are more than a distance threshold from the current position of the robotic lawnmower 101, and adding newly acquired features to the local map that are within or less than the distance threshold from the position of the robotic lawnmower 101, the local map is kept to show only the immediate environment of the robotic lawnmower 101. One advantage of doing this is that large storage devices are no longer needed.

[0074] In one embodiment, the size of the local map may be limited by the number of consecutive images stored so that the local map can be generated at any given time. For example, the local map may include N most recently transformed images, where N is a number between 2 and 50. It is understood that the maximum number of images used to generate the local map can be any number chosen based on the memory and computing power of the robotic lawnmower 101.

[0075] To further reduce computational burden, the frame rate of the imaging device can be limited to capture continuous images at a predetermined frame rate, selected based on the desired mowing speed of the robotic lawnmower 101. Similarly, the robotic lawnmower 101 can also be configured such that its speed does not exceed the frame rate of the imaging sensor and the corresponding time required to run the VSLAM algorithm on each frame of the image stream, ensuring complete local maps that do not include blank spots or uninformed map areas. By keeping the robotic lawnmower speed low, this eliminates the need for expensive imaging and processing equipment, thus reducing costs. For example, the robotic lawnmower speed can be between 0.25 and 0.5 meters per second.

[0076] Alternatively, the frame rate can be dynamically adjusted to compensate for speed variations in the robotic lawnmower 101.

[0077] Figure 5 It provides an intuitive representation of the local map. Figure 5 Showing with Figure 2 A similar diagram. In Figure 5In the image, the robotic lawnmower 101 is located on the lawn area 103. The local map is visually represented by map area 110, which is defined by the maximum distance radius 112 from the current position of the robotic lawnmower 101.

[0078] Although the local map 110 is illustrated as a circle, it can be of any shape. In one example, the local map 110 is formed from a series of images captured by an imaging sensor, the position of which allows its field of view to include a wide-angle view of the environment in front of the robotic lawnmower 101, relative to the speed of the robotic lawnmower 101. For example, the field of view could be 130 degrees. As the robotic lawnmower 101 moves forward and captures more images in the series, the local map essentially becomes rectangular because newly acquired transformed images are stitched in front of previously acquired transformed images to form part of the local map 110. The wide field of view of the imaging sensor provides images of the path the robotic lawnmower is about to traverse, these images being arced on both sides of the robotic lawnmower. This means the robotic lawnmower can effectively perceive a cone-shaped view of the world in front of it. Although no image of the environment on every side adjacent to the front of the robotic lawnmower is captured, the wide field of view of the previously captured images ensures that when new images are received to update the local map, the local map includes grass and non-grass features, such as boundaries and hazards in the environment on both sides adjacent to the front of the robotic lawnmower. The local map 110 is continuously updated as it receives newly captured images. Simultaneously, elements constituting the local map are deleted from the local map 110. Figure 1 Partially transformed images are used, containing features that are no longer near the robotic lawnmower or within a distance threshold. This ensures that the size of the local map 110 does not exceed the maximum size specified by the distance threshold.

[0079] In one example, the size of the local map is 0.5 to 2 square meters, while the field of view of the imaging sensor is approximately 0.7 meters. As described above, by keeping the local map small in size, and deleting and replacing portions of the local map when the distance to the robotic lawnmower exceeds a maximum distance threshold, there is no need to store a large map, thereby reducing the memory requirements of the robotic lawnmower 101.

[0080] The control module 1013 of the robotic lawnmower 101 can further utilize other sensor data to supplement the aforementioned process of forming and maintaining the local map 110. For example, according to process 400, the update rate or frequency of the local map 110 and features such as boundaries, obstacles / hazards, and lawn relative to the position of the robotic lawnmower 101 may be faster or more frequent than each iteration of capturing images, transforming images, and stitching the newly transformed images into the local map. In particular, the local map is updated based on steps 401 to 405 by adding the transformed images to the local map 110. These steps of the process are limited by the computing devices and sensors used to perform these steps. For example, the imaging sensor is limited by the frame rate, and the performance of the semantic segmentation algorithm and the VSLAM algorithm in forming the local map is limited by the processing power of the control module 1013. These constraints, such as the frame rate, are kept at a nominally low / non-dense value so that process 400 can be executed on a less complex and smaller computer and computer processor. For example, the frame rate is kept at one frame per second. This means that semantic segmentation and process 400 are only repeated once per second when a new image is received. This allows semantic segmentation and processing 400 to be performed on low-cost and small computer processors (such as mobile phone processors).

[0081] Because process 400 repeats at a low rate, the robotic lawnmower may move between capturing and processing consecutive images. This means that the robotic lawnmower 101 may move between local map updates, causing the local map to become outdated in terms of the relative positions of features such as boundaries and obstacles. To address this, additional sensor data can be used to predict or update the local map and the positions of features such as boundaries, obstacles / hazards, and lawn relative to the robotic lawnmower 101, starting from the latest available version of the local map 110. In particular, the distance traveled by the robotic lawnmower 101 since the most recent update of the local map 110 can be calculated using odometry data from odometry sensors or velocity / acceleration data from an IMU (Inertial Measurement Unit). The positions of features present in the local map 110 can then be recalculated based on the measured distance traveled by the robotic lawnmower. Upon receiving an update to the local map 110, the positions of the features are adjusted according to process 400.

[0082] Additional sensor data can be used to more accurately calculate the positions of features such as boundaries, obstacles / hazards, and lawns relative to the robotic lawnmower 101. For example, IMU data can be combined with odometry data to provide sensor redundancy. Using IMU data helps eliminate drift in odometry data, which may be caused by wheel rotation or similar reasons. IMU data can also be combined with odometry data to determine the position of the robotic lawnmower relative to features contained in the local map between local map updates.

[0083] While the examples above discussed the use of IMU and odometer data, the invention is not limited to any particular type of additional sensor. As will be understood, other types of sensors, such as infrared sensors, ultrasonic sensors, GPS systems, or similar devices, can be used in conjunction with imaging sensors to develop and maintain local maps 110 between map updates in process 400.

[0084] Compared to semantic segmentation in process 400 itself, additional sensors (such as IMUs) used to predict the robot lawnmower's position between updates to the local map in process 400 are relatively low-cost and computationally efficient. By prioritizing the use of these additional sensors to maintain the local map, updating features in the local map to show relative proximity to the robot lawnmower's current position, rather than using semantic segmentation and repeating process 400 more frequently, the local map can be maintained at a lower computational cost. This means the robot lawnmower is more efficiently controlled and uses less complex hardware. The additional sensors can be used to maintain the local map in real time between updates in process 400.

[0085] In one embodiment, further processing is performed on portions of the local map 110 that no longer correspond to the field of view of the imaging sensor. Specifically, the control module 1013 may perform further processing on areas of the local map 1013 corresponding to the environment near the left and right sides of the robotic lawnmower 101 relative to its forward movement direction. The left and right sides of the robotic lawnmower 101 may be outside the field of view of the imaging sensor. However, due to the wide field of view of the imaging sensor, the environment near the left and right sides of the robotic lawnmower 101 is still mapped into the local map based on one or more previously captured images as the robotic lawnmower moves forward. Therefore, it is important to maintain the local map 110, including obstacle positions determined from the field of view, so that the robotic lawnmower 101 knows the relative positions of features on the map (e.g., obstacles on the left and right sides of the robotic lawnmower 101) for navigation. In other words, the local map preserves the position of potential obstacles relative to the robotic lawnmower 101, so the vehicle can use its motion control mechanisms (e.g., based on odometry) to determine its position relative to each of the features on the map and navigate past that feature or each feature without colliding with any of them.

[0086] In one example, the additional processing step includes the control module 1013 simulating the object positions in the respective regions on the left and right sides of the robotic lawnmower 101 in more detail. This additional processing step involves iteratively running a proportional-integral-derivative (PID) loop using the Bressonnum algorithm on features appearing in the continuously segmented images. Thus, when a feature is in the field of view of the imaging sensor, the control function generates a tracking curve for that feature, and once the feature is no longer in the field of view of the imaging sensor, the tracking curve is inferred. Once a feature is no longer in the field of view of the imaging sensor, the tracking profile can be used to determine the estimated distance to the feature. This facilitates more accurate localization of features on the local map. This additional processing step can be viewed as providing data similar to lidar-based feature distance measurements, although no additional hardware is required. Preserving object positions on the local map allows the control module 1013 to monitor boundaries, hazards, or obstacles on one side of the robotic lawnmower 101 without expensive hardware. Therefore, feature positions on the local map can be used in conjunction with one or more additional sensors (such as an IMU) as described above, or they can be used independently. During operation, the robotic lawnmower 101 uses a local map to navigate, mow, and avoid obstacles / hazards and boundaries. The local map is used to measure one or more distances from the robotic lawnmower's current position to features such as obstacles / hazards, in order to control the robotic lawnmower to avoid or bypass these features. The local map can be used in conjunction with path planning algorithms and other sensors such as GPS sensors and / or odometry sensors.

[0087] When the robotic lawnmower 101 is deployed in the lawn area 103, which is unknown to the robotic lawnmower 101, the robotic lawnmower 101 can navigate according to a path planning algorithm. The path planning algorithm can be used to ensure that the robotic lawnmower can safely and efficiently bypass any obstacles / hazards or boundaries in order to cut the lawn in the lawn area 103.

[0088] In the embodiments described in this invention, the control module 1013 can be implemented as any form of computing and / or electronic device. It should be understood that although the control module 1013 of the robotic lawnmower 101 is responsible for image processing methods, methods for forming local maps, methods for simulating LiDAR, and path planning methods, these methods can be executed or computed through interaction with an external server, computer, or satellite communicating with the robotic lawnmower 101.

[0089] The robotic lawnmower 101 may be part of a system that includes a charging station 102 or any other entity that may communicate with the robotic lawnmower 101 to control the robotic lawnmower 101 or otherwise perform one or more of the method steps described above.

[0090] The control module 1013 may include computing and / or electronic devices, which may include one or more processors, such as microprocessors, controllers, or any other suitable type of processor, for processing computer-executable instructions to control the operation of the device, thereby collecting and recording routing information. In some examples, such as when using a system-on-a-chip architecture, the processor may include one or more fixed-function blocks (also called accelerators) that may implement part of the method in hardware (rather than in software or firmware). Platform software, consisting of an operating system or any other suitable platform software, may be provided on the computing-based device to execute application software on the device.

[0091] The methods described above can be executed using computer-executable instructions, which can be provided using any computer-readable medium accessible to a computing device. For example, computer-readable media can include computer storage media (such as memory) and communication media. Computer storage media (such as memory) include volatile and non-volatile, removable and non-removable media implemented in any way or technology for storing computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computer device. Conversely, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals (such as carrier waves) or other transmission mechanisms. As defined herein, computer storage media does not include communication media.

[0092] As used herein, the term "computer" refers to any device with processing power capable of executing instructions. Those skilled in the art will recognize that this processing power is integrated into many different devices; hence the term "computer"...

[0093] This includes personal computers, servers, mobile phones, personal digital assistants, and many other devices.

[0094] Those skilled in the art will recognize that storage devices used to store program instructions can be distributed across a network. For example, a remote computer can store examples of the processes described above. A local computer or terminal computer can access a remote computer and download some or all of the software to run the program. Alternatively, a local computer can download some of the software as needed, or execute some software instructions on a local terminal and execute some software instructions on a remote computer (or computer network). Those skilled in the art will also recognize that, by utilizing conventional techniques known to those skilled in the art, all or part of the software instructions can be executed by dedicated circuitry, such as digital signal processors, programmable logic arrays, etc.

[0095] It is understood that the above benefits and advantages may relate to one embodiment or multiple embodiments. These embodiments are not limited to those that solve any or all of the described problems, nor are they limited to those that have any or all of the described benefits and advantages.

[0096] Any reference to "one" item means one or more of these items. The term "comprising" as used herein means including the identified method steps or elements, but this is not a unique list, and a method or apparatus may contain additional steps or elements.

[0097] The order of steps in the methods described herein is exemplary, but these steps can be performed in any suitable order, or simultaneously where appropriate. Furthermore, steps can be added or substituted, or individual steps can be deleted from any method, without departing from the scope of the subject matter described herein. Various aspects of any of the above embodiments can be combined with various aspects of any other embodiment to form further embodiments without losing the desired effect.

[0098] It is understood that the above description of preferred embodiments is given by way of example only, and various modifications can be made by those skilled in the art. Although various embodiments have been described to some extent above, or reference has been made to one or more individual embodiments, those skilled in the art can make numerous modifications to the disclosed embodiments without departing from the spirit or scope of the invention.

Claims

1. A computer-based method for controlling a robotic lawnmower, the method comprising: Receive images from the robotic lawnmower's onboard imaging sensor, the images including the ground along the path the robotic lawnmower is about to traverse; A semantic segmentation algorithm is applied to generate a segmented image from the received image, the segmented image including regions corresponding to features in the image; A perspective transformation is performed on the segmented image to obtain a top-view transformed image, and the region is preserved in the transformed image; Based on the transformed image, determine the position of each region relative to the current position of the robotic lawnmower; Based on the positions of each of the aforementioned regions relative to the current position of the robotic lawnmower, a local map of the environment of the robotic lawnmower is drawn. This local map is an environmental map within a finite distance or radius from the robotic lawnmower, wherein the distance or radius is smaller than the size of the lawn area. The robotic lawnmower is controlled to navigate within the lawn area using the local map.

2. The method of claim 1, wherein the region comprises one or more regions containing non-grassland features, wherein, The non-grassland features include one or more of obstacles, hazards, and / or lawn area boundaries.

3. The method according to claim 1, wherein the size of the local map is limited according to a threshold distance from the current position of the robotic lawnmower, such that the area included in the local map is at a distance from the current position of the robotic lawnmower that is less than the threshold distance.

4. The method of claim 1, wherein drawing the local map comprises: The previous iteration of obtaining the local map was drawn using previously received image data; as well as One or more regions from the transformed image corresponding to the received image are added to the previous iteration of the local map.

5. The method of claim 4, wherein drawing the local map comprises: Delete the portions of the local map that correspond to one or more regions in the previous iteration, where the distance between these regions and the current position of the robotic lawnmower is greater than a threshold distance; as well as One or more regions from the transformed image corresponding to the received image are added to the previous iteration of the local map, and the distance between one or more regions and the current position of the robot lawnmower is less than a threshold distance.

6. The method of claim 4, wherein drawing the local map comprises: Record the number of previous images used in the previous iteration to draw the local map; Compare the number of recorded images with the maximum number of images; as well as When the number of recorded images used to draw the local map in the previous iteration equals the maximum number of images: After receiving an image from the imaging sensor, delete the portion of the local map that corresponds to the oldest image in the previous iteration; as well as One or more regions from the transformed image corresponding to the received image are added to the previous iteration of the local map, so as not to exceed the maximum number of images in the local map.

7. The method according to claim 1, wherein determining the position of each of the regions relative to the current position of the robotic lawnmower based on the transformed image comprises: The VSLAM algorithm is used to process the transformed image to obtain the position of each region in the transformed image relative to the current position of the robotic lawnmower.

8. The method of claim 1 further includes: Acquire additional data related to the movement of the robotic lawnmower; as well as Based on the additional data, the position of each region in the local map relative to the current position of the robotic lawnmower is updated, wherein the additional data includes sensor data from one or more additional sensors, including one or more of the following: mileage data, IMU data, and GPS data.

9. The method of claim 8, wherein updating the position of each region in the local map relative to the current position of the robotic lawnmower based on the additional data comprises: For each region in the local map: Obtain the previous location of the area from the local map; as well as The previous position of the region is modified based on the additional sensor data to determine the updated position of the region.

10. The method of claim 8, wherein the additional data includes region tracking data for one or more regions in the received image.

11. The method of claim 10, further comprising: The region tracking data is generated for one or more regions present in the received image using the following method: Identify one or more regions in one or more previously received images; By tracking one or more regions using one or more previously received images and the received images, a tracking path can be determined for each of the one or more regions.

12. The method of claim 11 further includes: When one or more regions no longer appear in subsequently received images, infer the tracking path of one or more regions; as well as Based on the inferred tracking path, update the position of one or more regions in the local map relative to the current position of the robotic lawnmower.

13. The method of claim 8, wherein the frequency of updating the position of the region in the local map relative to the current position of the robotic lawnmower based on the additional data is higher than the frequency of updating the position of the region in the local map relative to the current position of the robotic lawnmower based on the received image.

14. The method of claim 1, wherein controlling the robotic lawnmower to navigate within the lawn area using the local map comprises: Access the local map; as well as The robotic lawnmower is controlled to navigate within the lawn area based on the location indicated by the local map.

15. The method of claim 14, wherein controlling the robotic lawnmower to navigate within the lawn area based on the area location indicated by the local map comprises: Control one or more drive mechanisms of the robotic lawnmower to move the robotic lawnmower to navigate within a lawn area.

16. Robotic lawnmower, including: One or more drive mechanisms; Imaging sensor; Storage module; as well as The control module includes a processor communicatively coupled to one or more of the drive mechanisms, the imaging sensor, and the storage module; the processor is configured to perform the method of claim 1.

17. The robotic lawnmower of claim 16, wherein the imaging sensor is a camera configured to capture wide-angle images with a field of view exceeding 100 degrees.

18. The robotic lawnmower of claim 16 further includes: One or more additional sensors are communicatively connected to the processor, and the one or more additional sensors are configured to provide sensor data relating to at least one of the following: The speed of the robotic lawnmower; The location of the robotic lawnmower; The direction of the robotic lawnmower; The distance traveled by the robotic lawnmower; The acceleration of the robotic lawnmower; and The distance between the robotic lawnmower and one or more surrounding obstacles.

19. The robotic lawnmower of claim 16, wherein the control module is configured to communicate with an external server.

20. A computer-readable medium having instructions stored thereon that, when executed by a processor, cause the processor to perform the method of claim 1.

Images