Autonomous machine navigation and training using vision systems

Abstract

Autonomous machine navigation technology can generate a 3D point cloud representing at least one work area based on feature data and matching data. Attitude data associated with the points in the 3D point cloud can be generated, representing the attitude of the autonomous machine. The attitude data can be used to determine boundaries for subsequent navigation of the autonomous machine within the work area. Non-visual sensor data can be used to determine the attitude. The attitude can be updated based on visual attitude data. The autonomous machine can navigate within the boundaries of the work area based on the updated attitude. The 3D point cloud can be generated based on data captured during the inspection phase. Boundaries can be generated based on data captured during the mapping phase.

Description

[0001] This application refers to application number 62 / 716,208, filed on August 8, 2018.

[0002] The benefits of U.S. Provisional Application No. 62 / 716,716, filed August 9, 2018; Application No. 62 / 741,988, filed October 5, 2018; and Application No. 62 / 818,893, filed March 15, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to autonomous machine navigation. More specifically, this disclosure relates to autonomous machine navigation for ground maintenance machines. Background Technology

[0004] Ground maintenance machines, such as lawnmowers and garden machines, are known for performing a variety of tasks. For example, homeowners and professionals use powered lawnmowers to maintain lawn areas in their properties or yards. Lawnmowers that perform mowing autonomously are also known. Some lawnmowers operate within a predefined boundary of the work area. Such lawnmowers may rely on a navigation system that helps the lawnmower autonomously remain within the predefined boundary. For example, certain boundaries defined by wires can be detected by the lawnmower. The lawnmower navigates by moving freely within the boundary and redirects its trajectory when a boundary wire is detected. For certain work areas or certain autonomous maintenance tasks, the use of boundary wires may be undesirable. For example, boundary wires may be expensive and cumbersome to install, may break and become inoperable, or may be difficult to move to redefine the desired boundary of the work area. However, the mobile nature of lawnmowers limits available computing resources such as processing power, storage capacity, and battery life, which could be used for other, more complex navigation types of lawnmowers. Summary of the Invention

[0005] Embodiments of this disclosure relate to navigation for autonomous machines, more specifically to autonomous navigation and operation within the boundaries of a work area, and even more specifically, can be adapted to autonomous machines with limited computing resources. The techniques of this disclosure provide robust processes for training autonomous machines to navigate within a work area.

[0006] In one aspect, a method for navigating an autonomous machine includes determining a current pose of the autonomous machine based on non-visual pose data captured by one or more non-visual sensors of the autonomous machine. The pose represents one or both of the autonomous machine's position and orientation within a work area defined by one or more boundaries. The method further includes determining visual pose data based on image data captured by the autonomous machine. The method also includes updating the current pose based on the visual pose data to correct or reposition the current pose, and providing an updated pose of the autonomous machine within the work area for navigating the autonomous machine within the work area.

[0007] In another aspect, an autonomous machine includes: a housing coupled to a maintenance tool; a set of wheels supporting the housing on the ground; a propulsion controller operatively coupled to the set of wheels; a vision system having at least one camera adapted to capture image data; and a navigation system operatively coupled to the vision system and the propulsion controller. The navigation system is adapted to guide the autonomous machine within a work area. The navigation system is configurable to determine the current attitude of the autonomous machine based on non-visual attitude data captured by one or more non-visual sensors of the autonomous machine. The attitude represents one or both of the autonomous machine's position and orientation within a work area defined by one or more boundaries. The navigation system is configurable to include determining visual attitude data based on image data captured by the at least one camera. The navigation system is configurable to update the current attitude based on the visual attitude data to correct or reposition the current attitude, and to provide the autonomous machine with an updated attitude within the work area for navigating the autonomous machine within the work area.

[0008] In another aspect, a method for navigation training of an autonomous machine may include: during a patrol phase of a training mode, guiding the autonomous machine along at least one of the perimeter or interior of a work area to record a first set of patrol images associated with the perimeter or a second set of patrol images associated with the interior; during an offline mode, generating a three-dimensional point cloud (3DPC) based on at least one of the first set of patrol images and the second set of patrol images; and during a rendering phase of a training mode, guiding the autonomous machine along one or more paths to record sensor fusion data, thereby defining one or more boundaries of the work area in a navigation map.

[0009] The summary is not intended to describe every embodiment or implementation of this disclosure. A more complete understanding of this disclosure will become apparent and understood by referring to the following detailed description and claims, taken in conjunction with the accompanying drawings. Attached Figure Description

[0010] Exemplary embodiments will be further described with reference to the accompanying drawings, in which:

[0011] Figure 1 This is a schematic side view of an autonomous ground maintenance machine with a vision system according to an embodiment of the present disclosure.

[0012] Figure 2A This is an embodiment of the usability according to the present disclosure. Figure 1 A plan view of the work area within the boundary where the machine operates.

[0013] Figure 2B yes Figure 2A The diagram shown is a plan of the partitions within the work area and is based on an embodiment of this disclosure. Figure 1 The example shown illustrates how a machine plans a path within the boundaries of the defined partition.

[0014] Figure 3 According to one embodiment of this disclosure, it includes the ability to use Figure 1 The diagram shows the work area of ​​the exclusion zone and transfer zone in which the machine operates.

[0015] Figure 4 According to one embodiment of this disclosure Figure 1 A schematic view of the various systems of the machine shown.

[0016] Figure 5 According to one embodiment of this disclosure Figure 1 Schematic views of various modes of the machine shown.

[0017] Figure 6 According to an embodiment of this disclosure, the direction and Figure 1 The diagram shows a schematic view of the sensor data provided by the navigation system communicating with the platform of the machine.

[0018] Figure 7 According to one embodiment of this disclosure, and Figure 6 A schematic view of sensor data input and sensor fusion processing in a sensor fusion module used in the navigation system shown.

[0019] Figure 8 According to one embodiment of this disclosure Figure 5 A functional diagram of the vision system during the training mode shown.

[0020] Figure 9 According to one embodiment of this disclosure Figure 5 The diagram shows the functionality of the vision system during offline mode.

[0021] Figure 10According to one embodiment of this disclosure Figure 5 The diagram shows the functionality of the vision system during online mode.

[0022] Figure 11 It is according to one embodiment of this disclosure that training images are used to generate in Figure 5 The diagram shows a 3D point cloud used during offline mode.

[0023] Figure 12 This is based on an embodiment of the present disclosure and its use. Figure 9 The diagram shows the pose estimation associated with the 3D point cloud generated by the visual map building module.

[0024] Figure 13 This is a schematic diagram of pose estimation associated with a low-quality portion of a 3D point cloud of a specific work area according to an embodiment of the present disclosure.

[0025] Figure 14 According to one embodiment of this disclosure, and Figure 13 A schematic diagram of the pose estimation associated with the updated 3D point cloud of the working area shown.

[0026] Figure 15 According to one embodiment of this disclosure Figure 9 A schematic view of the visual map building methods used during the visual map building module shown.

[0027] Figure 16 Training according to an embodiment of this disclosure Figure 1 The flowchart shows a training method for the machine shown.

[0028] Figure 17 This is an embodiment of the present disclosure for operation Figure 1 The flowchart shows the autonomous machine navigation method of the machine shown.

[0029] Figure 18 Training according to an embodiment of this disclosure Figure 1 A flowchart of another training method for the machine.

[0030] Figure 19 According to one embodiment of this disclosure Figure 18 The flowchart shows the inspection phase of the training method.

[0031] Figure 20 This is for at least partially performing an embodiment of the present disclosure. Figure 18 The flowchart shows the specific steps of the training method.

[0032] Figure 21 It is possible to use according to an embodiment of the present disclosure Figure 1 A perspective view of the handle assembly of the machine shown.

[0033] Figure 22 Training according to an embodiment of this disclosure Figure 1 The flowchart shows another training method for the machine.

[0034] Figure 23 According to one embodiment of this disclosure Figure 3 A schematic view of the base shown.

[0035] These figures are given primarily for clarity and are therefore not necessarily drawn to scale. Furthermore, various structures / components, including but not limited to fasteners, electrical components (wiring, cables, etc.), may be shown schematically or removed from some or all views to better illustrate various aspects of the depicted embodiments, or where such structures / components are not necessarily required for understanding the various exemplary embodiments described herein. However, the omission of illustrations / descriptions of such structures / components in certain figures should not be construed as limiting the scope of the various embodiments in any way. Detailed Implementation

[0036] In the following detailed description of exemplary embodiments, reference is made to the accompanying drawings, which form a part thereof. It should be understood that other embodiments not described and / or shown herein are of course conceivable.

[0037] All headings provided herein are for the convenience of the reader and should not be used to limit the meaning of any text following the heading, unless so specified. Furthermore, unless otherwise indicated, all figures indicating quantity and all terms indicating direction / orientation (e.g., vertical, horizontal, parallel, perpendicular, etc.) in the specification and claims shall in any case be understood to be modified by the term “about”. Additionally, the term “and / or” (if used) refers to one or all of the listed elements, or any combination of two or more of the listed elements. Moreover, “that is” may be used herein as an abbreviation of “idet” and means “that is,” while “for example” may be used as an abbreviation of “exempli gratia” and means “for instance.”

[0038] Embodiments of this disclosure provide autonomous machine navigation methods and systems for autonomous navigation and operation within the boundaries of a work area, particularly for ground maintenance, such as lawn mowing. The autonomous machine can be configured in different modes to perform various navigation functions, such as a training mode, an offline mode, and an online mode. The autonomous machine can define one or more boundaries of the work area using, for example, a vision system and non-visual sensors instead of boundary lines. The autonomous machine can correct its position or orientation within the work area by using a position or orientation determined by the vision system, which is determined or estimated using one or more non-visual sensors. The training of the autonomous machine can be performed during a training mode, which may include one or more phases, such as a survey phase and a mapping phase.

[0039] Some aspects described herein relate to defining the boundaries of a work area using a vision system and non-visual sensors. Some aspects of this disclosure relate to using a vision system to correct estimated positions within the work area. The vision system may utilize one or more cameras. Images can be recorded by guiding an autonomous machine along a desired boundary path (e.g., during training mode). Algorithms can be used to extract features to match features between different images and generate a 3D point cloud (3DPC or 3D point cloud) that at least corresponds to the work area (e.g., during offline mode). The position and orientation of the autonomous machine during image recording can be determined for each point in the 3DPC, for example, based on the position of each point in the 3DPC and the position of corresponding features in the recorded images. The position and orientation can also be recovered directly during point cloud generation. At least the position information can be used to determine the boundaries of the work area for subsequent navigation of the autonomous machine within the work area. During operation (e.g., during online mode), the vision machine can record operational images and determine the vision-based position and orientation of the autonomous machine. The vision-based position can be used to update or correct errors in the position determined or estimated based on non-visual sensors. The aspects described herein relate to utilizing limited computational resources while achieving proper navigation of the work area. Processing of the recorded images can occur during offline mode (e.g., when the autonomous machine is charging overnight). The vision system can be used at a low refresh rate to complement a high refresh rate non-vision-based navigation system.

[0040] Although described in the example as an autonomous lawnmower, such a configuration is merely exemplary, as the systems and methods described herein can also be applied to other autonomous machines, including, for example, commercial lawnmower products (e.g., user-driven ride-on lawnmowers or green lawnmowers), other ground-working machines or vehicles (e.g., debris blowers / vacuum cleaners, aerators, de-icing machines, material spreaders, snowplows, weed trimmers for weed control), and indoor work vehicles such as vacuum cleaners and floor scrubbers / cleaners (e.g., indoor work vehicles that may encounter obstacles), construction and multi-purpose vehicles (e.g., trenchers), observation vehicles, and load transport (including people and goods, such as passenger vehicles and hauling equipment). Furthermore, the autonomous machines described herein can employ one or more of various types of navigation, such as stochastic, modified stochastic, or specific path planning, to perform the intended function.

[0041] It should be noted that the terms “have,” “comprising,” “including,” and variations thereof are not restrictive in the appended specification and claims, and are used in their open-ended sense, generally meaning “including but not limited to.” Furthermore, “a,” “an,” “the,” “at least one,” and “one or more” are used interchangeably herein. Additionally, relative terms such as “left,” “right,” “front,” “front part,” “forward,” “rear,” “rear part,” “backward,” “top,” “bottom,” “side,” “up,” “down,” “above,” “below,” “horizontal,” “vertical,” etc., may be used herein, and if used, these terms derive from the perspective shown in a particular figure, or when the machine 100 is in an operating configuration (e.g., as shown in…). Figure 1 As shown, when vehicle 100 is positioned such that wheels 106 and 108 rest on a generally level ground 103. However, these terms are used only for simplicity of description and are not intended to limit the interpretation of any of the described embodiments.

[0042] As used herein, the terms “determine” and “estimate” may be used interchangeably depending on the specific context in which they are used, for example, to determine or estimate the position, posture, or characteristics of lawnmower 100.

[0043] While the actual construction of the ground maintenance machine may not be at the core of this disclosure, Figure 1The diagram illustrates an example of an autonomous ground maintenance machine (e.g., an autonomously operated vehicle, such as an autonomous lawn mower 100) of a lawn mowing system (for ease of description, a lawnmower 100 is schematically shown). As shown in this view, the lawnmower 100 may include a housing 102 (e.g., a frame or chassis with a guard) that carries and / or encloses various components of the lawnmower as described below. The lawnmower 100 may also include ground support components such as wheels, rollers, or tracks. In an exemplary embodiment, the ground support components shown include one or more rear wheels 106 and one or more front wheels 108 that support the housing 102 on the ground (grass) 103. As shown, the front wheels 108 support the front end portion 134 of the lawnmower housing 102, while the rear wheels 106 support the rear end portion 136 of the lawnmower housing.

[0044] One or both of the rear wheels 106 may be driven by a propulsion system (e.g., including one or more electric wheel motors 104) to propel the lawnmower 100 across the ground 103. In some embodiments, the front wheels 108 may be freely rotatable relative to the housing 102 (e.g., about a vertical axis). In such a configuration, the direction of the lawnmower can be controlled by means of the differential rotation of the two rear wheels 106 in a manner similar to that of a conventional zero turning radius (ZTR) ride-on lawnmower. That is, the propulsion system may include separate wheel motors 104 for each of the left and right rear wheels 106, allowing the speed and direction of each rear wheel to be controlled independently. Alternatively or as an alternative, the front wheels 108 may be actively steerable by a propulsion system (e.g., including one or more steering motors 105) to assist in controlling the direction of the lawnmower 100, and / or may be driven by the propulsion system (i.e., to provide a front-wheel drive or all-wheel drive lawnmower).

[0045] An implement (e.g., a mowing element, such as blade 110) can be coupled to a cutting motor 112 (e.g., an implement motor) carried by housing 102. When motors 112 and 104 are energized, the lawnmower 100 can be advanced across ground 103 such that vegetation (e.g., grass) passed by the mower is cut by blade 110. Although shown herein as using only a single blade 110 and / or motor 112, lawnmowers comprising multiple blades powered by single or multiple motors are contemplated within the scope of this disclosure. Furthermore, while described herein in the context of one or more conventional “blades,” other cutting elements are of course possible without departing from the scope of this disclosure, including, for example, discs, nylon rope or thread elements, blades, cutting spools, etc. Further, embodiments combining various cutting elements are also contemplated, such as a rotating blade with an edge-mounted cord trimmer.

[0046] The lawnmower 100 may also include a power source, in one embodiment of which is a battery 114 having a lithium-based chemistry (e.g., lithium-ion). Other embodiments may utilize batteries with other chemistry or utilize entirely other power source technologies (e.g., solar energy, fuel cells, internal combustion engines) without departing from the scope of this disclosure. It should also be noted that although shown as using separate blades and multiple wheel motors, such a configuration is merely exemplary, as embodiments where the blades and wheels are powered by a single implement motor are also conceivable.

[0047] The lawnmower 100 may also include one or more sensors to provide location data. For example, some embodiments may include a Global Positioning System (GPS) receiver 116 (or other location sensors that can provide similar data), adapted to estimate the position of the lawnmower 100 within a work area and provide such information to the controller 120 (described below). In other embodiments, one or more of the wheels 106, 108 may include an encoder 118 that provides rotational / speed information of the wheels that can be used to estimate the lawnmower's position within a given work area (e.g., based on an initial starting position). The lawnmower 100 may also include a sensor 115 adapted to detect boundary threads, which can be used in conjunction with other navigation technologies described herein.

[0048] The lawnmower 100 may include one or more front obstacle detection sensors 130 and one or more rear obstacle detection sensors 132, as well as other sensors, such as side obstacle detection sensors (not shown). The obstacle detection sensors 130 and 132 can be used to detect obstacles in the path of the lawnmower 100 as the lawnmower 100 travels in either a forward or reverse direction. The lawnmower 100 is capable of mowing while moving in either direction. As shown, the sensors 130 and 132 may be located at the front end 134 and rear end 136 of the lawnmower 100, respectively.

[0049] Sensors 130 and 132 may use contact sensing, non-contact sensing, or both. For example, depending on the state of the lawnmower 100 (e.g., moving within or between zones), both contact and non-contact sensing may be enabled simultaneously, or only one type of sensing may be used. An example of contact sensing includes using a contact buffer protruding from housing 102, or the housing itself, which can detect when the lawnmower 100 comes into contact with an obstacle. Non-contact sensors may sometimes detect obstacles at a distance from the lawnmower 100 using sound or light waves (e.g., using infrared, radio detection and odometer (radar), light detection and ranging (LiDAR), etc.).

[0050] The lawnmower 100 may include one or more vision-based sensors to provide positioning data, such as position, orientation, or speed. The vision-based sensors may include one or more cameras 133 that capture or record images for use by the vision system. Cameras 133 may be described as part of the lawnmower 100's vision system. Image types include, for example, training images and / or operational images.

[0051] One or more cameras are capable of detecting visible light, invisible light, or both. The one or more cameras can establish a total field of view of at least 30 degrees, at least 45 degrees, at least 60 degrees, at least 90 degrees, at least 120 degrees, at least 180 degrees, at least 270 degrees, or even at least 360 degrees around the autonomous machine (e.g., lawnmower 100). The field of view can be defined horizontally, vertically, or both. For example, the total horizontal field of view could be 360 ​​degrees, and the total vertical field of view could be 45 degrees. The field of view can capture image data above and below the height of the one or more cameras.

[0052] In some embodiments, the lawnmower 100 includes four cameras 133. A camera 133 may be positioned in each of one or more directions, including a forward direction, a backward direction, a first lateral direction, and a second lateral direction (e.g., relative to the basic orientation of the lawnmower 100). One or more camera orientations may be positioned orthogonal to one or more other cameras 133 or opposite to at least one other camera 133. Cameras 133 may also be offset from any of these directions (e.g., at 45 degrees or other non-right angles).

[0053] The lawnmower 100 can be guided along a path, for example, manually using the handle assembly 90. Specifically, the manual direction of the lawnmower 100 can be used during training mode to learn the work area or the boundaries associated with the work area. The handle assembly 90 can extend outward and upward from the rear end 136 of the lawnmower 100.

[0054] The camera 133, positioned in the forward direction, can have a pose representing the autonomous machine's attitude. The camera's attitude can be a six-degree-of-freedom attitude, which can include all position and orientation parameters in three-dimensional space (see [link to previous section]). Figure 6 (See related description). In some embodiments, the position and orientation of the camera may be defined relative to either the geometric center of the lawnmower 100 or the boundary of the lawnmower 100.

[0055] The sensors of the lawnmower 100 can also be described as vision-based sensors and non-vision-based sensors. Vision-based sensors may include a camera 133 capable of recording images. These images can be processed and used to construct a 3D PC or for optical ranging (e.g., optical coding). Non-vision-based sensors (non-visual sensors) may include any sensor other than the camera 133. For example, a wheel encoder that uses optical (e.g., photodiode), magnetic, or capacitive sensing to detect wheel rotations can be described as a non-visual sensor that does not use a camera. Wheel coding data from the wheel encoder can also be described as ranging data. In some embodiments, the non-visual sensor does not include a boundary line detector. In some embodiments, the non-visual sensor does not include receiving signals from an external system, such as from GPS satellites or other transceivers.

[0056] Optical coding can be used to determine or estimate the distance traveled between images by taking a series or sequence of images and comparing features in the images. Optical coding may be less susceptible to wheel slippage than wheel encoders used to determine distance or speed.

[0057] In addition to the sensors mentioned above, other sensors that are known now or will be developed in the future may also be incorporated into the lawnmower 100.

[0058] The lawnmower 100 may also include a controller 120 adapted to monitor and control various lawnmower functions. The controller 120 may include a processor 122 that receives various inputs and executes one or more computer programs or applications stored in a memory 124. The memory 124 may include computer-readable instructions or applications that, when executed, for example, by the processor 122, cause the controller 120 to perform various calculations and / or issue commands. That is, the processor 122 and the memory 124 together may define a computing device operable for processing input data and generating desired outputs to one or more components / devices. For example, the processor 122 may receive various input data, including position data, from a GPS receiver 116 and / or an encoder 118, and generate commands for the speed and steering angle of one or more wheel motors 104 to rotate the drive wheels 106 (at the same or different speeds and in the same or different directions). In other words, the controller 120 can control the steering angle and speed of the lawnmower 100, as well as the speed and operation of the cutting blades.

[0059] Typically, based on GPS receiver 116 ( Figure 1The GPS data generated from the data of the lawnmower 100 can be used in various ways to determine the attitude of the lawnmower 100. In some embodiments, the GPS data can be used as one of the non-visual sensors to help determine non-visual attitude data. Visual attitude data can be used to update or correct non-visual attitude data. GPS data can also be used to help update or correct estimated attitude, which may be based on non-visual and / or visual attitude data. In some embodiments, the GPS data can be enhanced using GPS-specific correction data such as real-time kinematics (RTK) data. GPS-RTK data can provide a more accurate or precise position to correct for GPS timing anomalies compared to nominal GPS data.

[0060] Various parameters, data, or data structures can be referenced here. These parameters, data, or data structures can be processed in the controller 120, for example, by being processed by the processor 122 or stored in or retrieved from the memory 124.

[0061] The controller 120 may use the processor 122 and memory 124 in a variety of different systems. Specifically, one or more processors 122 and memory 124 may be included in each different system. In some embodiments, the controller 120 may at least partially define a vision system, which may include the processor 122 and memory 124. The controller 120 may also at least partially define a navigation system, which may include processors 122 and memory 124 separate from those of the vision system.

[0062] Each system can also be described as having its own controller 120. For example, a vision system can be described as including one controller 120, and a navigation system can be described as having another controller 120. Thus, a lawnmower 100 can be described as having multiple controllers 120. Generally, as used herein, the term "controller" can be used to describe a component of a "system" that provides commands to various other components of the control system.

[0063] Furthermore, the lawnmower 100 can communicate with a separate device, such as a smartphone or remote computer. Applications on smartphones or remote computers can be used to identify or define problem areas or obstacles. For example, a user can identify problem areas or obstacles on a map of the mowing area. An example of an obstacle is a permanent obstacle, such as a boulder. The lawnmower 100 can receive the identified problem area or obstacle from a separate device. In this case, the lawnmower 100 can be configured to mow only in a specific direction when passing through the problem area upon receiving the identified problem area, or the lawnmower can be configured to adopt an active avoidance strategy to avoid hitting obstacles when crossing slopes upon receiving the identified obstacle, and can establish a exclusion zone around permanent obstacles.

[0064] In view of the foregoing, it will be apparent that the functionality of controller 120 can be implemented in any manner known to those skilled in the art. For example, memory 124 may include any volatile, non-volatile, magnetic, optical, and / or electrical medium, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), electrically erasable programmable ROM (EEPROM), flash memory, and / or any other digital medium. Although shown as both memory 124 and processor 122 being included in controller 120, they may be contained in separate modules.

[0065] Processor 122 may include any one or more of the following: a microprocessor, a controller, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and / or equivalent discrete or integrated logic circuitry. In some embodiments, processor 122 may include multiple components, such as one or more microprocessors, one or more controllers, one or more DSPs, one or more ASICs, and / or one or more FPGAs, and any combination of other discrete or integrated logic circuitry. The functionality of controller 120 and / or processor 122 herein may be implemented as software, firmware, hardware, or any combination thereof. Some functionality of controller 120 may also be performed in the cloud or other distributed computing systems operatively connected to processor 122.

[0066] exist Figure 1The diagram illustrates the schematic connections between the controller 120 and the battery 114, one or more wheel motors 104, blade motor 112, optional boundary wire sensor 115, radio 117, and GPS receiver 116. These interconnections are merely exemplary, as the various subsystems of the lawnmower 100 can be connected in virtually any manner, such as directly, wirelessly, via a bus architecture (e.g., a controller area network (CAN) bus), or any other connection that allows the transmission of data and / or power between the various components of the lawnmower. Although connections to some sensors 130, 132, 133 are not shown, these sensors and other components of the lawnmower 100 can be connected in a similar manner. The radio 117 can be connected via cellular or other wide area networks (e.g., even the Internet), local area networks (e.g., IEEE 802.11 “Wi-Fi” devices), or peer-to-peer (P2P) connections (e.g., BLUETOOTH). TM The network communicates with mobile devices 119 (e.g., mobile computing devices, mobile computers, handheld computing devices, smartphones, cellular phones, tablets, desktop computers, wearable computers, smartwatches, etc.). Furthermore, mobile devices 119 can communicate with other devices via similar networks; for example, they can be used to connect lawnmower 100 to the internet.

[0067] In some embodiments, various functions of one or more controllers 120 described herein may be offloaded from lawnmower 100. For example, recorded images may be transmitted to a remote server (e.g., in the cloud) and processed or stored using wireless device 117. Stored images or other processed data may be received using wireless device 117 and stored on or further processed by lawnmower 100.

[0068] Figure 2 and Figure 3 A work area 200 or fenced areas 202, 210 within the work area 200 are shown. Boundaries may be defined or defined around the work area 200. The lawnmower 100 may cover the work area 200 using various methods (e.g., traversing or passing through the work area to mow the grass). In some embodiments, the lawnmower 100 may traverse the work area 200 randomly, semi-randomly, or along a planned path. In some embodiments, depending on the method used to cover the work area 200, other boundaries may be defined around the fenced areas 202, 210 within the boundaries of the work area 200. For example, fenced areas 202, 210 may be mobile fenced areas or static fenced areas.

[0069] Figure 2AThe image illustrates an example of covering a work area 200 using multiple zones 202, 210 (e.g., fenced areas) with a lawnmower. The work area 200 can represent an outdoor area or a maintenance area, such as a lawn. The lawnmower 100 can be operated to travel along multiple paths through the work area 200 to adequately cut the grass within all work areas 200. The lawnmower 100 can be recharged as needed, for example, when moving between zones 202, 210. A rechargeable base or stand (similar to...) Figure 3 258) can be located within or along the work area 200.

[0070] Boundaries can be used to define the work area 200. Boundaries can be defined manually or autonomously using a training mode of the lawnmower 100. Alternatively, fixed property boundaries or other types of boundaries can be used to define boundaries. In some embodiments, boundaries can be defined by guiding the lawnmower 100 along the work area 200, specifically along a desired boundary path 250 of the work area 200.

[0071] Boundaries can be defined relative to the work area 200 for various purposes. For example, boundaries can be used to define fenced areas, such as zone 202, zone 210, or work area 200. Typically, the lawnmower 100 is guided to travel within the boundaries of a fenced area for a period of time. Another boundary can be used to define an exclusion zone. An exclusion zone can represent an area within work area 200 that the lawnmower 100 avoids or travels around. For example, an exclusion zone can include obstacles (such as a landscaping garden) or problem areas (such as a steep slope). Another boundary can be used to define a transfer zone, which can also be described as a transfer path. Typically, a transfer zone is a zone connecting two areas, such as a path connecting different fenced areas. A transfer zone can also be defined between a point in the work area and the "home" location or base. Maintenance tasks may or may not be performed in the transfer zone. For example, the lawnmower 100 may not mow in the transfer zone. In an example involving a yard separated by a driveway, the transfer zone may include the entire driveway, or at least a path spanning the driveway between two patches of lawn traversed by the lawnmower 100.

[0072] The work area 200 can be drawn as a topographic map. For example, the topographic map can be developed during the lawnmower's teaching mode or during subsequent mowing operations. In any case, the topographic map can contain information about the terrain of the work area 200, such as elevation, slope, identified obstacles (e.g., permanent obstacles), identified jamming areas (e.g., areas where the lawnmower gets stuck due to slope or other traction conditions), or other information that can facilitate the lawnmower 100's ability to traverse the work area.

[0073] Coordinate system 204 is shown for illustrative purposes only. The resolution of points stored in the topographic map may be sufficient to provide useful elevation and / or slope information (e.g., in feet or decimeters) about the terrain within the work area 200. For example, the resolution of points may correspond to the spacing between points that is less than or equal to the width of the lawnmower 100. In some cases, different functions of the path planning may use different levels of resolution. For example, path planning for mapping fenced or exclusion zones may have the highest resolution (e.g., in centimeters). In other words, points near, adjacent to, or close to irregular boundaries or obstacles may have a finer granular resolution.

[0074] The lawnmower 100 can cover the work area 200 starting from, for example, the boundary of the work area. The lawnmower 100 can define a first partition 202. The partition 202 can be located adjacent to the boundary of the work area 200, or, as shown, can be further located within the work area. In one embodiment, the partition 202 covers the entire work area 200.

[0075] In another embodiment, partition 202 does not cover the entire work area 200. When the lawnmower 100 finishes mowing partition 202, it can begin mowing another partition (e.g., partition 210, which may be dynamic or fixed).

[0076] The lawnmower 100 can determine starting coordinates or a starting point 206 within the first zone 202. For example, the starting coordinates 206 can be selected from the highest elevation point within zone 202 or from a point on the edge of zone 202. If needed, the lawnmower 100 can rotate to orient itself from its current position at the boundary of the work area 200 toward the starting coordinates 206. The lawnmower 100 can propel itself toward the starting coordinates 206.

[0077] After reaching starting coordinates 206, lawnmower 100 can begin traveling through partition 202 to mow the lawn within that partition. As described below, lawnmower 100 can use randomly generated destination waypoints within the partition. Alternatively, or in an alternative, lawnmower 100 can use a planned mode with planned waypoints within the partition. This mode of mowing can use planned waypoints to create and cover the area.

[0078] When lawnmower 100 reaches final destination waypoint 208, it has completed mowing within the current partition 202. Lawnmower 100 can determine the next partition 210 (partition 210 may or may not be directly adjacent to partition 202) and the next starting point 212 within that partition. Lawnmower 100 can orient itself and begin moving towards the next starting point 212. A path 220 starting from final destination waypoint 208 in partition 202 or toward the next starting point 212 in the next partition 210 can be described as a "destination-to-target" path (e.g., it may traverse transfer zones).

[0079] Once the lawnmower 100 reaches the next starting point 212, it can begin moving through the next section 210. The process of creating and moving through the fenced areas can be repeated multiple times to provide adequate coverage of the work area 200.

[0080] Figure 2B A top view of one method 300 for covering area 302 is shown, illustrating a series of paths for enabling lawnmower 100 to traverse at least a portion of the partition. The paths of lawnmower 100 shown can be applied, for example, when the boundary is in work area 200 (…). Figure 2A When the fenced area surrounding partition 302 is defined within the boundary of the lawnmower 100, the lawnmower 100 operates.

[0081] In the illustrated embodiment, the lawnmower 100 travels from a starting point 304 to a destination path point 306. After reaching the destination path point 306, the lawnmower 100 may determine a second destination path point 308, rotate X1 degrees, and travel toward the second destination path point. This sequence of rotation and travel can continue to reach a third destination path point 310, a fourth destination path point 312, and a final destination path point 314 (e.g., by rotating X2, X3, and X4 degrees, respectively). Although only a few destination path points 306, 308, 310, 312, and 314 are shown in this illustration, the lawnmower 100 may travel to more path points to adequately cover section 302. In some embodiments, the lawnmower 100 may select the minimum angle available for rotation and to move itself toward the next destination path point (e.g., rotating 90 degrees counterclockwise instead of 270 degrees clockwise).

[0082] Figure 3 An example of a work area 251 is shown, which includes a transfer area 252 or transfer path, such as a driveway, extending across the exclusion zone 254. Mowing areas or static fenced areas 256 of the work area 251 may be located on either side of the driveway, but no mowing area connects these two sides. To train the lawnmower 100 to pass through area 252, it may first be positioned at the desired starting point (see [reference]). Figure 3(Part indicated by solid lines in the middle lawnmower 100). Handle assembly 90 ( Figure 1 It can be in manual mode. Then, mobile device 119 ( Figure 1 The training phase or mode is initiated. Once initiated, the lawnmower 100 can be pushed or driven along the desired transfer zone 252. Once the desired path has been traversed (see...), Figure 3 In the case of a lawnmower 100 with a dotted line, the operator can end the training period and save the transfer zone. During autonomous lawnmower operation, the lawnmower 100 will use the defined transfer zone 252 to cross from one side of the driveway or exclusion zone 254 to the other. Multiple traversing transfer zones can be trained in any exclusion zone.

[0083] Once all boundaries (including exclusion zones) and transfer zones have been taught, a map of the work area can be presented to the user on the mobile device 119 so that the operator can confirm that all boundaries (including exclusion zones) and transfer zones have been properly calculated. The operator can then confirm that the boundaries and transfer zones are properly presented before the autonomous lawn mowing operation can begin. In some embodiments, the operator may be able to delete and / or modify boundaries and transfer zones using the mobile device during this check.

[0084] A transfer zone can be used to define how the lawnmower 100 moves from one part of the work area to another (or to a separate second work area). For example, the transfer zone can be configured to guide the lawnmower: to a specific mowing area; across a drainage area, such as a sidewalk, yard, or driveway that branches off the work area; or through a gate in a fenced yard. The lawnmower typically does not enter the drainage area unless the transfer zone is trained to traverse it. Furthermore, the lawnmower may not mow while moving along a portion of the transfer zone.

[0085] Not all exclusion zones can include transfer zones. For example, some exclusion zones may be limited to the area around obstacles that the lawnmower 100 cannot cross. Transfer zones that cross such exclusion zones may not be limited.

[0086] The base 258 can be set and positioned in or near the work area 251. The base 258 can be connected to a fixed or portable power source. The base 258 provides a storage location for the lawnmower when it is not in operation and includes a self-engaging electrical connection to allow the lawnmower to autonomously return to the base 258 and to power the lawnmower's battery 114 when needed. Figure 1 )Charge.

[0087] Figure 4 The image shows a lawnmower 100 ( Figure 1-3 The diagram illustrates the connections between various systems. The vision system 402 is operatively connectable to the navigation system 404. The navigation system 404 is operatively connectable to the propulsion system 406.

[0088] The navigation system 404 may record non-visual data during training mode, while the vision system 402 records images, such as training images. Although the lawnmower 100 can be manually operated by a user, in some embodiments, the navigation system 404 may autonomously guide the machine during training mode. The vision system 402 may include one or more cameras to record or capture images. In some embodiments, the controller of the vision system 402 may provide position and / or orientation data to the navigation system 404 based on the recorded images, which may facilitate navigation of the lawnmower 100. For example, the vision system 402 may provide the navigation system 404 with an estimated position and / or orientation of the lawnmower 100 based on visual sensor data.

[0089] In some embodiments, navigation system 404 may primarily use position and / or orientation based on non-visual sensor data for navigation. For example, non-visual sensor data may be based on output from an inertial measurement unit or wheel encoder. During training and / or offline modes, for example, the controller of navigation system 404 may use non-visual sensor data and vision-based data to determine boundaries for subsequent navigation of the autonomous machine within a work area. During online modes, for example, the controller of navigation system 404 may determine attitude based on visual attitude data, non-visual attitude data, or both. In some embodiments, attitude may be determined based on non-visual sensor data and updated based on visual attitude data. Navigation system 404 may compare vision-based position and / or orientation with non-visual-based position and / or orientation to correct errors and update position; this can be described as sensor fusion. In some embodiments, sensor data other than vision-based sensor data (e.g., GPS data) may be used to correct errors and update position.

[0090] The controller of navigation system 404 can command propulsion system 406 based on updated attitude. For example, corrected or updated position and / or orientation can be used by navigation system 404 to provide propulsion commands to propulsion system 406. Propulsion system 406 (e.g., propulsion hardware) can be defined as including, for example, motors 112, 104 and wheels 106, 108. Figure 1 ) and / or any associated drive (e.g., motor controller or microchip).

[0091] Figure 5 The image shows a lawnmower 100 ( Figure 1-3The diagram illustrates the available modes or states. As shown, lawnmower 100 can be configured to be in training mode 412, offline mode 414, and online mode 416. Lawnmower 100 can switch between various modes, which can also be described as configurations or states. Some functions of lawnmower 100 can be used during certain modes, for example, to optimally utilize computing resources.

[0092] As used herein, the term "training mode" refers to a routine or state of an autonomous machine (e.g., lawnmower 100) used to record data for later or subsequent navigation within a work area. During training mode, the machine can traverse the work area without performing maintenance functions. For example, a training mode for an autonomous lawnmower may include guiding the mower back and forth along a portion or all of the work area (e.g., along a desired boundary path) or partitions within the work area (e.g., fenced areas or exclusion zones), and allowing or disabling the use of mowing blades in that partition or work area. In some cases, a handle (e.g., Figure 1 The handle assembly 90) manually guides the lawnmower. In other cases, the lawnmower can be autonomously guided by the navigation system.

[0093] As used herein, the term "offline mode" refers to a routine or state for charging a portable power source or processing data recorded during online or training modes of an autonomous machine (e.g., lawnmower 100). For example, an offline mode for an autonomous lawnmower may include parking the lawnmower overnight at a charging station and processing data recorded during training or online modes.

[0094] As used herein, the term "online mode" refers to a routine or state for an autonomous machine (e.g., lawnmower 100) to operate in a work area, which may include traversing the work area and performing maintenance functions using maintenance tools. For example, the online mode of an autonomous lawnmower may include guiding the mower to cover or traverse a work area, or to partitions within the work area, and using mowing blades to mow the grass in the partitions or work areas.

[0095] Typically, the lawnmower 100 can be used with the mobile device 119 during, for example, training mode 412 and / or online mode 416. Figure 1 Interaction.

[0096] In some embodiments, when a user manually guides the lawnmower 100 during training mode, the mobile device 119 can be used to provide training speed feedback. The feedback can use, for example, a color-coded dashboard to indicate whether the user is moving the autonomous machine too quickly during training.

[0097] In some embodiments, the mobile device 119 can be used to notify a user of certain areas, partitions, or portions of a work area where the acquired images are insufficient. For example, an error may be detected in a certain area, and the mobile device 119 may notify the user where the area is, and may even guide the user along a path to record additional images to correct the detected error.

[0098] In some embodiments, the mobile device 119 can be used to select the type of boundary or partition used for training: a fenced area, an exclusion area, or a transfer area.

[0099] In some embodiments, the mobile device 119 can be used to provide real-time partition shape feedback. The partition shape may or may not be associated with real-world scale and orientation. For example, a map based on sensor data can be used to provide partition shape feedback to the mobile device 119.

[0100] Lawn mowers can provide completion time estimates via an app running on a mobile device or via periodic notifications (such as text messages) provided to the mobile device.

[0101] Figure 6 The image shows an autonomous machine (e.g., Figures 1 to 3 A schematic view of various systems of the lawnmower 100. Sensor 420 is operatively coupled to navigation system 404 to provide various sensor data, for example, during online mode. Vision system 402 (e.g., vision controller) and navigation system 404 (e.g., navigation controller) may each include their own processor and memory. Various modules of navigation system 404 are shown to implement various functions for autonomous machine navigation. Navigation system 404 may be operatively coupled to platform 460 to control the physical movements of the autonomous machine.

[0102] Sensor 420 may include sensors associated with navigation system 404, vision system 402, or both. Both navigation system 404 and vision system 402 may include the same type of sensors. For example, systems 402 and 404 may both include inertial measurement units (IMUs).

[0103] As used herein, the term "platform" refers to the lawnmower (e.g., the one that supports sensor 420 and navigation system 404) that supports sensor 420 and navigation system 404. Figures 1-3 The structure of the lawnmower 100. For example, the platform 460 may include a propulsion system 406 (e.g., a motor and wheels), a housing 102 ( Figure 1 ), Cutting motor 112 ( Figure 1 ) and maintenance tools 110 ( Figure 1 (and other possible components. In some embodiments, the entire autonomous machine may be described as being on platform 460.)

[0104] In the illustrated embodiment, sensor 420 includes a vision system 402 and non-visual sensors 422. Sensor data from sensor 420 can be provided to sensor fusion module 430. Specifically, vision system 402 can provide an estimated visual pose containing position and orientation parameters to sensor fusion module 430. Non-visual sensors 422 may include, for example, an IMU and / or wheel encoders. Sensor fusion module 430 can provide an estimated pose of the autonomous machine based on sensor data from sensor 420. Specifically, sensor fusion module 430 can estimate a non-visual pose based on data from non-visual sensors 422, which can be corrected or updated using a visual pose estimate determined based on visual sensor data from vision system 402.

[0105] As used herein, the term “attitude” refers to both position and orientation. Attitude can be a six-degree-of-freedom (6DOF) attitude, which may include all position and orientation parameters in three-dimensional space. Attitude data may include three-dimensional position and three-dimensional orientation. For example, the position may include at least one position parameter selected from: x-axis coordinates, y-axis coordinates, and z-axis coordinates (e.g., using a Cartesian coordinate system). Any suitable angular orientation representation may be used. Non-limiting examples of angular orientation representations include yaw, pitch, and roll representations, Rodriguez representations, quaternion representations, and direction cosine matrix (DCM) representations, which may be used individually or in combination. In one example, orientation may include at least one orientation parameter selected from yaw (e.g., vertical z-axis orientation), pitch (e.g., lateral y-axis orientation), and roll (e.g., longitudinal x-axis orientation).

[0106] The path planning module 440 can receive estimated attitudes of the autonomous machine from the sensor fusion module 430 and use the estimated attitudes for autonomous navigation. The path planning module 440 can receive other information or data to facilitate navigation. The obstacle detection module 432 can provide information about the presence and location of obstacles in the work area based on sensor data from the sensor 420. The navigation system 404 can also define and update a map 434 or navigation map of at least one work area. The map 434 can be defined or updated to define one or more fenced zones, exclusion zones, transfer zones, and mowing history. Each of the above can be provided to the path planning module 440 for navigation. The mowing history can also be provided to the scheduling management module 436. The scheduling management module 436 can be used to notify the path planning module 440 of various tasks of the autonomous machine, such as when to start mowing the work area within a week. Furthermore, the path planning module 440 can perform both global path planning (e.g., determining partitions within the work area) and local path planning (e.g., determining waypoints or starting points).

[0107] The propulsion controller 450 can receive data from the path planning module 440, the sensor fusion module 430, and the sensor 420. The propulsion controller 450 can use the received data to provide propulsion commands to the propulsion system 406. For example, the propulsion controller 450 can determine the speed or traction level based on data from the sensor 420. The path planning module 440 can provide the propulsion controller 450 with one or more waypoints or starting points, which can be used to traverse a portion or all of the work area. The sensor fusion module 430 can provide the propulsion controller 450 with the autonomous machine's rate or speed data, acceleration, position, and orientation. The propulsion controller 450 can also determine whether the autonomous machine is traversing the path determined by the path planning module 440 and can accordingly facilitate the correction of the machine's path.

[0108] Other information or data related to the maintenance functions of the autonomous machine can be provided to the propulsion controller 450 to control maintenance tools, such as grass-cutting blades. For example, the motor drive current for the grass-cutting blade motor can be provided to the propulsion controller 450. The propulsion controller 450 can also provide maintenance commands, for example, to control the maintenance tools on the platform 460.

[0109] Figure 7 An example of implementing sensor fusion module 430 using sensor data from sensor 420 is shown. Any suitable sensor data from various sensors 420 can be used. As shown, sensor 420 includes an inertial measurement unit 470, a wheel encoder 472, and a vision system 402.

[0110] Inertial measurement data from inertial measurement unit 470 can be used by attitude determination module 474. Attitude determination module 474 can provide an estimated attitude of the autonomous machine based at least in part on the inertial measurement data. Specifically, attitude determination module 474 can provide at least one of estimated position and orientation. In some embodiments, attitude determination module 474 can even provide one or more velocities (e.g., speed or rate).

[0111] Kalman filter 482 can be used to provide attitude estimation data to attitude determination module 474, which can be used to provide an estimated attitude for an autonomous machine. Specifically, Kalman filter 482 can provide at least one of estimated delta position, delta velocity, and delta orientation. As used herein, the term "delta" refers to a change in a variable or parameter. In some embodiments, the output data from Kalman filter 482 can be used to correct errors in the attitude estimated based on data from inertial measurement unit 470. Attitude determination module 474 can provide the corrected or updated attitude in sensor fusion output 484.

[0112] Kalman filter 482 can receive information or data based on the outputs from wheel encoder 472 and vision system 402. Wheel encoder 472 can provide wheel speed 476 to Kalman filter 482. Vision system 402 can provide optical ranging 478 and visual position correction 480. Optical ranging 478 can utilize images and determine information about the autonomous machine's movement, such as the distance the autonomous machine has traveled. Typically, optical ranging 478 can be used to determine changes in position, changes in orientation, linear velocity, angular velocity, or any combination thereof. Depending on the specific autonomous machine and application, any suitable optical ranging algorithm available to those skilled in the art can be used. Visual position correction 480 provided by vision system 402 can include vision-based attitude data, such as vision-based attitude estimation.

[0113] The attitude determination module 474 can receive or process data from the Kalman filter 482 at a low refresh rate and update it at a low rate. Data from the inertial measurement unit 470 can be received or processed at a high refresh rate and updated at a high rate faster than the Kalman filter data. The output from the sensor fusion output 484 can be fed back as input to the Kalman filter 482 to facilitate its operation. In other words, the attitude determination module 474 can provide an estimated attitude at a higher rate than the output of the Kalman filter 482 or the Kalman filter input (wheel speed 476, optical ranging 478, or visual position correction 480). For example, the visual position correction 480 can be performed at various rates on the order of one to four times per minute (e.g., approximately 1 / 10 Hz or 1 / 100 Hz), while the attitude determination module 474 can provide attitude on the order of 6000 times per minute (e.g., approximately 100 Hz). In some embodiments, the higher rate can be one, two, three, four, five, or even six times the lower rate.

[0114] In some embodiments (not shown), the Kalman filter 482 may be included in the attitude determination module 474. In some embodiments, the Kalman filter 482 may use a high refresh rate.

[0115] Figure 8 The diagram illustrates an example of training pattern 412 used for recording data, which can be generated by a vision system (e.g., Figure 4 A schematic representation of the various data and data structures used by the vision system 402. Typically, during training mode, data is recorded while the autonomous machine is guided along a task area (e.g., along the desired boundary of the task area). Specifically, training images can be recorded as the autonomous machine is guided along the task area.

[0116] During training mode, signals from one or more cameras (e.g., Figure 1 Camera data 502 from camera 133 (which may include front-facing, back-facing, left-facing, and right-facing cameras) can be provided as training images to data structure 510 and stored in data structure 510. Although camera data 502 from four cameras is shown, data from any number of cameras can be used. Camera data 502 may include images that can be described as image data or image data with timestamps. Camera data 502 can be described as vision-based data.

[0117] Furthermore, non-visual data can also be recorded during training mode. In the illustrated embodiment, non-visual data includes GPS data 504, IMU data 506, and ranging data 508 (e.g., wheel encoder data). Non-visual data can be provided to and stored in data structure 512. Non-visual data may include timestamped non-visual data. Any combination of non-visual data can be used. In some embodiments, non-visual data is optional and may not be used by the vision system.

[0118] While the vision system records data, the autonomous machine's navigation system can be used to observe and define the boundaries of fenced areas, exclusion zones, and transfer zones. These boundaries can be stored in the navigation system for subsequent navigation during online mode.

[0119] Figure 9 The diagram illustrates a schematic representation of various data, data structures, and modules of a vision system in an example of offline mode 414 for processing data. Offline mode 414 can be used in training mode 412 ( Figure 5 This is used afterward. Camera data 502, which may have already been stored as training images in data structure 510 during training mode, can be provided to feature extraction module 520. Feature extraction module 520 can use a feature extraction algorithm, a descriptor algorithm, or both to extract feature data, which, based on the results of the feature extraction or descriptor algorithm, is provided to data structure 528 and stored in data structure 528.

[0120] As used herein, the term "feature" refers to two-dimensional (2D) data obtained from identifying one or more points (particularly keypoints or points of interest) in a two-dimensional image. Features can be identified and extracted from an image using feature detector algorithms. Depending on the specific autonomous machine and application, any suitable feature detector algorithm available to those skilled in the art can be used. In some embodiments, each specific feature refers to only one point or point of interest in the image or 3DPC. Features can be stored as feature data containing coordinates defined relative to an image frame. In some embodiments, feature data may also include descriptors applied to, associated with, or corresponding to the feature. The term "feature data" refers to a data structure representing a feature and may include two-dimensional locations and multi-dimensional descriptors (e.g., two-dimensional or three-dimensional).

[0121] Key points for identifying features can be extracted from various objects in an image. In some embodiments, objects can be permanent, temporary, or both. In some embodiments, objects can be natural, man-made, or both. An example of a permanent feature is the corner of a house. An example of a natural feature is the edge of a tree trunk. Some examples of temporary and man-made features include tree stumps in the ground and targets on trees. Man-made features can be temporarily placed and used to increase feature density within a work area (e.g., to improve low-quality sections of a 3DPC). Man-made features can be powered, for example, and may include a light emitter that emits visible or invisible light detectable by a camera. Man-made features can be unpowered, for example, and may include visible or invisible patterns detectable by a camera. Some man-made features can be permanently placed. As used herein, the term "invisible" refers to light that emits or reflects wavelengths invisible to the human eye, but may emit or reflect wavelengths visible to a camera, such as an infrared camera on an autonomous machine.

[0122] As used herein, the term "descriptor" refers to two-dimensional data generated by a descriptor algorithm. A descriptor describes features within the context of an image. In some embodiments, a descriptor may describe pixel values, image gradients, scale-space information, or other data near or around a feature in the image. For example, a descriptor may include an orientation vector of a feature or may include image patches. Depending on the specific autonomous machine or application, any appropriate descriptor algorithm available to those skilled in the art for providing context for features in an image may be used. Descriptors may be stored as part of the feature data.

[0123] The techniques described herein for feature detection, descriptor, feature matching, or visual map construction may include or utilize algorithms such as Scale Invariant Feature Transform (SIFT), Accelerated Robust Features (SURF), Oriented Fast and Rotated Short Range (ORB), KAZE, Accelerated KAZE (AKAZE), linear feature tracking, camera fusion, loop closure, incremental structures from motion, or other suitable algorithms. Such algorithms may, for example, provide one or more features and descriptors to the feature matching module 522 and the visual map construction module 524 described below.

[0124] The output of the feature extraction module 520 and / or the feature data stored in the data structure 528 can be provided to the feature matching module 522. The feature matching module 522 can utilize a feature matching algorithm to match features identified in different training images. Different images may have different lighting around the same physical keypoints, which can lead to some differences in descriptors of the same features. Features with similarity higher than a threshold can be identified as the same feature.

[0125] Depending on the specific autonomous machine and application, any suitable feature matching algorithm available to a person skilled in the art can be used. Non-limiting examples of suitable algorithms include brute-force algorithms, Approximate Nearest Neighbor (ANN) algorithms, and a Fast Library for Approximate Nearest Neighbor (FLANN) algorithms. A brute-force algorithm matches features by selecting one feature and checking if all other features match. Feature matching module 522 can provide matching data based on the results of the feature matching algorithm and store it in data structure 530.

[0126] The output of feature matching module 522 and / or the matching data stored in data structure 530 can be provided to visual map building module 524. Visual map building module 524 can utilize map building techniques, such as... Figure 15 The method shown is used to create 3DPCs. Generally, the techniques for generating 3DPCs using vision-based sensors described herein can be described as Structure from Motion (SfM) or Simultaneous Localization and Mapping (SLAM) techniques, either of which can be used in various embodiments of this disclosure, for example, depending on the specific autonomous machine and application.

[0127] As used herein, the terms "3D point cloud," "3D point cloud," or "3DPC" are data structures that represent or contain three-dimensional geometric points corresponding to features extracted from an image. A 3DPC can be associated with various attributes such as pose. In some embodiments, the geometric points and pose can be defined in a coordinate system that may or may not be arbitrary. In some embodiments, a 3DPC may or may not be associated with real-world scale, orientation, or both, for example, until a map registration process has been performed. A 3DPC can be generated based on feature-matching data. Graphs or visual maps can be generated based on 3DPCs to provide a human-visible representation of the 3DPC.

[0128] In some embodiments, the visual map building module 524 can establish a correspondence between 3D points and 2D features even if a 2D-to-2D correspondence from the feature matching module 522 has not yet been established. In other words, the visual map building module 524 may not require all features to be matched before the visual map building process begins.

[0129] Other data can be associated with points in the 3DPC. Non-limiting examples of data that can be associated with each point in the 3DPC include: one or more images, one or more descriptors, one or more poses, positional uncertainties, and pose uncertainties of one or more poses. The 3DPC and associated data can be provided to and stored in data structure 532.

[0130] In some embodiments, the associated data may include one or more poses determined by and associated with points in the 3DPC as pose data, which can describe the position and / or orientation of the system's platform or some other components of the system when features associated with the 3DPC are observed. For example, the position and orientation of an autonomous machine during image recording can be determined based on the positions of different points in the 3DPC and the positions of corresponding features in the recorded image. Position and orientation, or pose, can also be determined directly during point cloud generation. The pose representing position, orientation, or both types of data can be used by a navigation system for boundary determination or attitude correction.

[0131] The output of the visual map building module 524 and / or 3DPC, along with associated data stored in data structure 532, can be provided to the map registration module 526. The associated data may include multiple 6DOF poses, 3DPCs, and multiple boundary points from the visual map or navigation map. Optionally, non-visual data such as GPS data, IMU data, and ranging data from data structure 512 can also be provided to the map registration module 526. The map registration module 526 can determine and provide pose data based on the registration map, which can be provided to and used by the navigation system 404. In some embodiments, pose data is provided to the navigation system 404 by the map registration module 526. The pose data may be estimated visual pose data. The registration map can also be provided to and stored in data structure 534.

[0132] As used herein, the term "registration map" refers to a 3D PC that has been associated with a real-world scale, real-world orientation, or both. In some embodiments, the registration map may be associated with a real-world map or frame of reference. For example, GPS can be used to associate a 3D PC with a real-world map service, such as Google Maps. TM In some embodiments, when using the techniques described herein, a 3DPC can typically be scaled from about 0.5x to about 2x when registered to a real-world map or reference frame. However, the scaling ratio is generally not limited to these ranges.

[0133] As used herein, the term "real world" refers to Earth or other existing frames of reference in the operational area. Non-real world frames of reference can be described as arbitrary frames of reference.

[0134] Figure 10 The diagram illustrates a schematic representation of various data, data structures, and modules of a vision system in an example of online mode 416 for pose estimation. Online mode 416 can be used after training mode 412, offline mode 414, or both. Image data from camera data 502 can be used during operation instead of capturing images for training. Such image data can be described as operational image data that includes the operational images.

[0135] The operational images in camera data 502 can be provided to feature extraction module 520. The same or different algorithms used to extract feature data from training images during offline mode 414 can be used on the operational images in online mode 416.

[0136] Feature data from feature extraction module 520 can be provided to feature matching module 522. Feature matching module 522 can use the same or different algorithms during offline mode 414 to match the feature data from feature extraction module 520 with features in the registration map data from data structure 534. In some embodiments, feature data from data structure 528 can also be used as input to feature matching module 522. Feature matching module 522 can use 2D correspondence, 3D correspondence, correspondence between 2D image location and 2D projection of 3D data, descriptor values, or any combination thereof to match features. Matching data from feature matching module 522 may include 2D or 3D correspondence, which can be provided to pose estimation module 540.

[0137] The pose estimation module 540 can provide an estimated pose, such as a 6-DOF pose, and can be described as a vision-based pose. The vision-based pose data from the pose estimation module 540 can be provided to the navigation system 404, the pose filter 542, the feature matching module 522, or any combination thereof.

[0138] The pose data can be used by the feature matching module 522 to identify features that may be seen in the camera data 502 based on the autonomous machine's estimated pose and the locations where these features might be seen. This information can be used as input to one or more algorithms of the feature matching module 522.

[0139] The attitude filter 542 can use attitude data (e.g., based on previously estimated attitude) to identify which attitudes are possible. The filtered attitude data from the attitude filter 542 can be provided back to the feature matching module 522 to identify features that may be seen in the camera data 502 based on the filtered attitude data and the locations where these features might be seen. This information can be used as input to one or more algorithms of the feature matching module 522.

[0140] In some embodiments, the attitude filter 542 may be derived from an IMU, a wheel encoder, or an optical encoder (e.g., Figure 6-7 The attitude filter 542 can be described as using information from a non-visual sensor (e.g., an inertial navigation system (or INS) containing an inertial measurement unit) to inform which attitudes can be filtered. In some embodiments, the attitude filter 542 can be described as using attitudes from a non-visual sensor (e.g., an inertial navigation system (or INS) containing an inertial measurement unit) to inform which attitudes can be filtered. Figure 6 The navigation system 404 can use, for example, independent attitude filters in the sensor fusion module 430. The resulting outputs or attitude data from different attitude filters can be compared to correct any output or as a redundancy check for any output.

[0141] Additionally, the feature matching module 522 can use feature data from the data structure 528, which may include features and / or descriptors, to filter out feature data from the feature extraction module 520 that is dissimilar to any of the features extracted during the training mode.

[0142] Figure 11 A series of timestamped images 550, 560, 570, and 3DPC580 are shown to illustrate an example of visual map construction. For example, using a feature detector algorithm, keypoints in images 550, 560, and 570 are identified as two-dimensional features 552, 562, and 572, respectively.

[0143] Each feature 552, 562, and 572 is extracted, and a descriptor algorithm can be applied to generate multidimensional descriptors 554, 564, and 574, respectively, associated with features 552, 562, and 572. Descriptors 554, 564, and 574 are shown as circles surrounding the corresponding features 552, 562, and 572. Features 552, 562, and 572, and descriptors 554, 564, and 574 can be included in the feature data.

[0144] During feature matching, a feature matching algorithm can be used to determine whether features 552, 562, and 572 are sufficiently similar based on descriptors 554, 564, and 574. Features can then be matched within the matching data.

[0145] During visual map construction, map-building techniques can be applied to feature data and matching data to identify 3D point 582 in 3DPC580 corresponding to features 552, 562, and 572. Each point in 3DPC580 can be determined in a similar manner.

[0146] exist Figure 12 In the diagram, the 3DPC600 is shown with attitude point 602. In the illustration, the attitude point is drawn as a red dot that appears to form a path, the visible outline of which is roughly drawn using a white dashed line. This path can be described as a path around the boundary of the work area. Points corresponding to feature locations are drawn as black dots. During visual map construction, attitude point 602 can be determined together with points corresponding to feature locations. Each attitude point 602 corresponds to the estimated attitude of the camera used to record an image. Each attitude point 602 can be included in the attitude data provided to the navigation system, which can be used for boundary determination or attitude correction.

[0147] The boundary can be defined using a linear fit or a curve fit of attitude point 602. The boundary can also be defined relative to a linear fit, a curve fit, or attitude point 602. For example, the boundary can be defined as one foot outside of a linear fit, a curve fit, or attitude point 602.

[0148] The quality of the 3DPC600 can be evaluated. Quality levels or parameters can also be assigned to individual parts of the 3DPC600. The quality level of the 3DPC can be evaluated based on various parameters, such as at least one of the following: the number of reconstructed poses, the number of reconstructed points, reprojection error, point triangulation uncertainty, and reconstructed pose uncertainty.

[0149] exist Figure 13 In the diagram, 3DPC 610 is shown as having attitude point 612 and a low-quality portion 604 of the 3DPC. An autonomous machine can be guided along a path represented by attitude point 612. The path can be described as a path around the boundary of the work area. One or more portions 604 of 3DPC 600 can be identified or determined as low-quality portions 604. For example, a portion of the 3DPC can be determined to have a quality level below a quality threshold. This quality level can be based on, for example, an uncertainty value associated with a point, uncertainty in the pose corresponding to those points, or a low density of points. For example, when guiding an autonomous machine along a path during training mode, certain areas of the work area may have very few visible keypoints for one or more camera-recognized features (e.g., near an open field), or the path may be so close to obstacles that keypoints just above or behind the obstacles are not visible on the path (e.g., near a fence, where limited vertical field of view obstructs the view of trees behind or above the fence).

[0150] It is expected that the quality level of this portion of the 3DPC can be improved. The coordinates or points associated with the low-quality portion can be provided to the navigation system. The navigation system can guide the autonomous machine through the work area to record additional training images, for example, along a different or secondary path than the original expected boundary path, which is likely to record additional images of key points that may be in the low-quality portion. The navigation system can guide the autonomous machine along the secondary path, for example, during training mode or online mode. In other words, the autonomous machine can be guided to record images of areas in the work area associated with the low-quality portion of the 3DPC, which can be used to improve the quality of said portion of the 3DPC or "fill" said portion of the 3DPC.

[0151] Figure 14 The 3DPC620 is shown, which is represented by the 3DPC610 ( Figure 13 The same working area. However, the autonomous machine is guided along the path represented by attitude point 622 to record images used to generate 3DPC620. As shown, 3DPC622 does not include low-quality portions. This path can be described as a secondary path. At attitude point 612 ( Figure 13 The secondary path is defined within the boundary or original path represented by 622. A secondary path can be described as traversing (traversing) the interior of the work area. A secondary path may include more turns or "zigzag" paths through the work area to capture more viewpoints via one or more cameras on the autonomous machine. Any type of path modification, such as random, semi-random, or planned path changes, can be made to determine the secondary path. When the secondary path represented by attitude point 622 is used to fill 3DPC 620, the original path represented by attitude point 612 can still be used as the boundary defining the work area.

[0152] In some embodiments, GPS data (e.g., GPS-RTK data) can be used to help navigate an autonomous machine through areas in a work area associated with low-quality portions of a 3DPC. GPS data can be provided to a sensor fusion module as one of the non-visual sensors. In one example, the autonomous machine can rely more heavily on GPS data when traversing an area associated with a low-quality portion of a 3DPC. When relying more heavily on GPS data, it can be "weighted" more heavily than vision-based data. GPS data can be used for attitude correction or even as the primary non-visual sensor input for sensor fusion. For example, when the autonomous machine is traversing (traversing) a work area containing one or more components that might obstruct the GPS receiver 116 ( Figure 1When an obstacle receives appropriate timing signals from GPS satellites, the autonomous machine can "weight" vision-based data more heavily than GPS data. The autonomous machine can also "weight" vision-based data more heavily, for example, when it is traversing areas within the work zone that are not associated with low-quality portions of the 3DPC.

[0153] Figure 15 The diagram shows a flowchart of an example of the visual map building method used by the visual map building module 524. At the end of the visual map building at 664, the 3DPC can be stored in data structure 532. Typically, the visual map building method may employ the removal of irrelevant points that could confuse various map building algorithms. For example, before using certain map building algorithms to generate the 3DPC, points associated with high uncertainty values ​​or weak matches can be removed from the data. Multiple 6DOF poses and multiple boundary points determined based on the multiple 6DOF poses can be stored in data structure 532.

[0154] In step 650, weak matches from the matching data in the data structure can be rejected. Specifically, matches in the matching data below a matching threshold can be rejected and not used to generate 3DPC. A weak match can be defined as two features with similar descriptors that make them match using a matching algorithm. However, this feature can be located in different positions within the job area. Any suitable algorithm available to those skilled in the art can be used to filter out such weak matches. Some algorithms provide feedback on correlation or ratios. For example, the result of a ratio test can represent the probability of a good match. A threshold can be used to determine whether the result of the ratio test does not meet or exceeds such a matching threshold. One or more of these tests can be used to determine whether a match is weak. In some embodiments, a weak match can be determined by stratified testing and determining whether the total probability does not meet or exceeds a matching threshold.

[0155] In step 652, a portion of the 3DPC can be initialized using data based on the first and second training images (e.g., any image pair). Specifically, the portion of the 3DPC can be initialized using feature data corresponding to the first and second training images. The feature data may include features and descriptors. The training images can be selected to be sufficiently spaced apart in distance or time within the work area, which can be considered as alternative distances as the autonomous machine traverses the work area. The training images can also be selected such that a sufficient number of features are visible in both images. In some embodiments, the first two training images are selected such that one, two, three, or more features are shared between the training images, such that the number of shared features exceeds a threshold number of features, and the training images are not immediately subsequent recorded images, such that the training images are spaced apart by a certain threshold number in distance, time, or number of images (e.g., one, two, three, or more images recorded during the interval).

[0156] In step 654, a third training image with overlapping correspondences with a portion of the 3DPC is selected. Specifically, a third training image with overlapping correspondences with a portion of the 3DPC can be selected.

[0157] A third training image can be selected to validate points identified in the existing partial 3DPC based on the first two images. Overlap correspondence can be evaluated to determine if the third training image has a strong connection to the existing partial 3DPC. In other words, a third training image can be selected such that some features are shared between the images in quantities exceeding a threshold, and the third training image is spaced apart from the first and second training images by a certain distance, time, or number of images at a certain threshold.

[0158] Typically, if each of the three images has one or more common points (e.g., a sufficient number of common points), these points can be matched in three-dimensional space. Furthermore, the camera pose can generally be determined under these same conditions. In step 656, the pose of the camera used to capture the third image can be estimated based on a portion of the 3DPC.

[0159] In step 658, a new partial 3DPC can be determined based on the feature data of the third training image and the partial 3DPC. Specifically, the position of any new feature relative to the partial 3DPC can be estimated using matching data associated with the third training image and matching data associated with the first and second training images.

[0160] In step 660, a graph optimizer may be applied to a portion of the 3DPC and the training images used. Specifically, the portion of the 3DPC may be updated using the graph optimizer to refine the estimated location of features or subsets of features, refine the reacquired camera pose or subset of camera pose, or refine both feature location and pose.

[0161] Graph optimizers, also known as bundle adjustment, are often referred to as a specific application of graph optimization. Graph optimizers can be based on mathematical data science techniques, similar to least squares regression, but applied to more tightly connected data structures. Map points constrain the graph, connecting points in the 3DPC to the two-dimensional image space. Connections between 3D and 2D points form the edges of the graph. The optimization problem can assume some information is imperfect and that the most probable locations of graph nodes can be determined or estimated based on all available information (e.g., the coordinates of points in the 3DPC and vision-based pose). In other words, the graph optimizer recognizes that the generated 3DPC may be "noisy" and finds the "best-fit" 3DPC based on all available information.

[0162] If additional unused training images are available in step 662, then in step 654, additional unused training images with overlapping correspondences to a portion of the 3DPC can be selected. In steps 656 and 658, the pose and position in each additional training image can be estimated. In step 660, the graphics optimizer can be run on the estimated positions of the features and the unused training images.

[0163] If no unused training images are available in step 662, a partial 3DPC can represent the complete 3DPC. The visual map construction method can end in step 664, and the 3DPC and pose data are stored in data structure 532.

[0164] Figure 16 The diagram illustrates a schematic representation of an autonomous machine navigation method 700 for training. In step 702, in training mode, the autonomous machine can be guided along a work area, for example, along a desired boundary path of the work area. While guiding the autonomous machine in training mode, training images can be captured by a vision system on the autonomous machine. The autonomous machine can be guided manually or autonomously along the desired boundary path. The machine can also be guided along a path offset from the desired boundary path (e.g., a predetermined distance).

[0165] In step 704, in offline mode, a 3DPC representing the work area and / or the area outside or around the work area can be generated. For example, the 3DPC may also include points outside or even outside the work area's boundaries (e.g., when the boundaries are confined within the work area). The 3DPC can be generated based on feature data containing two-dimensional features extracted from training images. Alternatively, the 3DPC can be generated based on matching data related to features from feature data from different training images.

[0166] In step 706, pose data can be generated and associated with points in a 3DPC representing the pose of the autonomous machine. The pose data can be described as vision-based pose data. The pose data can include position and orientation representing the location of the camera or autonomous machine during training mode. In some embodiments, the pose data includes at least a three-dimensional position representing the pose of the autonomous machine during training mode. The pose of a forward-facing camera can be used to estimate the position of the autonomous machine.

[0167] In step 708, the autonomous machine's navigation system can be used to determine boundaries using non-visual sensor data and attitude data associated with the 3DPC. The boundaries can be used, for example, for subsequent navigation of the autonomous machine within the work area during online mode. Non-visual sensor data can be obtained from, for example, an inertial measurement unit. Vision-based attitude data associated with the 3DPC can be used to estimate or correct the boundaries.

[0168] Figure 17 The diagram illustrates a schematic representation of an autonomous machine navigation method 800 for operation. In step 802, in online mode, the autonomous machine's navigation system can be used to determine the autonomous machine's attitude based on non-visual sensor data. For example, the sensor data can be based on the output of an inertial measurement unit.

[0169] In step 804, the autonomous machine's vision system can be used to determine vision-based pose data (visual pose data) based on the received operational images and a 3DPC generated based on feature data extracted from training images. The vision-based pose data can be determined independently of non-visual sensor data. In some embodiments, the vision-based pose data can be determined at least in part based on feedback from visual pose estimation or visual pose filtering.

[0170] In step 806, the navigation system can update the predetermined attitude based on the visual attitude data. The visual attitude data can be updated at a rate slower than the rate at which the navigation system updates its attitude. That is, the attitude can be determined once or more from the visual attitude data without input or correction.

[0171] In step 808, the navigation system can navigate the autonomous machine within the boundaries of the work area based on the updated attitude. For example, the navigation system can be used to provide propulsion commands to the autonomous machine's propulsion system.

[0172] Any suitable technique can be used to train an autonomous machine for navigation. In one or more embodiments described herein, the training method for the autonomous machine may include one, two, or more distinct phases. During training, the machine may also switch between different modes, such as an offline mode, between different phases of the training mode. Furthermore, before each phase of the training mode begins, the autonomous machine may perform a battery check, which ensures that the machine is able to perform the tasks required during each phase.

[0173] Figure 18 An example of different stages used in training method 820 is shown. Specifically, 3DPC and bounds can be trained in different stages, for example, with Figure 16 Compared to training method 700, training method 700 can train the 3DPC and boundaries in a single stage. Training method 820 may include a patrol stage in step 822, wherein the work area is patrolled by guiding the autonomous machine within the work area. Images and other sensor data may be recorded during the patrol stage.

[0174] Training method 820 may also include an offline phase in step 824, in which the autonomous machine generates a 3D PC while docked at a base, for example, in offline mode. Point clouds can be generated using images and other sensor data recorded during the exploration phase.

[0175] Furthermore, training method 820 may include the mapping phase in step 826, in which the autonomous machine is guided within the work area according to desired boundaries. The machine may be guided manually, which could include being pushed or driven by a user or remotely controlled. During the traversal phase, images and other sensor data may be recorded. For example, sensor fusion data can be used to determine the autonomous machine's position along the path.

[0176] Once the boundaries have been drawn, training method 820 may include the map generation phase in step 828, in which the autonomous machine generates a navigation map. The map generation phase in step 828 may include generating a navigation map based on sensor fusion data recorded during the drawing phase. The navigation map may include some or all of the boundaries trained by the user during the drawing phase.

[0177] The map generation phase in step 828 may include generating representations of one or more paths traversed by the autonomous machine during the mapping phase. For example, the representation of one or more paths may be a visual representation displayed to the user on a user interface device. In some embodiments, the user interface device may be coupled to the autonomous machine for use in the traversal or mapping phase. An example of a user interface device is a smartphone, which may be physically docked or coupled to the autonomous machine in a user-visible location, or may be operatively connected to the autonomous machine via a wireless or wired connection for remote operation of the machine.

[0178] In some embodiments, one or more processes of training method 820 can be repeated even after the navigation map has been tested and used for autonomous navigation. For example, a user may wish to change one or more boundaries in response to physical changes in the work area (e.g., adding exclusion zones by adding flower beds to a yard) or changes in preferences over time. In such embodiments, the autonomous machine can be configured to repeat one or more of the roving phase in step 822, the offline phase in step 824, the drawing phase in step 826, and the map generation phase in step 828. For example, in some embodiments, if it is not necessary to update or regenerate the 3DPC, only the drawing phase in step 826 and the map generation phase in step 828 can be repeated.

[0179] Figure 19 An example of a patrol phase 822 that can be used in the overall training method 820 is shown. Patrol phase 822 may include connecting the autonomous machine to a user interface device in step 832. Patrol phase 822 may also include instructing the user to patrol various portions of the work area. As shown, patrol phase 822 may include displaying user instructions regarding the boundaries of the work area in step 834. The boundaries of the work area may correspond to the perimeter of the work area, such as the outer perimeter of the work area. This process allows the user to define the extent of the work area.

[0180] The inspection phase 822 may also include displaying user instructions within the inspection work area in step 836. Inspecting the interior of the work area can provide images that can be processed to identify features used to construct the 3DPC. In one example, inspecting the interior of the work area may correspond to guiding an autonomous machine according to a raster pattern to roughly cover various areas of the work area. The raster pattern may not completely cover the entire work area.

[0181] The inspection phase 822 may include recording a set of images during the inspection in step 838, for example, when guiding an autonomous machine in a work area. The machine may be guided according to user instructions. The recorded set of images may be processed to identify features. Non-visual sensor data, such as wheel encoder data or IMU data, may also be recorded during the travel phase 822.

[0182] In some embodiments, the inspection phase 822 may include inspecting the boundary, inspecting the interior, or both the boundary and the interior. For example, an inspection of the interior may be requested only if the features identified in the images recorded during the inspection of the perimeter are insufficient to construct a robust 3DPC for autonomous navigation.

[0183] In other embodiments, the perimeter and interior can be inspected without considering the results of the perimeter inspection. Multiple sets of images of the boundaries and interior can be recorded in the same or different phases. For example, two sets of images can be recorded without an offline phase between them. Each set of images may include images captured by one or more vision systems of the autonomous machine.

[0184] Figure 20 A specific example of method 870, which can be used to perform at least a portion of method 820, is shown. Method 870 may include generating a 3DPC in step 824, which may be performed after the patrol phase. In step 844, the 3DPC may be analyzed and it may be determined whether the 3DPC includes any low-quality parts. If one or more low-quality parts are insufficient or unacceptable for 3DPC-based navigation, method 870 may include performing an autonomous or manually supplemental training run to improve the feature density in the identified low-quality parts.

[0185] In some cases, if the quality level of the 3DPC does not meet a quality threshold, the 3DPC may be insufficient for navigation. The presence of one, two, or more low-quality segments may be sufficient to determine that the 3DPC's quality level is insufficient. In some embodiments, method 870 may include determining that the 3DPC is sufficient for navigation even in the presence of one or more low-quality segments. For example, a lawnmower may use non-low-quality segments of the 3DPC near the low-quality segments for position correction or updates during navigation. Furthermore, the autonomous machine may use images from other training modes or operations to periodically improve the 3DPC without performing dedicated supplemental training runs.

[0186] Method 870 may include displaying a user instruction to place the marker in step 846. In some embodiments, the user instruction may be displayed on the user's smartphone. The marker or target may be identified by sensor data. For example, the marker may be visible to a vision system, and the vision system may identify one or more artificial features of the marker for use in generating a 3DPC. The marker may be placed temporarily or permanently in the work area for future navigation.

[0187] In step 848, the autonomous machine can be guided along the identified low-quality sections. The machine can be guided autonomously using sensor fusion to navigate non-low-quality sections of the work area, or it can be manually guided by a user, which can be done physically or using remote control. In step 850, while guiding the machine along the work area, a new set of inspection images can be recorded to capture features identified in the low-quality sections; these features can be man-made.

[0188] Having recorded a new set of inspection images, the machine can return to the dock for offline mode. In step 852, during offline mode, method 870 may include regenerating the 3DPC based on the new set of inspection images. If necessary, the process of remedying low-quality portions of the 3DPC can be repeated.

[0189] In some embodiments, a new set of inspection images may be recorded repeatedly or periodically, or the 3DPC may be regenerated in addition to detecting low-quality areas. A repeatedly recorded set of new inspection images can be used to adjust the 3DPC and navigation map to adapt to seasonal or other changes in the work area. For example, a new set of inspection images may be set to be recorded four times a year or once per local season.

[0190] Method 870 may include the execution process in step 826 to define specific boundaries within a work area. In some embodiments, defining specific boundaries may be performed after determining that the 3DPC is acceptable or sufficient for generating the 3DPC, or after one or more low-quality portions of the 3DPC have been remedied. Method 870 may include displaying user instructions in step 854 to guide the autonomous machine to perform boundary training. The user may select or be instructed to train various types of boundaries, such as exclusion zones, transfer zones, or fenced zones. One or more of these boundaries may be trained by guiding the autonomous machine along one or more paths representing these boundaries. The autonomous machine may be manually guided by the user, which may be done physically or using remote control.

[0191] When the machine is guided, in step 856, the drawn image and other sensor fusion data may be recorded. Specifically, the machine may record sensor fusion data, which can be used to determine the position using non-visual sensor data, and the position can be located or corrected using vision system data and 3DPC. Specifically, the position can be located in a coordinate system defined by 3DPC.

[0192] Method 870 may include displaying representations of one or more paths traversed by the machine before defining the relevant boundaries. In one example, a rough shape of the path traversed by the machine may be displayed to the user on a user interface device (e.g., a smartphone) before defining the relevant boundaries. In some embodiments, after the machine traverses the path of each boundary, a visual representation may be compiled and displayed, and the user may confirm that the representation is acceptable before proceeding to train the next boundary.

[0193] Various techniques can be used to compile a rough shape displayed to the user. In some embodiments, the rough shape can be generated based on the original position of the autonomous machine determined by sensor fusion data. The position of the autonomous machine's wheels can be determined by sensor fusion data and can be used to define the rough shape. Specifically, the wheels can be used as vertices of a trapezoidal shape used to "depict" the machine's path. In some embodiments, the original position data can be smoothed for use in generating the representation.

[0194] In one or more embodiments, the visual representation associated with each path may be based on the outer perimeter of the respective path. For example, a user may guide the machine to a corner of the work area and move the machine back and forth to turn it until it covers the edge of the work area near the corner. Instead of showing all the forward and backward movements in the visual representation, the outer perimeter of the machine path is shown as a rough shape.

[0195] Method 870 may include generating a navigation map in step 828 (e.g., in offline mode). The navigation map may define one or more training boundaries. The navigation map may be generated and stored separately from the 3DPC. For example, when sensor fusion data is used to generate boundaries, the coordinate system of the navigation map may be located in the coordinate system of the 3DPC. The navigation map may be generated as a 2D or 3D representation of the boundaries of the work area. The navigation map may be generated during the map generation phase in training mode or during offline mode. In some embodiments, the navigation map including the training boundaries may be displayed to a user via a user interface device. Because, for example, positioning or correction is performed using vision-based sensor data and 3DPC, the training boundaries may appear different to the user than the visual representation of the path.

[0196] The navigation map can be used for the operation of an autonomous machine within a work area. In some embodiments, after generating the navigation map, method 870 may include testing the navigation map before operating the autonomous machine using the navigation map in step 862. For example, the machine may autonomously traverse paths or training boundaries. If the test is successful, the navigation map can be used for the autonomous operation of the machine within the work area, for example, to perform a lawnmowing task within the work area. The 3DPC or boundaries may be retrained or redefined as needed.

[0197] Figure 21 An example of a handle assembly 90 is shown. In some embodiments, a bracket 900 may be attached to the grip portion 902 of the handle assembly 90 and is part of the handle assembly 90. The bracket 900 may be adapted to accommodate and hold a mobile device 119 (e.g., a smartphone) in a direction visible to an operator standing or walking behind the housing (when the handle assembly is in the manual mode position). The mobile device 119 may support a wireless device 117 of the lawnmower 100 ( Figure 1 The communication protocol is compatible, for reasons that will be further described below. Alternatively, the lawnmower 100 and the bracket 900 may include a controller 120 for wired connection (e.g., serial, universal serial bus, etc.) to the lawnmower 100. Figure 1 The device. Regardless of the control interface provided to the operator, he or she can control and manipulate the lawnmower by interacting with controls associated with the handle assembly 90 (e.g., virtual controls on a mobile device).

[0198] For autonomous operation, the boundaries of the work area are trained and stored in the lawnmower 100. While various boundary detection systems are known, as described in more detail below, a lawnmower according to an embodiment of this disclosure can determine the boundaries of the work area by initially undergoing a training procedure or phase. In training mode, the lawnmower is configured in manual mode, wherein the handle assembly can be in the manual mode position.

[0199] The bracket 900 can house a mobile device 119 (e.g., a smartphone) that supports a communication protocol (wired or wireless) compatible with the lawnmower 100's radio 117. For example, the mobile device 119 can support short-range wireless communication via Bluetooth. As further described below, the controller 120 can communicate with the mobile device 119 to provide various control and operator feedback in the lawnmower's training mode.

[0200] To enter training mode, the handle assembly 90 can (if not already in place) first unfold from a first or autonomous mode position or move to a second or manual mode position. After the handle assembly is in place, the mobile device 119 can be placed in the bracket 900 as described above. The operator can then initiate communication between the mobile device 119 and the controller 120. This initiation may include pairing or otherwise connecting the mobile device 119 to the lawnmower 100, enabling the two devices to communicate wirelessly with each other. Although described herein as wireless communication (e.g., Bluetooth), alternative embodiments may again provide wired interconnection. The operator can then launch application-specific software on the mobile device, which presents status information 904 to the operator in training mode. The software may also allow the operator to issue commands during the training process via input provided by virtual buttons 906 displayed on the display screen 908. For example, the application may allow the operator to issue and receive commands and instructions for: entering training mode; starting / stopping recording of data related to crossing boundaries of work areas, exclusion zones, or transfer zones; and when to push or drive the lawnmower along identified boundaries or paths.

[0201] When the operator is ready to initiate training mode, they can use the handle assembly 90 to push the lawnmower to the perimeter of the work area (or the perimeter of the exclusion zone). At this point, training can begin by selecting the appropriate training mode presented on the display screen 166 (e.g., boundary training mode for the work area or exclusion zone, or transfer zone training mode). Then, in boundary training mode, the operator can begin traversing the boundary of the work area.

[0202] During boundary training mode, lawnmower 100 can record boundary-related data as it crosses the boundary. Lawnmower 100 can also (via application software running on mobile device 119) present various status information of the training mode to the operator during crossing / training (see, for example, 904). For example, display screen 908 can plot the lawnmower's area coordinates in real time during boundary recording. Additionally, display screen 908 can present instructions requesting the operator to change (e.g., reduce) the lawnmower speed. Maintaining the lawnmower speed below a threshold during training to ensure the lawnmower captures sufficient data may be important, especially for vision-based systems.

[0203] Such speed-related instructions / feedback can be presented to the operator in text or graphics. For example, feedback and / or other status information can be presented as a quantitative speed indicator (e.g., a speedometer) or speed-related icons or objects (e.g., color-changing icons: green for acceptable speeds, yellow or red for unacceptable speeds). In other embodiments, display 908 can indicate whether a speed change is needed by showing the speedometer reading next to the desired target speed or by showing an "up" or "down" arrow to indicate a recommended faster or slower speed. In other embodiments, the display can provide a simple "pass / fail" indication or provide audible indications (via mobile device 119 or lawnmower / controller) during or after training mode.

[0204] Figure 22 The image shows information about lawnmower 100 ( Figure 1 An exemplary method or process for boundary training of a lawnmower is described below. In some embodiments, method 920 may be part of a drawing phase of the training pattern. It should be noted that this process describes only an exemplary boundary training method. It should be understood that other operations may be required before or after method 920 to allow autonomous operation of the lawnmower. However, these other operations are not specifically described herein. The operator may first train the boundaries of the work area and then continue training the exclusion zone and transfer zone. The method assumes that the lawnmower 100 is located at or near the boundary of the work area, or at or near the boundary of one of the exclusion zones. Method 920 will be described in the context of training the boundaries of the work area, but the method will also be applied with minor variations to the exclusion zone boundaries or transfer zone boundaries or paths.

[0205] Method 920 begins at step 922. Once the lawnmower 100 is positioned along the boundary, a training mode or pattern can be initiated at step 924. Initiating training may include extending the handle (e.g., moving the handle to the manual mode position as described herein), and moving the mobile device 119 ( Figure 1 Placed in bracket 900 ( Figure 21 This involves interaction with software running on mobile devices. Once started, the operator can choose whether the boundary to be trained is the work area boundary, exclusion zone boundary, or transfer zone boundary or path.

[0206] In step 926, when the lawnmower crosses the boundary, the operator can, for example, via the display screen 166 of the mobile device 119 (… Figure 21 The lawnmower 100 can be commanded to record its movement via interaction with the rear wheel 106. Once recording begins, the lawnmower 100 can use various sensors (e.g., GPS, wheel encoders, vision systems, lidar, radar, etc.) to record its path as it is manually guided, pushed, or driven around a boundary. In some embodiments, the lawnmower can communicate with the rear wheel 106. Figure 1 ) provides auxiliary torque to assist the operator when guiding the lawnmower around the boundary. Additionally, the cutting blade 110 ( Figure 1 The cutting blade 110 can be activated or deactivated in training mode. Activating the cutting blade 110 during training can provide feedback on the actual cutting path the mower will take when guiding the lawnmower around a boundary. If activation of the cutting blade 110 is permitted, the cutting blade 110 can be controlled during training via options presented on the display screen 166. This operation may require operator-in-sight control (e.g., on the handle itself or on the display screen 166 of the mobile device 119).

[0207] Because the cutting width of the lawnmower 100 is narrower than that of the housing 102 ( Figure 1 The width of the blade 110 is such that the top of the housing 102 may include visual markings to indicate the cutting width of the lawnmower to the operator. Such markings may be helpful to the operator when the blade 110 is not powered in training mode.

[0208] When the lawnmower 100 is pushed, guided, or driven around the boundary, the lawnmower 100 (e.g., via display screen 166) may optionally indicate training status and / or training alarms to the operator in step 930. For example, the controller 120 may suggest slowing down the ground speed graphically or audibly to improve data capture.

[0209] Once the operator and lawnmower have completed the boundary crossing in step 932 (e.g., moved slightly beyond the original starting point), the operator can indicate (e.g., via the mobile device) that the boundary crossing is complete in step 934. The controller 120 and / or the mobile device 119 can then compile the collected boundary data in step 936 to finally create the boundary path of the drawn work area (or exclusion zone or transfer zone or path) of the transferred area.

[0210] In step 938, the lawnmower can provide feedback (via an onboard display or via mobile device 119) regarding the status of the training process (e.g., the status of boundary recording). For example, upon completion, the lawnmower 100 can provide an indication of successful boundary training on the mobile device (e.g., data meeting predetermined path criteria or multiple criteria) by displaying a status such as a simple "pass / fail" indication in step 938. Path criteria that may affect training success include determining whether the drawn boundary path defines a bounded area (e.g., forming a closed or bounded area or shape). Other path criteria may include determining whether a bottleneck factor exists. For example, a bottleneck factor may exist when the boundary path of the drawn work area is within a threshold distance of the target or another drawn boundary path (e.g., the boundary is too close for the path width to allow the lawnmower to pass easily).

[0211] In step 940, if training is successful, then in step 942, the operator can move the handle assembly to a first or autonomous mode position and command the lawnmower 100 to autonomously traverse the training boundaries of the work area (and / or exclusion zone or transfer zone or path). Assuming the operator determines the training path is acceptable in step 944, the method terminates in step 946. Alternatively, if training is determined to be unsuccessful in step 940, or if the operator finds autonomous operation unacceptable in step 944, the method can return to step 924 and re-execute the training (or a portion thereof). The method can then be repeated for each boundary (including exclusion zones) and transfer zone. In some embodiments, software running on the mobile device 119 may allow the operator to modify, add, and / or delete some or all of the boundaries or a portion thereof during method 920.

[0212] In addition to fenced / exclusion zone training, the lawnmower 100 can also be trained to utilize one or more "Return to Base" transfer zones ("RTB transfer zones") via the handle assembly 90 in manual mode. That is, the lawnmower 100 can also be trained to use which path(s) to return to base 258. Figure 3 Training RTB transfer zones can help assist or accelerate the lawnmower's return to the base, for example, considering complex yards, or otherwise allowing the operator to restrict the lawnmower's preferred return path. Any number of RTB transfer zones can be trained. During autonomous operation, the lawnmower 100 can guide itself to the nearest RTB transfer zone and then return to the base 258 along that path when operation is complete or the lawnmower battery needs to be recharged. Of course, to allow for RTB transfer zone training, the lawnmower / controller can also allow the operator to establish or otherwise train the "home" position of the base 258.

[0213] Before autonomous lawn mowing can occur, the yard or work area can be mapped. Yard mapping involves defining the mowing area (e.g., work area boundaries), defining all exclusion zones, identifying the “original” location of base 258, and optionally identifying transfer zones.

[0214] Figure 23 It shows what can be used as a base 258 ( Figure 3An example of a base for a lawnmower 100. As shown, base 950 includes a housing 952 defining a storage location 960 for the lawnmower 100. A charging connector 958 may be exposed in the storage location 960 for connecting the lawnmower 100 for recharging. Base 950 may include a solar panel 956 coupled to housing 952 and operatively coupled to charging connector 958. Energy generated by solar panel 956 may be used to recharge the lawnmower 100 directly or indirectly. Solar panel 956 may be coupled to charging connector 958, to an optional battery 954, or to both. In some embodiments, solar panel 956 may directly charge lawnmower 100 during the day via charging connector 958. In some embodiments, solar panel 956 may indirectly charge lawnmower 100 during the day by charging battery 954, which is used to charge lawnmower 100 during the day or night via charging connector 958.

[0215] The base 950 can be optionally connected to an external power source, such as a building electrical outlet. The base 950, which has a solar panel 956, a battery 954, or both, can continue to operate even when the external power source is unavailable (e.g., due to power loss).

[0216] In some embodiments, the base 950 is not plugged into an external power source and does not supply power to the boundary wire to define the boundary. Even when the base 950 loses all power supplied from any power source (e.g., solar cell 956 or battery 954) (e.g., when navigation does not depend on the boundary wire powered by the base 950), the lawnmower 100 can continue to operate and navigate. In other embodiments, the base 950 may supply power to the boundary wire and be plugged into an external power source.

[0217] Exemplary embodiments

[0218] While this disclosure is not limited thereto, an understanding of various aspects of this disclosure will be gained through the specific exemplary embodiments provided below. Various modifications to the exemplary embodiments and additional embodiments of this disclosure will become apparent herein.

[0219] In embodiment A1, a method for navigating an autonomous machine includes: determining a current attitude of the autonomous machine based on non-visual attitude data captured by one or more non-visual sensors of the autonomous machine, wherein the attitude represents one or both of the position and orientation of the autonomous machine in one or more boundary-defined work areas; determining visual attitude data based on image data captured by the autonomous machine; and updating the current attitude based on the visual attitude data to correct or position the current attitude, and providing an updated attitude of the autonomous machine in the work area for navigating the autonomous machine in the work area.

[0220] In embodiment A2, a method includes the method according to embodiment A1, wherein the step of determining the visual pose data includes: matching the image data with one or more points in a three-dimensional point cloud (3DPC) representing the work area.

[0221] In embodiment A3, a method includes the method according to embodiment A2, further comprising: capturing training image data using the autonomous machine; generating 3DPC based on: feature data, the feature data comprising two-dimensional features extracted from the training image data; and matching data, the matching data associating features in the feature data from different training images of the training image data.

[0222] In embodiment A4, a method includes the method according to embodiment A3, wherein the step of generating the 3DPC further includes: rejecting matches in the matching data that are below a matching threshold; initializing a portion of the 3DPC using feature data corresponding to a first training image and a second training image; selecting a third training image that has an overlapping correspondence with the portion of the 3DPC; using the third training image, estimating the vision-based pose of the autonomous machine relative to the portion of the 3DPC, the 3DPC using matching data associated with the third training image and matching data associated with the first and second training images; using the third training image, estimating the position of any new feature relative to the portion of the 3DPC, the 3DPC using matching data associated with the third training image and matching data associated with the first and second training images; and updating the portion of the 3DPC using a graphics optimizer on the estimated position of the feature and the training image used.

[0223] In embodiment A5, a method includes the method according to embodiment A4, further comprising: selecting additional unused training images that have overlapping correspondence with a portion of the 3DPC, and continuing to estimate the pose and position of each training image.

[0224] In embodiment A6, a method includes the method according to embodiment A5, further comprising: storing the 3DPC when no unused training images are available.

[0225] In embodiment A7, a method includes the method according to any one of embodiments A3-A6, wherein the 3DPC is generated to define a point in a coordinate system based on an arbitrary reference frame.

[0226] In embodiment A8, a method includes the method according to any one of embodiments A3-A7, further comprising: recording a set of patrol images associated with the autonomous machine traversing one or both of the periphery and interior of the work area to provide at least a portion of the training image data; generating the 3DPC based on the set of patrol images of the training image data; recording a set of rendering images to provide at least a portion of the training image data after generating the 3DPC; and determining the one or more boundaries of the work area based on the set of rendering images and the 3DPC.

[0227] In embodiment A9, a method includes the method according to embodiment A8, wherein the step of recording the set of inspection images includes: recording a first set of inspection images associated with the autonomous machine traversing the periphery of the work area; optionally recording a second set of inspection images associated with the interior of the work area within the periphery of the autonomous machine traversing the work area; and generating the 3DPC based on the first set of inspection images and the second set of inspection images.

[0228] In embodiment A10, a method includes the method according to any one of embodiments A8-A9, further comprising: determining whether the quality level of the 3DPC does not meet a quality threshold before recording the set of drawn images.

[0229] In embodiment A11, a method includes the method according to embodiment A10, further comprising: determining the quality level of the 3DPC based on at least one of the following: the number of reconstructed poses, the number of reconstructed points, reprojection error, point triangulation measurement uncertainty, and reconstructed pose uncertainty.

[0230] In embodiment A12, a method includes the method according to any one of embodiments A10-A11, further comprising: recording a new set of inspection images in the work area when it is determined that the quality level of the 3DPC does not meet a quality threshold; and regenerating the 3DPC based on the new set of inspection images.

[0231] In embodiment A13, a method includes the method according to any one of embodiments A7-A12, further comprising: registering the coordinate system of the 3DPC with the scale and orientation of the real world in the navigation map.

[0232] In embodiment A14, a method includes the method according to embodiment A13, further comprising: autonomously operating the autonomous machine in the work area based on the navigation map.

[0233] In embodiment A15, a method includes the method according to any one of embodiments A13-A14, further comprising: testing the navigation map by navigating the autonomous machine within the work area based on the navigation map before autonomously operating the autonomous machine in the work area.

[0234] In embodiment A16, a method includes the method according to any one of embodiments A3-A15, wherein the 3DPC is generated or regenerated during the offline mode of the autonomous machine while the 3DPC is operatively coupled to a base for charging.

[0235] In embodiment A17, a method includes the method according to embodiment A16, further comprising: performing a battery check before leaving the offline mode.

[0236] In embodiment A18, a method includes the method according to any one of the preceding embodiments A, further comprising: periodically recording a new set of image data.

[0237] In embodiment A19, a method includes the method according to any one of the preceding embodiments A, wherein: the work area is an outdoor area; the autonomous machine is a ground maintenance machine; or the work area is a lawn, and the autonomous machine is a lawn maintenance machine.

[0238] In embodiment A20, a method includes the method according to any one of the preceding embodiments A, wherein the one or more boundaries of the work area are used to define one or more peripheries of the work area, a fenced area in the work area, an exclusion area in the work area, or a transfer area in the work area.

[0239] In embodiment A21, a method includes the method according to any one of the preceding embodiments A, wherein each pose represents one or both of the autonomous machine’s three-dimensional position and three-dimensional orientation.

[0240] In embodiment A22, a method includes the method according to any one of the preceding embodiments A, further comprising: determining the one or more boundaries of the work area based on non-visual pose data and visual pose data for subsequent navigation of the autonomous machine in the work area.

[0241] In embodiment A23, a method includes the method according to any one of the preceding embodiments A, wherein the step of determining the current pose of the autonomous machine based on non-visual pose data is repeated at a first rate, and the step of updating the current pose based on visual pose data is repeated at a second rate slower than the first rate.

[0242] In embodiment A24, a method includes the method according to any one of the preceding embodiments A, wherein the non-visual attitude data includes one or both of inertial measurement data and wheel coding data.

[0243] In embodiment A25, a method includes the method according to any one of embodiments A2-A24, further comprising: associating a point in the 3DPC with at least one of the following: one or more images, one or more descriptors, one or more poses, positional uncertainty, and pose uncertainty of one or more poses.

[0244] In embodiment A26, a method includes the method according to any one of embodiments A2-A25, wherein the feature data includes two-dimensional location and multi-dimensional descriptor.

[0245] In embodiment A27, a method includes the method according to any one of the preceding embodiments A, wherein the step of determining visual pose data is based at least in part on feedback from visual pose estimation or visual pose filtering.

[0246] In embodiment B1, a method for training autonomous machine navigation includes:

[0247] During the patrol phase of the training mode, the autonomous machine is guided along at least one of the perimeter or interior of the work area to record a first set of patrol images associated with the perimeter or a second set of patrol images associated with the interior; during the offline mode, a 3D point cloud (3DPC) is generated based on at least one of the first set of patrol images and the second set of patrol images; and during the rendering phase of the training mode, the autonomous machine is guided along one or more paths to record sensor fusion data, thereby defining one or more boundaries of the work area in the navigation map.

[0248] In embodiment B2, a method includes the method according to embodiment B1, wherein during the drawing phase of the training mode, the step of guiding the autonomous machine along one or more paths includes: evaluating at least one of the one or more boundaries defined based on sensor fusion data; determining whether the at least one boundary satisfies a path criterion; and displaying the status of the drawing phase based on whether the at least one boundary satisfies the path criterion.

[0249] In embodiment B3, a method includes the method according to the aforementioned embodiment B2, wherein the step of displaying the state of the drawing phase occurs during the crossing of the boundary of the work area.

[0250] In embodiment B4, a method includes the method according to any one of embodiments B2-B3, wherein the step of determining whether the at least one boundary satisfies a path criterion includes: determining whether the at least one boundary defines a bounded region.

[0251] In embodiment B5, a method includes the method according to any one of the preceding embodiments B, further comprising:

[0252] The handle assembly connected to the housing of the autonomous machine is deployed from a first position to a second position; and a mobile computer including a user interface is placed on a bracket connected to the handle assembly for the training mode.

[0253] In embodiment B6, a method includes the method according to embodiment B5, further comprising: returning the handle assembly to the first position; and guiding the autonomous machine to autonomously traverse the boundary of the work area.

[0254] In embodiment B7, a method includes the method according to any one of the preceding embodiments B, further comprising: if it is determined that the quality level of the 3DPC does not meet a quality threshold, guiding the autonomous machine to record a new set of inspection images of one or more partitions of the work area, the one or more partitions of the work area being associated with one or more low-quality portions of the 3DPC; and regenerating the 3DPC based on the new set of inspection images during the offline mode.

[0255] In embodiment B8, a method includes the method according to embodiment B7, further comprising: deploying one or more artificial features along one or more partitions of the work area, the partitions of the work area being associated with one or more low-quality portions of the 3DPC, prior to guiding the autonomous machine to record the set of new inspection images.

[0256] In embodiment B9, a method includes the method according to any one of the preceding embodiments B, further comprising: displaying a representation of the one or more paths to a user before defining the one or more boundaries in the navigation map.

[0257] In embodiment B10, a method includes the method according to embodiment B9, wherein the representation associated with each path is based on the outer boundary of the respective path.

[0258] In embodiment B11, a method includes the method according to any one of the preceding embodiments B, further comprising: operably connecting a user interface device to the autonomous machine for use in the inspection phase or the drawing phase.

[0259] In embodiment B12, a method further includes the method according to embodiment B11, further comprising: initiating communication between the user interface device and an electronic controller associated with the autonomous machine; and entering the training mode of the autonomous machine through interaction with the user interface device.

[0260] In embodiment B13, a method includes the method described in any one of embodiments B above, further comprising: displaying instructions to a user for manually guiding the autonomous machine along the periphery, the interior, or the periphery and the interior of the work area during the patrol or drawing phase of the training mode.

[0261] In embodiment B14, a method includes the method according to any one of the preceding embodiments B, wherein the one or more boundaries are used to define one or more peripheries of the work area, a restriction area in the work area, an exclusion area in the work area, or a transfer area in the work area.

[0262] In embodiment C1, an autonomous machine is adapted to perform the method according to any one of embodiments A or B.

[0263] In embodiment C2, a machine includes the machine according to embodiment C1, further comprising: a housing coupled to a maintenance tool; a set of wheels supporting the housing on the ground; a propulsion controller operatively coupled to the set of wheels; a vision system including at least one camera adapted to capture image data; and a navigation system operatively coupled to the vision system and the propulsion controller, the navigation system being adapted to guide the autonomous machine within the work area.

[0264] In embodiment C3, a machine includes the machine according to embodiment C2, wherein the propulsion controller is adapted to independently control the speed and rotation direction of the wheels, thereby controlling the speed and direction of the housing on the ground.

[0265] In embodiment C4, a machine includes the machine according to any one of embodiments C2-C3, wherein the at least one camera adapted to capture image data provides a total horizontal field of view of at least 90 degrees around the autonomous machine.

[0266] In embodiment D1, a method for autonomous machine navigation includes: generating a three-dimensional point cloud representing at least one work area based on: feature data, the feature data including two-dimensional features extracted from training image data, and matching data that associates features in the feature data from different training images; generating pose data associated with points in the three-dimensional point cloud, the pose data representing the pose of the autonomous machine; and using the pose data to determine boundaries for subsequent navigation of the autonomous machine in the work area.

[0267] In embodiment D2, a method includes the method according to embodiment D1, wherein the step of determining the boundary is based on non-visual sensor data and the attitude data.

[0268] In embodiment D3, a method includes the method according to any one of the preceding embodiments D, wherein the pose data includes at least a three-dimensional position representing the pose of the autonomous machine during training mode.

[0269] In embodiment E1, the autonomous machine includes: a housing coupled to a maintenance tool; a set of wheels supporting the housing on the ground; a propulsion controller operatively coupled to the set of wheels, wherein the propulsion controller is adapted to independently control the speed and rotation direction of the wheels, thereby controlling the speed and direction of the housing on the ground; a vision system including at least one camera adapted to record training images and a controller adapted to: generate a three-dimensional point cloud representing at least one work area based on feature data containing two-dimensional features extracted from the training images and feature-related matching data from the feature data of different training images; and generate attitude data associated with points in the three-dimensional point cloud, the attitude data representing the attitude of the autonomous machine; and a navigation system operatively coupled to the vision system and the propulsion controller, the navigation system being adapted to: guide the autonomous machine in the work area to record training images; and determine boundaries using the non-visual sensor data and the attitude data for subsequent navigation of the autonomous machine in the work area.

[0270] In embodiment E2, the autonomous machine includes: a housing coupled to a maintenance tool; a set of wheels supporting the housing on the ground; a propulsion controller operatively coupled to the set of wheels, wherein the propulsion controller is adapted to independently control the speed and rotation direction of the wheels, thereby controlling the speed and direction of the housing on the ground; a vision system including at least one camera adapted to record images, and a controller adapted to provide visual pose data based on received operational images and a three-dimensional point cloud generated based on feature data extracted from training images; and a navigation system operatively coupled to the vision system and the propulsion controller, the navigation system being adapted to: determine the pose based on the non-visual sensor data; update the pose based on the visual pose data; and command the propulsion controller based on the updated pose.

[0271] In embodiment E3, the machine includes the machine according to any one of the preceding embodiments E, wherein the non-visual sensor data includes at least one of inertial measurement data and wheel coding data.

[0272] In embodiment F1, an autonomous machine includes: a housing coupled to a maintenance tool; a set of wheels supporting the housing on the ground; a propulsion controller operatively coupled to the set of wheels; a vision system including at least one camera configured to record inspection images and a controller configured to perform the following operations: during an inspection phase in training mode, recording at least one first set of inspection images associated with the periphery of the work area or a second set of inspection images associated with the interior of the work area when guiding the autonomous machine along at least one of the first and second sets of inspection images; generating a 3D point cloud based on at least one of the first and second sets of inspection images during an offline mode; and recording sensor fusion data of the work area as the autonomous machine traverses one or more paths during a rendering phase in training mode; and a navigation system operatively coupled to the vision system and the propulsion controller, the navigation system including a controller configured to: determine a navigation map of the work area based on the sensor fusion data recorded along the one or more paths, the navigation map representing one or more boundaries; and guide the autonomous machine in the work area based on the navigation map.

[0273] In embodiment F2, a machine includes the machine according to embodiment F1, wherein the propulsion controller is configured to independently control the speed and rotation direction of each of the set of wheels, thereby controlling the speed and direction of the housing on the ground.

[0274] In embodiment F3, a machine includes the machine according to any one of the preceding embodiments F, wherein the controller of the navigation system is further configured to display one or more path representations to a user via a user interface device before defining one or more boundaries in a navigation map.

[0275] In embodiment F4, a machine includes the machine according to any one of the preceding embodiments F, wherein the controller of the vision system is further configured to record a new set of inspection images of one or more areas in a work area associated with one or more low-quality portions of the 3D point cloud if it is determined that the quality level of the 3D point cloud does not meet a quality threshold.

[0276] In embodiment F5, a machine includes the machine according to any one of the preceding embodiments F, wherein the controller of the navigation system is further configured to test the navigation map by autonomous operation of the autonomous machine within the work area based on the navigation map.

[0277] In embodiment F6, a machine includes the machine according to any one of the preceding embodiments F, wherein the controller of the navigation system is configured to be operatively connected to a user interface device for use in a patrol phase or a drawing phase.

[0278] Therefore, this document discloses various embodiments of autonomous machine navigation and training using vision systems. Although reference is made to the accompanying drawings, which form part of this disclosure, it will be understood by at least one person skilled in the art that various modifications and variations of the embodiments described herein are within the scope of this disclosure, or do not depart from it. For example, aspects of the embodiments described herein can be combined with each other in various ways. Therefore, it should be understood that the claimed invention can be practiced in ways other than those expressly described herein within the scope of the appended claims.

[0279] All references and publications cited herein are expressly incorporated into this disclosure by reference unless they may directly conflict with this disclosure.

[0280] Unless otherwise stated, all scientific and technical terms used herein have their common meaning in the art. The definitions provided herein are for the purpose of facilitating understanding of certain terms used frequently herein and are not intended to limit the scope of this disclosure.

[0281] The numerical range described by the endpoints includes all numerical values ​​contained within that range (e.g., 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.80, 4, and 5) and any range within that range. In this document, the terms "up to" or "not greater than" a number (e.g., up to 50) include that number (e.g., 50), and the term "not less than" a number (e.g., not less than 5) includes that number (e.g., 5).

[0282] The term “connection” or “link” refers to components being directly (in direct contact with each other) or indirectly (with one or more components between and attached to these two components). Either term may be modified by “operationally” and “operably”, which are used interchangeably to describe a “connection” or “link” configured to allow components to interact to perform at least some functions (e.g., a propulsion controller is operably connected to a motor drive to electrically control the operation of a motor).

[0283] References to "an embodiment," "an embodiment," "some embodiments," or "a number of embodiments," etc., mean that a particular feature, configuration, composition, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Therefore, such phrases appearing in various places throughout the text do not necessarily refer to the same embodiment of this disclosure. Furthermore, specific features, constructions, compositions, or characteristics may be combined in any suitable manner in one or more embodiments.

[0284] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” cover embodiments having plural referents, unless expressly indicated otherwise. As used in this specification and the appended claims, the term “or” is generally used in its meaning including “and / or,” unless expressly indicated otherwise.

Claims

1. A method for navigating an autonomous machine, comprising: The autonomous machine is trained within the working area to obtain: two-dimensional training images from the camera of the autonomous machine, and pose data from the non-visual sensors of the autonomous machine; Feature data is extracted from the training images, and the feature data is associated with features in the training images; Generate matching data that matches features from different training images; The matching data and the pose data are used to determine the correspondence between the 3D points and the matching features in the working area; Generate a 3D point cloud representing at least the working area based on the 3D points and corresponding matching features: Generate pose data associated with points in the 3D point cloud, the pose data representing the pose of the autonomous machine; as well as The attitude data is used to determine the boundaries of the work area for subsequent navigation of the autonomous machine within that work area.

2. The method as described in claim 1, wherein, The autonomous machine responds to being guided around the working area during training to learn the boundary.

3. The method according to claim 1, wherein, In a training mode initiated by the user of the autonomous machine, the boundary is taught to the autonomous machine.

4. The method according to claim 1, wherein, The boundaries are determined based on non-visual sensor data and the attitude data during subsequent navigation.

5. The method according to claim 4, wherein, The pose data associated with the 3D point cloud is used to correct or locate the estimated pose determined using the non-visual sensor.

6. The method according to claim 1, wherein, The attitude data includes at least a three-dimensional position representing the attitude of the autonomous machine.

7. The method according to claim 1, wherein, Generating the 3D point cloud also includes associating the features of the feature data with the 3D points in the 3D point cloud.

8. The method according to claim 1, wherein, Training the autonomous machine includes recording a set of inspection images as the autonomous machine traverses a path around the perimeter of the work area and a secondary path along the interior of the work area to provide the inspection images.

9. The method of claim 1, further comprising determining whether the quality level of the three-dimensional point cloud does not meet a quality threshold before generating the pose data.

10. The method of claim 9, further comprising: In response to determining that the quality level of the 3D point cloud does not meet the quality threshold, a new set of 2D inspection images is recorded in the working area; as well as Based on the new set of two-dimensional inspection images, the three-dimensional point cloud is regenerated.

11. The method according to claim 10, wherein, The new set of two-dimensional inspection images was obtained along a path inside the work area.

12. The method according to claim 1 further includes registering the coordinate system of the three-dimensional point cloud with the scale and orientation of the real world in the navigation map.

13. The method of claim 1, wherein the different training images in the training images include a first training image and a second training image, the second training image not being recorded immediately after the first training image, such that the first training image and the second training image are spaced apart in distance or time, while still sharing the matching features.

14. The method of claim 13, further comprising determining a third training image having overlapping correspondence in the three-dimensional point cloud to verify points identified based on the first training image and the second training image.

15. The method of claim 1, further comprising registering the three-dimensional point cloud with an online map-making service using an Earth-based reference frame.

16. The method according to claim 1, wherein, The work area is an outdoor area, and the autonomous machine is a ground maintenance machine.

17. The method according to claim 1, wherein, The work area is a lawn, and the autonomous machine is a lawn maintenance machine.

18. The autonomous machine of claim 1, wherein the autonomous machine includes a controller capable of operating the controller to perform the method of claim 1.

Images