Autonomous machine navigation in low-light conditions

Abstract

An autonomous machine (100) navigation technique includes configuring camera (133) capture parameters using simulation. A method may include capturing image data of a scene, generating irradiance image data, determining at least one test camera capture parameter, determining simulated scene parameters, and generating at least one updated camera capture parameter. Image data for camera capture configuration may be captured while the autonomous machine is in motion. Particularly in low light conditions, the parameters captured by the camera (133) may be used to capture images when the autonomous machine (100) is slowed or stopped.

Description

[0001] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 831,347, filed on April 9, 2019, which is incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to autonomous machine navigation, and in particular, to autonomous machine navigation in low-light conditions. Background Art

[0003] Various machines, such as grounds maintenance machines for lawns and gardens, can perform a variety of tasks. For example, both homeowners and professionals use power mowers to maintain the grass on a property or in a yard. Some mowers are able to autonomously perform mowing within the predetermined boundaries of the work area. Summary of the invention

[0004] The technology of the present disclosure generally relates to an autonomous machine incorporating a low-light navigation system, which can be implemented in a controller or can be part of an overall navigation system to provide additional functionality to the machine. For example, a low-light navigation system can provide or assist in detecting a low-light environment (e.g., dark conditions during night, dawn, or dusk) and facilitate navigation in the low-light environment. Navigation in a low-light environment can include providing illumination during image recording. Although a strong illumination source can be used to facilitate continued operation of an autonomous machine in low-light conditions, illumination and / or long exposure times can be used in conjunction with slowing or stopping motion to facilitate low-light navigation. In some low-light environments, illumination may not be required or used for navigation. Certain low-light navigation techniques described herein, which may also be referred to as "slow and stare" or "stop and stare" techniques, can increase battery life and ease of manufacture at the expense of mowing speed (e.g., continuous operation). Techniques including camera capture configurations can facilitate improved nighttime navigation, for example, by reducing exposure times and reducing the use of active illumination to capture low-brightness images for positioning.

[0005] In independent aspect A1, a method for autonomous machine navigation includes: capturing image data of a scene using one or more cameras configured with at least one camera capture parameter; generating irradiance image data based on the image data of the scene and calibrated irradiance mapping data; determining at least one test camera capture parameter based on the irradiance image data of the scene; determining simulated scene parameters based on the irradiance image data and at least one test camera capture parameter; and generating at least one updated camera capture parameter based on the at least one test camera capture parameter in response to determining that the simulated scene parameters are acceptable.

[0006] In aspect A2, aspect A1 further comprises generating simulated image data of the scene based on the at least one test camera capture parameter.Determining the simulated scene parameters comprises determining the simulated scene parameters further based on the simulated image data of the scene.

[0007] In aspect A3, any of the preceding aspects A also includes, wherein determining at least one test camera capture parameter includes determining at least one test camera capture parameter also based on at least one previously stored camera capture parameter.

[0008] In aspect A4, any of the aforementioned aspects A also includes: determining captured scene parameters based on image data of the scene; determining whether the captured scene parameters are acceptable; in response to determining that the captured scene parameters are acceptable, capturing a positioning image using at least one camera capture parameter; and generating irradiance image data in response to determining that the captured scene parameters are unacceptable.

[0009] In aspect A5, aspect A4 further comprises determining that the captured scene parameter is acceptable in response to the captured scene parameter exceeding a captured scene parameter threshold.

[0010] In aspect A6, aspect A4 or A5 also includes, wherein the captured scene parameters include one or more of: an average pixel intensity, a median pixel intensity, or a weighted sum of pixel intensities.

[0011] In aspect A7, any one of aspects A4 to A6 further comprises: generating masked captured image data based on the image data of the scene in response to determining that the captured scene parameters are unacceptable; and generating irradiance image data based on the masked captured image data.

[0012] In aspect A8, any of the preceding aspects A further comprises: generating masked simulated image data based on the simulated image data of the scene; and determining simulated scene parameters based on the masked simulated image data.

[0013] In aspect A9, any of the preceding aspects A also includes determining that the simulated scene parameters are acceptable in response to the simulated scene parameters exceeding a simulated scene parameter threshold.

[0014] In aspect A10, any preceding aspect A also includes, wherein the simulated scene parameters include one or more of: an average pixel intensity, a median pixel intensity, or a weighted sum of pixel intensities.

[0015] In aspect A11, any of the preceding aspects A further comprises: in response to determining that the simulated scene parameters are unacceptable, updating at least one test camera capture parameter; and generating simulated image data of the scene based on the at least one test camera capture parameter.

[0016] In aspect A12, any of the preceding aspects A also includes, wherein one or both of the at least one updated camera capture parameter and the at least one test camera capture parameter include one or more of: exposure time, gain, and active lighting intensity.

[0017] In aspect A13, any of the aforementioned aspects A also includes, wherein using at least one updated camera capture parameter includes: in response to a calculated exposure time exceeding an exposure time threshold, using a reduced exposure time that does not exceed the exposure time threshold as a camera capture parameter, and using a gain as a camera capture parameter based on the calculated exposure time and the reduced exposure time.

[0018] In aspect A14, any preceding aspect A further comprises configuring one or more cameras or updating the positioning using at least one updated camera capture parameter in a subsequent capture of image data using the one or more cameras.

[0019] In aspect A15, any of the aforementioned aspects A also includes: capturing image data of a scene during movement of the autonomous machine under low-light conditions in the working area; determining whether to update the positioning of the autonomous machine during the movement of the autonomous machine; reducing the movement of the autonomous machine in response to determining the updated positioning; capturing an operating image of at least a portion of the working area under low-light conditions while reducing the movement of the autonomous machine using at least one updated camera capture parameter; updating a pose estimate of the autonomous machine based on the captured operating image; and restoring the movement of the autonomous machine within the working area based on the updated pose estimate.

[0020] In aspect A16, any of the aforementioned aspects A also includes: determining an uncertainty parameter based on the current posture; determining whether the uncertainty parameter exceeds an uncertainty threshold; reducing the movement of the autonomous machine and determining to update the positioning of the autonomous machine in response to the uncertainty parameter exceeding the uncertainty threshold; and capturing an operating image of at least a portion of the working area while the movement of the autonomous machine is reduced.

[0021] In aspect A17, aspect A16 also includes, wherein the uncertainty threshold is determined based on a distance to a boundary of the working area.

[0022] In aspect B1, the autonomous machine includes: a housing coupled to a maintenance tool; a propulsion system including at least one motor; at least one camera suitable for recording images under one or more lighting conditions; and a controller operably coupled to the at least one camera and the propulsion system, the controller being suitable for executing a method according to any of the aforementioned aspects A.

[0023] In aspect C1 , a computer-readable medium includes instructions stored thereon, which, when executed by a processing circuit, cause the processing circuit to perform a method according to any preceding aspect A.

[0024] In an independent aspect D1, an autonomous machine includes: a housing coupled to a maintenance tool; a propulsion system including at least one motor; at least one camera adapted to record images under one or more lighting conditions; and a controller operably coupled to the at least one camera and the propulsion system. The controller is adapted to: detect whether a low-light condition exists; determine whether to update a position of the autonomous machine; command the propulsion system to slow or stop movement of the autonomous machine in response to determining to update the position and detecting a low-light condition; command the at least one camera to record one or more images of a work area under low-light conditions; update a pose estimate of the autonomous machine based on the one or more recorded images; and command the propulsion system to resume movement of the autonomous machine based on the updated pose estimate.

[0025] In aspect D2, aspect D1 also includes, wherein the controller is further adapted to: compare image data based on one or more recorded images recorded under low-light conditions with daylight or low-light characteristic data; and determine vision-based gesture data based on the comparison of the image data with the daylight or low-light characteristic data.

[0026] In aspect D3, aspect D2 also includes, wherein the controller is further adapted to update the pose estimate of the autonomous machine based on the vision-based pose data.

[0027] In aspect D4, any of the preceding D aspects also includes, wherein the controller is further adapted to determine exposure time and / or additional illumination to record the image based on the detected lighting conditions.

[0028] In aspect D5, aspect D4 also includes, wherein the exposure time and / or additional lighting is determined based on a threshold value such that a low-light navigation map can be used or feature matching occurs between low-light and daylight features.

[0029] In independent aspect E1, a method for autonomous machine navigation includes: detecting whether a low-light condition exists; determining whether to update the positioning of the autonomous machine; slowing down or stopping the movement of the autonomous machine in response to determining to update the positioning and detecting the low-light condition; recording an image of at least a portion of a working area under the low-light condition; updating a pose estimate of the autonomous machine based on the recorded image; and resuming the movement of the autonomous machine within the working area based on the updated pose estimate.

[0030] In an independent aspect FI, a method for autonomous machine navigation includes: determining an uncertainty parameter based on a current posture; determining whether the uncertainty parameter exceeds an uncertainty threshold; in response to the uncertainty parameter exceeding the uncertainty threshold, slowing down or stopping the movement of the autonomous machine and determining to update the positioning of the autonomous machine; and recording an image of at least a portion of a working area when the autonomous machine slows down or stops.

[0031] The Summary is not intended to describe every aspect or every implementation of the present disclosure. A more complete understanding will become apparent and appreciated by referring to the following Detailed Description and Claims in light of the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Exemplary embodiments will be further described with reference to the accompanying drawings, in which:

[0033] Figure 1 is a schematic diagram of an autonomous working machine (e.g., a ground working machine such as a robotic lawn mower) incorporating a vision system according to the present disclosure;

[0034] Figure 2 The use according to the present disclosure Figure 1 A schematic diagram of an example of various systems of a machine;

[0035] Figure 3 The use according to the present disclosure Figure 1 A flowchart of an example of a navigation method of a machine;

[0036] Figure 4 For implementation according to the present disclosure Figure 3 A schematic diagram of an example of a navigation method;

[0037] Figure 5 According to the present disclosure Figure 2 A flow chart of an example of a functional relationship between a vision system and a controller;

[0038] Figure 6 The use according to the present disclosure Figure 1 A flowchart of an example of a navigation method of a machine;

[0039] Figure 7 According to the present disclosure, including the use Figure 1 A flowchart of an example of a navigation method for capturing a camera configuration of a machine;

[0040] Figure 8 According to the present disclosure, including the use Figure 1 A flowchart of another example of a navigation method for capturing a camera configuration of a machine; and

[0041] Fig. 9 The use according to the present disclosure Figure 1 A flow chart of an example of a method for automatic camera capture configuration of a machine.

[0042] These figures are presented primarily for clarity and are therefore not necessarily drawn to scale. In addition, various structures / components (including, but not limited to, fasteners, electrical components (wiring, cables, etc.), etc.) may be schematically illustrated or removed from some or all views to better illustrate various aspects of the depicted embodiments, or the inclusion of such structures / components is not necessary for understanding the various exemplary embodiments described herein. However, the lack of illustration / description of such structures / components in a particular figure should not be construed as limiting the scope of the various embodiments in any way. DETAILED DESCRIPTION

[0043] In the following detailed description of illustrative embodiments, reference is made to the accompanying drawings which form a part hereof. It should be understood that other embodiments may of course be contemplated which may not be described and / or illustrated herein.

[0044] Unless otherwise stated, all titles provided herein are for the convenience of the reader and should not be used to limit the meaning of any text after the title. In addition, unless otherwise stated, all numbers representing quantity and all terms representing direction / orientation (e.g., vertical, horizontal, parallel, perpendicular, etc.) in the specification and claims should be understood to be modified by the term "precise" or "approximately" in all cases. Unless the context clearly states otherwise, the term "or" is usually used with its inclusive meaning, such as to represent "and / or". The term "and / or" (if used) refers to one or all listed elements, or a combination of any two or more listed elements. The term "ie" is used as an abbreviation of the Latin phrase id est, meaning "that is". The term "eg," is used as an abbreviation of the Latin phrase exempli gratia, meaning "for example".

[0045] The present disclosure provides an autonomous machine incorporating a low-light navigation system, which can be implemented in a controller or can be part of an overall navigation system to provide additional functionality to the machine. For example, a low-light navigation system can provide or assist in detecting a low-light environment (e.g., dark conditions during night, dawn, or dusk) and facilitate navigation in a low-light environment. Navigation in a low-light environment can include providing lighting during image recording. Although a strong lighting source can be used to facilitate continuous operation of an autonomous machine in low-light conditions, lighting and / or long exposure times can be used in conjunction with slowing or stopping motion to facilitate low-light navigation. In some low-light environments, lighting may not be required or used for navigation. Certain low-light navigation techniques described herein, which may also be referred to as "slow and stare" or "stop and stare" techniques, can increase battery life and ease of manufacture at the expense of mowing speed (e.g., continuous operation). Techniques including camera capture configurations can facilitate improved nighttime navigation, for example, by reducing exposure times and reducing the use of active lighting to capture low-light images for positioning.

[0046] The techniques described herein can use a photometric calibration (or model) of an imaging camera system of an autonomous machine. The model may or may not be calibrated for each image sensor (imager), or a single model may be used for all imagers in the system.

[0047] Images can be captured while the machine is in motion, even when there is significant motion blur due to the long exposure times required. The blurred images can be used in conjunction with the imager model to select acceptable exposure, gain, and optional active lighting intensity parameters.

[0048] In some aspects, an image is captured and evaluated to determine whether it is "well exposed". The evaluation can be performed in any suitable manner. In one example, the average pixel intensity can be determined and compared to an average pixel intensity threshold or an interval of an acceptable range of average pixel intensity. Other non-limiting examples include determining a median pixel intensity or a weighted sum of pixel intensities (e.g., point weighted or area weighted), which can be compared to a corresponding threshold or interval of an acceptable range. In some aspects, intensity can also be described in relative terms, such as percentage brightness.

[0049] If the image is well exposed, the camera capture parameters (or even the image itself) can be passed to the absolute positioning algorithm. If the image is not well exposed, the camera capture parameters can be improved before being sent to the absolute positioning algorithm.

[0050] In some aspects, image masking is applied to focus image analysis on relevant portions of the image. In one example, image edges and excessively high or low intensities associated with seeing a machine can be masked.

[0051] An image can be converted from pixel intensity to irradiance units by applying a calibrated camera irradiance map. A camera irradiance map can be provided as a lookup table (LUT) or function generated by calibration tests performed on one or more image sensors or camera boards. These measurements only need to be performed once for a few camera modules and are typically applicable to all cameras of that model. A camera irradiance map can relate the scene irradiance seen by each pixel to the pixel intensity recorded by each pixel. Such a map can facilitate estimation of scene brightness and simulation of images if different parameters, such as longer exposure times, were used.

[0052] The pixel intensity images produced by the trial camera capture parameters can be simulated by entering a search loop to estimate better camera capture parameters. Simulated images are generated using different trial set camera capture parameters during the search loop. A weighted mask can be applied to focus the image analysis on the most important parts of the image. It can then be determined whether it is "well exposed".

[0053] If the simulated image is well exposed, the camera capture parameters can be passed to the absolute positioning algorithm for capturing a new image. If the simulated image is not well exposed, the iterative search loop can continue to estimate better camera capture parameters. In some aspects, the search loop performs a binary search algorithm, a gradient-based search algorithm, or any other suitable search algorithm to find a better set of camera capture parameters.

[0054] The machine can navigate using dead reckoning (DR) using inertial sensors, wheel encoders, or other relative motion sensors to estimate machine position, orientation, and velocity (or attitude), and can also estimate attitude uncertainty. When the attitude uncertainty is greater than a threshold metric, the machine can slow down or stop to perform absolute positioning.

[0055] During absolute positioning, the machine can capture one or more images with the estimated acceptable exposure, gain, and lighting intensity parameters to capture one or more long exposure or high dynamic range (HDR) images. Capturing HDR images can include capturing multiple short exposure images at high gain, which can be stacked or otherwise combined to approximate a long exposure image based on multiple images. In many cases, the first long exposure image is sufficient for positioning, saving the time required for multiple exposures to adjust exposure, gain, and lighting intensity parameters.

[0056] Absolute positioning from images can provide position and position uncertainty estimates. This position and position uncertainty estimate can be used to improve the attitude estimate and attitude uncertainty estimate. When a satisfactory attitude uncertainty estimate is obtained, the machine stops and stares operation can return to dead reckoning for navigation and intermittently repeat the absolute positioning.

[0057] Although referred to herein as an autonomous lawn mower in the illustrative examples, this configuration is illustrative only, as the systems and methods described herein are also applicable to other autonomous machines, including, for example, commercial lawn mowing products, other ground-working machines or vehicles (e.g., debris blowers / vacuums, aerators, lawn mowers, spreaders, snowplows, weed whackers for weed control), indoor work vehicles such as vacuums and floor scrubbers / cleaners (e.g., that may encounter obstacles), construction and utility vehicles (e.g., trenchers), observation vehicles, and load transport (e.g., including people and objects, such as personnel moving and towing equipment). In addition, the autonomous machines described herein may employ various one or more types of navigation, such as random, modified random, or specific path planning, to perform their intended functions.

[0058] As used herein, the term "slowing down" refers to a reduction in the motion of an autonomous machine from a first speed to a second speed that is lower than the first speed. The first speed may correspond to a nominal operating speed used by the machine to perform tasks in a work area. The second speed may correspond to a speed used to capture images for vision-based localization under low light conditions. Typically, the second speed is slow enough to allow one or more images to be captured without significant blurring, such that features can be extracted from the captured one or more images.

[0059] As used herein, the term "stop" refers to slowing or reducing the motion of the autonomous machine to a speed equal to or nearly equal to zero (eg, less than or equal to 0.1 m / s, 0.05 m / s, or even 0.01 m / s).

[0060] It should be noted that the terms "having," "including," "comprising," and variations thereof, as they appear in the accompanying description and claims, are not intended to be limiting and are used in their open-ended sense, generally meaning "including but not limited to." Additionally, "a," "an," "the," "at least one," and "one or more" may be used interchangeably herein. In addition, relative terms such as "left," "right," "front," "front," "forward," "back," "rear," "backward," "top," "bottom," "side," "up," "down," "above," "below," "horizontal," "vertical," and the like may be used herein and, if used, are from the perspective shown in a particular figure, or when the machine 100 is in an operating configuration (e.g., as shown in FIG. 1 ). Figure 1 1, when the machine 100 is positioned so that the wheels 106 and 108 rest on a substantially horizontal ground surface 103). However, these terms are used only to simplify the description and do not limit the interpretation of any of the embodiments described.

[0061] The techniques of this disclosure may be used with any suitable autonomous machine. Figure 1One example of an autonomous ground maintenance machine (e.g., an autonomously operating vehicle, such as autonomous lawn mower 100) is shown of a lawn mowing system (for ease of description, lawn mower 100 is schematically shown). Lawn mower 100 may include functionality for autonomous navigation in various lighting conditions (e.g., during the day or at night).

[0062] As shown in the figure, the lawn mower 100 may include a housing 102 (e.g., a frame or chassis with a guard) that carries and / or surrounds various components of the lawn mower as described below. The lawn mower 100 may also include ground support members, such as wheels, rollers, legs, or tracks. In the illustrated embodiment, the ground support members shown include one or more rear wheels 106 and one or more front wheels 108 that support the housing 102 on a ground (grass) surface 103. As shown, the front wheels 108 are used to support a front end portion 134 of the lawn mower housing 102, and the rear wheels 106 are used to support a rear end portion 136 of the lawn mower housing.

[0063] One or both rear wheels 106 may be driven by a propulsion system (e.g., including one or more electric wheel motors 104) to propel the mower 100 on the ground surface 103. In some embodiments, the front wheel 108 may be freely castor-rotatable relative to the housing 102 (e.g., about a vertical axis). In such a configuration, the direction of the mower may be controlled by differential rotation of the two rear wheels 106 in a manner similar to some existing zero turning radius (ZTR) riding mowers. That is, the propulsion system may include an independent wheel motor 104 for each of the left and right rear wheels 106, so that the speed and direction of each rear wheel can be independently controlled. Additionally or alternatively, the front wheels 108 may be actively steered by a propulsion system (e.g., including one or more steering motors 105) to help control the direction of the mower 100, and / or may be driven by the propulsion system (i.e., providing a front-wheel or all-wheel drive mower).

[0064] A tool (e.g., a mowing element, such as a blade 110) can be coupled to a cutting motor 112 (e.g., a tool motor) carried by the housing 102. When the motors 112 and 104 are energized, the mower 100 can be propelled on the ground surface 103 so that vegetation (e.g., grass) passed by the mower is cut by the blade 110. Although only a single blade 110 and / or motor 112 are used herein for illustration, it is contemplated within the scope of the present disclosure to incorporate a mower with multiple blades powered by a single or multiple motors. Furthermore, although described herein in the context of one or more conventional "blades," other cutting elements including, for example, discs, nylon rope or wire elements, knives, cutting reels, and the like are certainly possible without departing from the scope of the present disclosure. Additionally, it is also contemplated to incorporate various cutting elements, such as embodiments of a rotary blade with an edge-mounted trimmer.

[0065] The lawn mower 100 may also include a power source, which in one embodiment is a battery 114 having a lithium-based chemistry (e.g., lithium-ion chemistry or lithium iron phosphate chemistry). Other embodiments may utilize batteries of other chemistries or other power technologies (e.g., solar, fuel cells, internal combustion engines) without departing from the scope of the present disclosure. It should also be noted that while shown as using separate blade and wheel motors, this configuration is merely illustrative, and embodiments are also contemplated in which a single motor provides blade and wheel power.

[0066] The lawn mower 100 may also include one or more sensors to provide position data. For example, some embodiments may include a global positioning system (GPS) receiver 116 (or other position sensor that can provide similar data) that is suitable for estimating the position of the lawn mower 100 within the working area and providing such information to the controller 120 (described below). In other embodiments, one or more wheels 106, 108 may include an encoder 118 that provides wheel rotation / speed information that can be used to estimate the position of the lawn mower within a given working area (e.g., based on an initial starting position). The lawn mower 100 may also include a sensor 115 suitable for detecting boundary lines, which may be used in addition to the other navigation techniques described herein.

[0067] The lawn mower 100 may include one or more front obstacle detection sensors 130, and one or more rear obstacle detection sensors 132, as well as other sensors, such as side obstacle detection sensors (not shown). The obstacle detection sensors 130, 132 may be used to detect obstacles in the path of the lawn mower 100 when it is traveling forward or backward, respectively. The lawn mower 100 is capable of mowing while moving in either direction. As shown, the sensors 130, 132 may be located at the front end portion 134 or the rear end portion 136 of the lawn mower 100, respectively. In addition to the sensors described, other sensors now known or later developed may also be incorporated into the lawn mower 100.

[0068] The sensors 130, 132 may use contact sensing and / or contactless sensing. For example, depending on the state of the lawn mower 100 (e.g., traveling within a zone or between zones), both contact and contactless sensing may be enabled, or only one type of sensing may be used. An example of contact sensing includes the use of a contact bumper protruding from the housing 102, or the housing itself, which may detect when the lawn mower 100 contacts an obstacle. Contactless sensors may use sound waves or light waves (e.g., using infrared, radio detection and ranging (radar), light detection and ranging (lidar), etc.), sometimes to detect obstacles at a distance from the lawn mower 100 before contact with the obstacle.

[0069] The lawn mower 100 may include one or more vision-based sensors to provide positioning data, such as position, orientation, or speed. The vision-based sensors may include one or more cameras 133 that capture or record images for use by the vision system. The cameras 133 may be referred to as part of the vision system of the lawn mower 100. Image types include, for example, training images and / or operational images.

[0070] One or more cameras 133 can detect visible light and / or invisible light (e.g., infrared light). Any suitable total field of view can be used. In some embodiments, one or more cameras 133 can establish a total field of view greater than or equal to 30, 45, 60, 90, 120, 180, 270, or 360 degrees relative to the horizontal plane around the autonomous machine (e.g., lawn mower 100). In some embodiments, one or more cameras 133 can establish a total field of view of, for example, less than or equal to 360, 270, 180, 120, 90, 60, 45, or 30 degrees relative to the horizontal plane around the autonomous machine. The field of view can be defined in the horizontal direction and / or vertical direction. For example, the total horizontal field of view can be less than or equal to 360 degrees, and the total vertical field of view can be 45 degrees. In some embodiments, the total field of view can be described in three-dimensional (3D) geometry, such as steradian. For example, the total field of view can be less than or equal to 4p steradian. The field of view can capture image data above and below the height of one or more cameras 133.

[0071] In some embodiments, the lawn mower 100 includes four cameras 133. One camera 133 can be located in each of one or more directions including a front direction, a rear direction, a first side direction, and a second side direction (e.g., relative to a cardinal direction of the lawn mower 100). One or more camera directions can be positioned orthogonally to one or more other cameras 133 or positioned opposite to at least one other camera 133. The cameras 133 can also be offset from any of these directions (e.g., at 45 degrees or another non-perpendicular angle).

[0072] The lawn mower 100 may include one or more illumination sources 140 that may be activated by the controller 120. The one or more illumination sources 140 may be incorporated into one or more cameras 133 or attached to the housing 102. The illumination sources 140 may illuminate one or more areas around the lawn mower 100. The illumination sources 140 may be configured to project light onto a ground surface. This may prevent one or more of the cameras 133 from being blinded by reflections from the ground. Any suitable type of illumination source 140 may be used, such as a light emitting diode.

[0073] In some embodiments, illumination source 140 may be configured to provide continuous or discrete amounts of light. For example, when a low-light image is to be recorded, illumination source 140 may "flash" or provide a short-duration burst of light.

[0074] Illumination source 140 can provide one or more lights in the visible light, infrared, or near infrared spectrum. In some embodiments, illumination source 140 only generates invisible light, such as infrared light. The invisible light or visible light provided by illumination source 140 can be detected by camera 133.

[0075] Various exposure times and / or lighting intensities may be used, for example, based on the environment. The exposure time may also be balanced with the amount of lighting intensity. In some environments, low intensity light or no light may be used, for example, in combination with long exposure times. In other environments, medium or high intensity light may be used in combination with short or medium exposure times.

[0076] The lawn mower 100 may be guided along the path, for example in a manual manner using a handle. In particular, manual guidance of the lawn mower 100 may be used during a training mode to learn a work area or boundaries associated with a work area.

[0077] The camera 133 located in the front direction may have a position and orientation representing the attitude of the autonomous machine. In some embodiments, the position and orientation of the camera may be defined relative to the geometric center of the lawn mower 100 or relative to one of the edges of the lawn mower 100 .

[0078] As used herein, the term "attitude" refers to position and orientation. The attitude can be a six-degree-of-freedom attitude (6DOF attitude) that can include all position and orientation parameters of three-dimensional space. The attitude data can include three-dimensional position and three-dimensional orientation. For example, the position can include at least one position parameter selected from (e.g., using a Cartesian coordinate system) x-axis, y-axis, and z-axis coordinates. Any suitable angular orientation representation can be used. Non-limiting examples of angular orientation representations include yaw, pitch, and roll representations, and can also be represented by Rodriguez representation, quaternion representation, and direction cosine matrix (DCM) representations independently or in combination. In one example, the orientation can include at least one orientation parameter selected from yaw (e.g., vertical z-axis orientation), pitch (e.g., lateral y-axis orientation), and roll (e.g., longitudinal x-axis direction).

[0079] The sensors of the lawn mower 100 may also be referred to as vision-based sensors or non-vision-based sensors. A vision-based sensor may include a camera 133 capable of recording images. The images may be processed and used to construct a three-dimensional (3D) point cloud or for optical odometry (e.g., optical encoding). A non-vision-based sensor may include any sensor that is not a camera 133. For example, a wheel encoder that uses optical (e.g., photodiode), magnetic, or capacitive sensing to detect wheel rotation may be referred to as a non-vision-based sensor that does not use a camera. Wheel encoded data from a wheel encoder may also be referred to as odometry data.

[0080] Optical encoding can be used by taking a series of images or a sequence of images and comparing features in the images to determine or estimate the distance traveled between images. Optical encoding may be less affected by wheel slip than wheel encoders used to determine distance or speed.

[0081] In addition to the sensors described above, other sensors now known or later developed may also be incorporated into the lawn mower 100 .

[0082] The lawn mower 100 may also include a controller 120 adapted to monitor and control various lawn mower functions. The controller 120 may include a processor 122 that receives various inputs and executes one or more computer programs or applications stored in a memory 124. The memory 124 may include computer-readable instructions or applications that, when executed, for example, by the processor 122, cause the controller 120 to perform various calculations and / or issue commands. That is, the processor 122 and the memory 124 may together define a computing device that is operable to process input data and generate desired outputs to one or more components / devices. For example, the processor 122 may receive various input data including position data from a GPS receiver 116 and / or an encoder 118, and generate speed and steering angle commands to one or more drive wheel motors 104 to rotate the drive wheels 106 (at the same or different speeds and in the same or different directions). In other words, the controller 120 may control the steering angle and speed of the lawn mower 100, as well as the speed and operation of the cutting blades.

[0083] The lawn mower 100 may include one or more illumination sources 140 that may be activated by the controller 120. The one or more illumination sources 140 may be incorporated into one or more cameras 133 or attached to the housing 102. The illumination sources 140 may illuminate one or more areas around the lawn mower 100. The illumination sources 140 may be configured to project light onto a ground surface. This may prevent one or more of the cameras 133 from being blinded by reflections from the ground. Any suitable type of illumination source 140 may be used, such as a light emitting diode.

[0084] Each system may also be referred to as having its own controller 120. For example, the vision system may be referred to as including one controller 120, and the navigation system may be referred to as having another controller 120. Thus, the lawn mower 100 may be referred to as having multiple controllers 120. In general, as used herein, the term "controller" may be used to describe a component of a "system" that provides commands to control various other components of the system.

[0085] In view of the above, it is apparent that the functionality of the controller 120 may be implemented in any manner known to those skilled in the art. For example, the memory 124 may include any volatile, non-volatile, magnetic, optical, and / or electrical media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), electrically erasable programmable ROM (EEPROM), flash memory, and / or any other digital media. The memory 124 and processor 122, while both shown as being incorporated into the controller 120, may also be included in separate modules.

[0086] The processor 122 may include any one or more of a microprocessor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or an equivalent discrete or integrated logic circuit. In some embodiments, the processor 122 may include multiple components, such as one or more microprocessors, one or more controllers, one or more DSPs, one or more ASICs, and / or one or more FPGAs, and a combination of other discrete or integrated logic circuits. The functions attributed to the controller 120 and / or the processor 122 herein may be embodied as software, firmware, and / or hardware. Certain functions of the controller 120 may also be performed by a cloud computing or other distributed computing system that is operably connected to an Internet connection of the processor 122.

[0087] exist Figure 1, schematic connections between the controller 120 and the battery 114, the wheel motor 104, the blade motor 112, the optional boundary line sensor 115, the radio 117, and the GPS receiver 116 are generally shown. Such interconnections are illustrative only, as the various subsystems of the lawn mower 100 can be connected in almost any manner, such as direct connections to each other that allow data and / or power to be transferred between the various components of the lawn mower, wireless connections, connections through a bus architecture (e.g., a controller area network (CAN) bus), or any other connection configuration. Although connections to some of the sensors 130, 132, 133 are not shown, these sensors and other components of the lawn mower 100 can be connected in a similar manner. The radio 117 can communicate with a separate user interface device 119 (e.g., a mobile device such as a smart phone, tablet, desktop, or wearable computer) via a cellular or other wide area network (e.g., even via the Internet), a local area network (e.g., an IEEE 802.11 "Wi-Fi" radio), or a peer-to-peer (P2P) (e.g., Bluetooth) network. In turn, the user interface device 119 can communicate with other devices via similar networks and, for example, can be used to connect the lawn mower 100 to the Internet. Additionally, the controller 120 of the lawn mower 100 can be in operative communication with a separate device such as a remote computer or other user interface device 119 (e.g., via the radio 117).

[0088] The user interface device 119 may or may not be coupled to the lawn mower 100. In some embodiments, the user interface device 119 may be used to provide user input or commands from a user to the lawn mower 100. In some embodiments, the user interface device 119 may be used to provide instructions to the user, such as a visual presentation on a graphical user interface (e.g., a screen or touch screen).

[0089] In some embodiments, various functions of one or more controllers 120 described herein can be offloaded from the lawn mower 100. For example, recorded images can be transmitted to a remote server (e.g., in the cloud) using the radio 117 and processed or stored. Stored images or other data resulting from processing can be received using the radio 117 and stored on the lawn mower 100 or further processed by the lawn mower 100.

[0090] The lawn mower 100 can utilize an exemplary vision and navigation system to allow the lawn mower to operate autonomously within a given work area. More information about exemplary operation and navigation of the lawn mower 100 can be found, for example, in U.S. Patent Application No. 16 / 534,515 filed on August 7, 2019, which is incorporated herein by reference.

[0091] In general, the controller 120 and various components or systems of the lawn mower 100 can be used to perform the various functions described herein. In some embodiments, components of the vision and navigation system (e.g., the camera 133) can be used in low light conditions. In addition to addressing navigation in dimly lit areas, the vision and navigation system can also be used to save power when mowing in low light conditions (e.g., at night). Some examples of systems of the lawn mower 100 are described in Figure 2 Shown in.

[0092] Figure 2 An autonomous machine (e.g., Figure 1 1 , an overview of the various systems in the lawn mower 100 of FIG. 1 . Although all of the systems 202, 204, 206 are shown as being completely part of the lawn mower 100, in some embodiments, some or all of each system may also be located outside of the lawn mower 100. The vision system 202 can be operably coupled to the navigation system 204, and the navigation system 204 can be operably coupled to the propulsion system 206. Any suitable navigation technology can be used for the lawn mower 100. In some embodiments, navigation technology can be used as described in U.S. Patent Application No. 16 / 534,515 filed on August 7, 2019 and U.S. Provisional Application No. 62 / 831,347 filed on April 9, 2019, which are incorporated herein by reference.

[0093] The sensors of the navigation system 204 can be used to record non-vision-based data during the training mode while the vision system 202 records images, such as training images. In some embodiments, although the lawn mower 100 can be manually guided by a user, the navigation system 204 can autonomously guide the machine during the training mode. The vision system 202 can include one or more cameras to record or capture images. In some embodiments, the controller of the vision system 202 can provide position and / or orientation data to the navigation system 204 based on the recorded images, and the recorded images can be used to facilitate navigation of the lawn mower 100. For example, the vision system 202 can provide the navigation system 204 with an estimated position and / or orientation of the lawn mower 100 based on the vision-based sensor data.

[0094] In some embodiments, the navigation system 204 may navigate primarily using a position and / or orientation based on non-visual sensor data. For example, non-visual sensor data may be based on output from an inertial measurement unit or a wheel encoder. During training mode or offline mode, for example, the controller of the navigation system 204 may use non-visual sensor data and visual data to determine boundaries for subsequent navigation of the autonomous machine in a work area. For example, during online mode, the controller of the navigation system 204 may determine a posture based on visual posture data and / or based on non-visual posture data. In some embodiments, a posture may be determined based on non-visual sensor data, and a posture may be updated based on visual posture data, which may be referred to as vision-based positioning. The navigation system 204 may compare data from various sensors to correct errors and update positions, which may be referred to as sensor fusion. For example, a visual-based position and / or orientation may be compared with a non-visual-based position and / or orientation. In some embodiments, sensor data other than visual-based sensor data may be used to correct errors and update positions, such as GPS data.

[0095] The navigation system 204 can extract feature data from images captured by the vision system 202. Feature data can be extracted from training images to provide training feature data. The navigation system 204 can use the training feature data to generate a three-dimensional point cloud (3DPC) and associated data, which may include multiple 6DOF poses of the autonomous machine to represent the work area. The 3DPC and pose can be registered in a navigation map. Optionally, non-visual-based data, such as GPS data, IMU data, and mileage data, can also be registered to a real-world coordinate system, such as a coordinate system of a map referenced by the earth. Feature data can be extracted from the operating image and compared with the training feature data, which can be referred to as feature matching. During operation, comparison of the operating feature data with the training feature data can be used to identify the position and / or orientation of the autonomous machine to facilitate navigation.

[0096] In some aspects, the operational image can be analyzed to determine whether the image is "well exposed." In one example, the operational image can be analyzed to obtain acceptable scene capture parameters (see Fig. 9 904).

[0097] As used herein, the term "feature" refers to two-dimensional (2D) data generated by identifying one or more points in a two-dimensional image, particularly key points or points of interest. Features can be identified and extracted from an image using a feature detector algorithm. Any suitable feature detector algorithm available to a person of ordinary skill in the art can be used according to a specific autonomous machine and application. In some embodiments, each unique feature refers only to a point or point of interest in an image or 3D PC. Features can be stored as feature data containing coordinates defined relative to an image frame. In some embodiments, feature data may also include descriptors applied to, associated with, or corresponding to features. The term "feature data" refers to a data structure representing a feature and may include a two-dimensional position and a multi-dimensional descriptor (e.g., various identification attributes of a reference feature).

[0098] The key points used to identify features can be extracted from various objects in the image. In some embodiments, the object can be permanent and / or temporary. In some embodiments, the object can be natural and / or artificial. An example of a permanent feature is the corner of a house. An example of a natural feature is the edge of a tree trunk. Some examples of temporary and artificial features include targets on the ground and on trees. Artificial features can be temporarily placed and used to increase the feature density in the working area (for example, to improve the low-quality part of 3DPC). Artificial features can be powered, and, for example, artificial features can include light emitters for visible light or invisible light that can be detected by a camera. Artificial features can be unpowered, and, for example, can include visible or invisible patterns that can be detected by a camera. Some artificial features can be permanently set. As used herein, the term "invisible" refers to light that emits or reflects wavelengths that are invisible to the human eye, but can emit or reflect wavelengths that are visible to cameras (for example, infrared cameras on autonomous machines).

[0099] As used herein, the term "descriptor" refers to two-dimensional data generated by a descriptor algorithm. A descriptor describes a feature in the context of an image. In some embodiments, a descriptor may describe pixel values, image gradients, scale space information, or other data in an image near or around a feature. For example, a descriptor may include an orientation vector for a feature or may include an image patch. Any suitable descriptor algorithm available to a person of ordinary skill in the art that provides context for a feature in an image may be used depending on the particular autonomous machine or application. The descriptor may be stored as part of the feature data.

[0100] Any suitable technique, including or utilizing algorithms such as Scale Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST and Rotated BRIEF (ORB), KAZE, Accelerated KAZE (AKAZE), linear feature tracking, camera merging, loop closure, moving incremental structure, or other suitable algorithms, can be used to extract features from an image. For example, such an algorithm can provide one or more features and associated descriptors.

[0101] Any suitable technique, including or utilizing algorithms, such as Brute-Force, Approximate Nearest Neighbor (ANN), and Fast Library of Approximate Nearest Neighbors (FLANN), can be used for feature matching. Brute-force algorithms can match features by selecting a feature and checking all other features for matching.

[0102] As used herein, the term "three-dimensional point cloud", "3D point cloud", or "3DPC" is a data structure representing or containing three-dimensional geometric points corresponding to features extracted from an image. The 3DPC can be associated with various attributes (e.g., pose). In some embodiments, the geometric points and pose may or may not be defined in a coordinate system based on an arbitrary reference system. In some embodiments, the 3DPC may or may not be associated with a scale and / or orientation corresponding to the real world, for example, until a map registration process has been performed. The 3DPC can be generated based on feature matching data. A graphic or visual map can be generated based on the 3DPC to provide a human-visible representation of the 3DPC.

[0103] Any suitable technique (which may include or utilize algorithms such as structure from movement (SfM) or simultaneous localization and mapping (SLAM)) may be used to generate the 3DPC, any of which may be used in various embodiments of the present disclosure, for example depending on the particular autonomous machine and application.

[0104] A controller, which may be independent of or dependent upon vision system 202 and / or navigation system 204, may command propulsion system 206 based on the updated pose. For example, a revised or updated position or orientation may be used to provide propulsion commands to propulsion system 206. Propulsion system 206 (e.g., propulsion hardware) may be defined to include, for example, motors 112, 104 and wheels 106, 108 ( Figure 1 ) or any associated driver (e.g., a motor controller or microchip).

[0105] In some embodiments, the autonomous lawn mower can use an onboard machine vision system and optionally other non-vision based sensors to learn and subsequently identify the boundaries of the work area. The vision system can utilize one or more cameras that together form part of a navigation system, as more fully described in U.S. Patent Application No. 16 / 534,515 filed on August 7, 2019, which is incorporated herein by reference.

[0106] The techniques of the present disclosure can utilize existing cameras that can be used in non-low-light conditions (e.g., in daylight conditions during the day) and can conserve battery power to facilitate effective coverage of a work area in low-light conditions.

[0107] exist Figure 3 , a schematic representation of one example of an autonomous machine navigation process 220 is shown.

[0108] At step 222, it may be detected whether a low light condition exists. For example, the low light condition may be detected by navigation system 204. The low light condition may be detected based on data received by navigation system 204 from vision system 202. In some aspects, process 220 may also include capturing image data when the machine is moving in a work area under low light conditions.

[0109] At step 224, it may be determined whether to update the localization of the autonomous machine. For example, the determination of whether to update the localization may be made by the navigation system 204. In some embodiments, the localization may include a vision-based pose estimation, for example, obtaining a new estimate of the machine's position and orientation from a vision system.

[0110] At step 226, a command may be provided by the controller to propulsion system 206 to slow or stop autonomous vehicle motion by providing a command to propulsion system 206. The command to slow or stop motion may be provided in response to determining an updated position at step 224 and detecting a low light condition at step 222.

[0111] At step 228, a command may be provided by the controller to the vision system 202 to record an image (e.g., a long exposure or HDR image). The recorded image may be an image of a working area of ​​the autonomous machine (e.g., the lawn mower 100). The image may be recorded by a camera (e.g., the camera 133) in low light conditions or non-low light conditions. In some aspects, the recorded image may be captured by one or more cameras using at least one camera capture configuration parameter calculated by the camera capture configuration operation in block 706 (see Figure 7 ).

[0112] At step 230, the pose estimate may be updated based on the recorded image. Any suitable technique may be used to estimate the pose, such as the technique described in U.S. Patent Application No. 16 / 534,515 filed on August 7, 2019, which is incorporated herein by reference.

[0113] At step 232, motion may be resumed. In some embodiments, if there are no low light conditions and an updated position fix is ​​determined, steps 226 and 232 may be skipped.

[0114] In some embodiments, the lawn mower 100 may be operated at a duty cycle that is less than the duty cycle of the lawn mower during daylight navigation.

[0115] exist Figure 4 , a schematic representation of various functions of a slow and stare or stop and stare operation of an autonomous machine (e.g., lawn mower 100) according to one embodiment is shown. The functions of the autonomous machine may include initializing a position, slowing down or stopping movement, and motion or operation (e.g., mowing).

[0116] For example, when the autonomous machine exits a docking station or base station, the initialize position function in block 420 may be used. The autonomous machine may use onboard lighting (e.g., illumination source 140) to illuminate a docking target located on or near the docking station to perform tracking. The onboard lighting may be used at a limited or predetermined distance. The docking target may be referred to as an artificial feature, which may be processed in the same or different manner as other features (e.g., natural features) in the work area. In some embodiments, different techniques (e.g., not using 3DPC) may be used to process the artificial features.

[0117] Slow or stop movement or motion function 422 can be used to selectively slow down or stop autonomous machine motion. For example, when the uncertainty parameter reaches a threshold and the navigation system needs or requests a positioning update, the motion can be slowed down or stopped. As used herein, the term "uncertainty parameter" refers to a threshold measure of the uncertainty estimate of the error of the entire or partial estimated state, such as the error uncertainty estimate in the estimated position. The uncertainty parameter can be compared with a threshold. In some embodiments, the uncertainty parameter can be based on the attitude filter covariance that may become larger and exceed the threshold. For example, the threshold can be determined based on the estimated position error from the attitude filter covariance. In some embodiments, the estimated position error can be determined based on an extended Kalman filter (EKF) parameter, an attitude filter parameter, or a parameter from a covariance matrix. When the positioning is to be updated, the navigation system (e.g., navigation system 204) can communicate with the vision system (e.g., vision system 202) to capture or record one or more long exposure images, for example, using a longer exposure time than the exposure time under nominal, daylight, or non-low light conditions to record each image.

[0118] The vision system can evaluate the image quality of each recorded image. If the quality of the recorded image is below a threshold, the vision system can determine that the exposure time needs to be adjusted. If adjustment is required, the vision system can adjust the exposure time and record another image at the adjusted exposure time. If the quality of the recorded image meets or exceeds the quality threshold, the image can be processed to determine the pose of the autonomous machine. In some embodiments, the pose can be reported or provided to a pose filter of the autonomous machine.

[0119] The mowing or motion function in block 424 can be used to put the autonomous machine into motion. The autonomous machine, for example, after slowing or stopping to record a low-light image, can start or resume motion. In some embodiments, the autonomous machine can cut or mow while in motion. The autonomous machine can move for a limited time or distance, for example, until a positioning update is needed or requested.

[0120] During motion, the pose filter covariance may increase. As it increases, the pose filter covariance may approach or exceed a threshold, which may be used to determine whether to record a low-light image.

[0121] exist Figure 5, an example of the relationship between the vision system 202 and the navigation system 204 during autonomous machine navigation is shown. The pose estimation module 404 can provide an estimated pose, which can be referred to as a vision-based pose. The vision-based pose data from the pose estimation module 404 can be provided to the navigation system 204, returned to the pose filter 406, returned to the feature matching module 402, or any combination of these operations. In some embodiments, the pose data can also be transmitted directly to any controller, such as a controller of the navigation system 204. In some embodiments, a machine vision module can be used instead of the feature matching module 402 or in addition to the feature matching module 402, and the machine vision module can, for example, utilize a neural network or other suitable machine vision technology to facilitate pose estimation. The navigation system 204 can also provide commands or information to the vision system 202 to record images in low light conditions, such as using longer exposure and / or lighting.

[0122] The gesture filter 406 can use the gesture data to identify which gestures are possible and provide filtered gesture data. The filtered gesture data from the gesture filter 406 can be provided back to the feature matching module 402 to, for example, identify certain scenes in the possible scenes based on the filtered gesture data for feature matching with features in the captured image, which can promote faster overall feature matching. The filtered gesture data from the gesture filter 406 can be provided to the gesture estimation module 404 to, for example, identify certain gestures in the possible gestures based on the filtered gesture data, which can promote faster gesture estimation.

[0123] In some embodiments, the attitude filter 406 may use data from other sensors, such as an IMU, wheel encoders, GPS, or optical encoders (e.g., Figure 1 sensor 132), and previous posture estimation information to filter the posture. Figure 2 The navigation system 204 of the embodiment can use independent pose filters. The resultant outputs or pose data from different pose filters can be compared to correct any output or as a redundancy check for any output. In some aspects, the pose filter 406 can select specific scenes in the navigation map (e.g., less than all scenes in the navigation map) to feature match with features extracted from the low-light image. Using the pose filter 406 can facilitate faster vision-based pose estimation or positioning.

[0124] exist Figure 6 In the figure, a method for Figure 1 A schematic representation of an autonomous machine navigation process 440 of a machine.

[0125] In step 442, an uncertainty parameter corresponding to the current pose and the filtered pose data may be determined. The uncertainty parameter may be determined by a controller (e.g., a controller of the navigation system 204) or the feature matching module 402. A non-limiting example of an uncertainty parameter is a covariance determined using a pose filter. In step 444, it may be determined whether the uncertainty parameter exceeds a threshold. If the uncertainty parameter does not exceed the threshold, the process may return to step 442. If the uncertainty parameter does exceed the threshold, the process 440 may advance to step 446. In step 446, in response to the uncertainty parameter exceeding the threshold, the autonomous machine movement may be slowed down or stopped. Slowing down or stopping the autonomous machine may include providing a command to a propulsion system (e.g., the propulsion system 206). In some embodiments, the machine may slow down or stop at or near a natural path location. For example, in a containment zone or work area where the machine normally slows down or stops and turns in non-low light conditions, the machine may slow down or stop navigation at its next boundary. Using natural slowing down or stopping to capture long exposure images during navigation may facilitate faster coverage of the work area in low light conditions. At step 448, images of at least a portion of the work area may be recorded while the autonomous machine is slowed or stopped.

[0126] Figure 7 It is shown that the use Figure 1 Flowchart of an example of a method for navigating a lawn mower 100 in low light conditions. As shown, the method 700 includes factory photometric calibration of one or more cameras in block 702. The factory photometric calibration may be referred to as camera calibration data, which may be stored as camera configuration data in block 708 (e.g., in the lawn mower 100 ( Figure 1 ) in a memory of ). Generally speaking, the camera configuration data may include camera calibration data and / or camera capture configuration data.

[0127] The camera calibration data provided by the factory photometric calibration in block 702 may include calibrated irradiance mapping data. The calibrated irradiance mapping data may be used to convert pixel intensities captured by one or more cameras into irradiance.

[0128] Any suitable data structure may be used to map pixel intensity to irradiance. In some aspects, a lookup table (LUT) is used to convert pixel intensity to irradiance. Pixel intensity may be used to describe grayscale or color space, such as RGB. In some aspects, pixel intensity ranges from 0 to 255. Irradiance may be used to describe the radiant flux (power) received by a surface per unit area. In some aspects, irradiance is expressed in watts per square meter (W / m 2 ) is measured in units of .

[0129] The camera calibration data provided by the factory photometric calibration in block 702 may also include one or more camera capture parameters that may be used to capture image data. The camera capture parameters provided in conjunction with the factory photometric calibration in block 702 may also be referred to as initial, nominal, or factory default camera capture parameters. In some aspects, the one or more camera capture parameters may include one or more of exposure time, gain, and active lighting intensity. For example, the one or more illumination sources 140 ( Figure 1 ) provides active lighting intensity.

[0130] The method 700 may also include machine operation in the work area at block 704. During machine operation in the work area, the lawn mower 100 may move throughout the work area to perform various operations, such as mowing the lawn. In low light conditions, the lawn mower 100 may slow down or stop as needed to perform vision-based positioning. In some aspects, the lawn mower 100 may minimize the amount of time it slows down or stops during machine operation, for example, by configuring camera capture parameters during operation.

[0131] A camera capture configuration operation may be included in block 706 of method 700. The camera capture configuration operation in block 706 may be used to update one or more camera capture parameters, which may be referred to as camera capture configuration data and may be stored in the camera configuration data in block 708. The one or more camera capture parameters may be used to capture images in low light conditions. The camera configuration data stored in block 708 may also be used as an initial estimate for subsequent camera capture configuration data operations in block 706. The camera capture configuration operation in block 706 may be used during machine operation in a work area. In some aspects, the camera capture configuration operation may capture image data while the lawn mower 100 is moving or in motion. In some aspects, the camera capture configuration operation may also capture image data while the lawn mower 100 is slowing down or stopped.

[0132] Once the camera capture configuration operation in block 706 is completed, method 700 may return to the machine operation in the work area in block 704. Although one operation may be completed before starting another operation, in other embodiments (see Figure 8 ), the machine operation in block 704 and the camera capture configuration operation in block 706 may also be performed simultaneously or in other parallel ways.

[0133] Figure 8 It shows that you can use Figure 1Flow chart of an example of a method of machine operation of the lawn mower 100 in low light conditions. As shown, the method 800 of machine operation may include one or more processes for camera capture configuration operation in block 706 and one or more processes for machine operation in a work area in block 704.

[0134] The camera capture configuration operation may include machine image capture during machine operation in block 802. In particular, the lawn mower 100 ( Figure 1 ) may move or otherwise move through the work area while capturing images used by the camera capture configuration. The camera capture configuration operation may also include automatic camera capture configuration in block 804. In some cases, the automatic camera capture configuration may also be referred to as automatic exposure control (AEC). Generally speaking, the automatic camera capture configuration in block 804 may provide at least one camera capture parameter used to capture the image, which may be stored as camera capture configuration data in the camera configuration data in block 708, particularly for positioning. At least one previously determined camera capture parameter may also be used to capture an initial image in the automatic camera capture configuration.

[0135] The machine operation of block 704 may include machine navigation in block 806. The lawn mower 100 may use non-vision-based and vision-based navigation techniques, for example in combination, to navigate around the work area. In one aspect, the lawn mower 100 may continuously use dead reckoning to navigate the work area, which may be augmented by intermittent vision-based positioning, for example to correct the position of the lawn mower from time to time.

[0136] The machine operation may also include an uncertainty parameter calculation in block 808. The uncertainty parameter calculation may determine an uncertainty parameter and compare the uncertainty parameter to a threshold. In some aspects, the uncertainty parameter may be determined based on a posture filter covariance that may become large and may exceed a threshold. For example, the threshold may be determined based on an estimated position error from the posture filter covariance.

[0137] The uncertainty parameter may be compared to an uncertainty threshold in block 810. In some aspects, in response to the uncertainty parameter not exceeding the uncertainty threshold, the machine operation may be returned to machine navigation in block 806. In some aspects, in response to the uncertainty parameter exceeding the uncertainty threshold, the machine operation may be caused to take other actions.

[0138] In some aspects, the uncertainty threshold is determined based on the distance from the boundary of the work area. In particular, the closer the lawn mower 100 is to the boundary of the work area, the smaller or lower the uncertainty threshold can be (e.g., to tolerate less uncertainty) than when the lawn mower is far from the boundary of the work area.

[0139] Machine operation may include slowing or stopping the machine for positioning in block 812. In some aspects, the mower 100 may slow or stop and capture images to perform vision-based positioning, which may include feature extraction, feature matching, and pose estimation.

[0140] In block 814, images for positioning can be captured. The one or more cameras used to capture images for positioning can be the same cameras used to perform camera capture configuration. The one or more cameras can be configured using the camera configuration data in the camera configuration data stored in block 708. Typically, image capture for positioning can be on the order of a few seconds or tens of seconds in low light conditions.

[0141] An estimated update to the machine pose may be determined in block 816. The machine pose estimate may be updated based on the positioning from the image capture.

[0142] The uncertainty parameter may be checked in block 818. For example, the uncertainty parameter corresponding to the current pose estimate and the filtered pose data may be determined. In response to the uncertainty parameter not exceeding the uncertainty threshold, the machine operation may be returned to the machine navigation in block 806. In response to the uncertainty parameter exceeding the uncertainty threshold, the machine operation may be looped back to the positioning image capture in block 814. In some aspects, the image may be captured with one or more cameras different from those previously used in the positioning image capture.

[0143] Fig. 9 804 may include capturing an initial image (or captured image) using one or more initial camera capture parameters in block 902 to generate initial image data. Figure 1 ) can move while capturing images for camera capture configuration.

[0144] It may be determined in block 904 whether the captured scene parameters are acceptable. Acceptable captured scene parameters may also be referred to as "well exposed". The captured scene parameters may be determined based on initial image data of the initial image. The initial image may be evaluated using any suitable captured scene parameters. In some aspects, the captured scene parameters may include or correspond to an average value of pixel intensities in the initial image data. The captured scene parameters may be compared to a captured scene parameter threshold value (e.g., an arithmetic mean (or average) pixel intensity threshold value of other "good exposure metrics"). In one example, the captured scene parameters may be acceptable when they exceed the captured scene parameter threshold value. In another example, the captured scene parameters may be acceptable when they fall within at least two captured scene parameter threshold values ​​defining at least one acceptable range.

[0145] In response to determining that the captured scene parameters are acceptable, method 804 may provide an update to at least one camera capture parameter of the camera capture configuration data at block 926. The at least one updated camera capture parameter may be used to capture subsequent images for camera capture configuration or positioning.

[0146] In some aspects, the parameters determined for and / or used by one or more cameras can be balanced or prioritized according to any suitable heuristic. In one example, the one or more camera capture parameters or test camera capture parameters can provide at least a calculated exposure time. An exposure time threshold can be selected. The exposure time threshold can be selected based on a maximum amount of time allowed for camera exposure (e.g., 12 seconds). The exposure time determined by method 804 can be compared to the exposure time threshold.

[0147] In response to the calculated exposure time of at least one camera capture parameter exceeding the exposure time threshold, a reduced exposure time can be calculated for use by one or more cameras. Based on the amount of reduction in exposure time (from the calculated exposure time to the reduced exposure time), a gain can be increased or otherwise selected for use by one or more cameras. The balance of exposure time and gain can provide an effective exposure time that is equivalent to or nearly equivalent to the exposure time calculated by method 804.

[0148] In block 906, a first mask may be applied to the initial image to provide a masked initial image and generate masked initial image data. The method 804 may apply the first mask in response to determining that the captured scene parameters are not acceptable. In block 920, the first mask may be applied according to a first mask configuration that may be stored in a memory of the lawn mower 100.

[0149] Any suitable type of masking can be used for the first masking. In some aspects, the masking can include hard or soft edge clipping options. Hard edge clipping can be referred to as image clipping. Hard edge clipping can be particularly useful for removing known objects from an image, such as a lawn mower guard or a ground surface (e.g., grass). Soft edge clipping can be referred to as a geometric gradient toward the image boundary, and can also be referred to as a weighted masking. Any suitable type of geometric gradient or shape can be used for soft edge clipping.

[0150] In some cases, the mask used may be static or predetermined, while in other cases, the mask may be adaptive or adaptable based on rules. Adaptive masking can help evaluate areas of a scene. In some aspects, adaptive masking can establish lower weights around image areas where pixel intensity or irradiance is "saturated." For example, adaptive masking can help place a lower weight (e.g., ignore) on "saturated" pixel areas associated with a 255 intensity from a porch light, and place a lower weight on pixels near the "saturated" area. Because useful information may not be provided for the camera capture configuration from "empty" pixel areas associated with 0 intensity, adaptive masking can also place lower weights on these areas.

[0151] In block 908, an irradiance image may be generated based on the initial image data or the masked initial image data to provide irradiance image data. The irradiance image data may be generated based on calibrated irradiance map data, which may be provided in block 924 and may be used in block 708 ( Figure 7 ) is stored as camera calibration data in the camera configuration data of the memory of the lawn mower 100. The calibrated irradiance mapping data may include data for converting pixel intensity to pixel irradiance. Each pixel in the image data may be converted from pixel intensity to pixel irradiance in this manner. The calibrated irradiance mapping data may be constructed as a LUT.

[0152] One or more test camera capture parameters may be determined in block 910. The test camera capture parameters may be determined based on the irradiance image data. For example, if the average irradiance is above an average irradiance threshold, then one or more test camera capture parameters may be selected to provide a lesser exposure than one or more initial camera capture parameters used to capture the initial image. In some aspects, the previous test camera capture parameters (which may also be obtained from the camera capture configuration data stored in block 926) may be used to capture the initial image. Figure 7 The factory photometric calibration generated in block 702 of ) can be used as an initial estimate of the current test camera capture parameters.

[0153] A simulated image may be generated to provide simulated image data in block 912. The simulated image of the scene may represent an estimate of image data that would be captured if the initial image was captured using one or more test camera capture parameters instead of one or more initial camera capture parameters. In some aspects, the simulated image data is defined by pixel intensities.

[0154] In block 914, a second mask may be applied to the simulated image to provide a masked simulated image and generate masked simulated image data. The second mask may be the same as or different from the first mask and may utilize any mask available to the first mask. In block 922, the second mask may be applied according to a second mask configuration that may also be stored in a memory of the lawn mower 100. In some aspects, the first mask is a static mask for hard edge clipping and the second mask is an adaptive mask for soft edge clipping.

[0155] It can be determined in block 916 whether the simulated scene parameters are acceptable. Acceptable simulated scene parameters may also be referred to as "good exposure". The simulated scene parameters may be determined based on simulated image data or masked simulated image data. Any suitable simulated scene parameters may be used to evaluate the simulated image. In some aspects, the simulated scene parameters may include or correspond to the average value of the pixel intensity in the simulated image data. The simulated scene parameters may be compared with a simulated scene parameter threshold value (e.g., an arithmetic mean (or average) pixel intensity threshold value). In one example, the simulated scene parameters may be acceptable when they exceed the simulated scene parameter threshold value. In another example, the simulated scene parameters may be acceptable when they fall within at least two simulated scene parameter threshold values ​​defining at least one acceptable range.

[0156] In response to determining that the simulated scene parameters are acceptable, the method 804 may provide an update to at least one camera capture parameter of the camera capture configuration data in block 926. The at least one updated camera capture parameter may be used to capture subsequent images for camera capture configuration or positioning. In response to determining that the simulated scene parameters are unacceptable, the method 804 may return to determining one or more updated test camera capture parameters that are different from one or more previous test camera capture parameters.

[0157] Any suitable technique may be used to determine the updated test camera capture parameters. In some aspects, a binary search tree may be used to determine the next or updated test camera capture parameters. Method 804 may continue to iterate until the test camera capture parameters provide acceptable simulated scene parameters, until the algorithm runs out of time before the next positioning image capture, until a predetermined time has been exceeded, or until it is determined that solutions to find possible test camera capture parameters are divergent (e.g., not easily solvable). In these cases, non-vision-based techniques may be used to locate the lawn mower 100. In some aspects, GPS data may be used.

[0158] Generally speaking, during camera capture configuration, test or trial camera capture parameters may be selected to be larger or smaller, such as exposure times longer or shorter than previous camera capture parameters, to drive simulated scene parameters to target values. A reasonable search window for exposure times for an application, such as a lawn mowing application, is 30 microseconds to 12 seconds.

[0159] Active lighting intensity can also be balanced with exposure time and gain in a similar manner. One or more of the exposure time, gain, and active lighting intensity parameters can also be selected or limited by the camera control range of a particular machine (e.g., a camera with a maximum exposure time or gain or maximum active lighting power).

[0160] In some aspects, the effect of camera gain can be effectively used as a multiplier for exposure time. Typically, a high gain may not be required, and a very long exposure time may not be required. The two camera capture parameters can be balanced. In other words, the calculated or nominal exposure time can be converted into a combination of parameters. For example, if the selected exposure time is less than a threshold (e.g., 2 seconds), the gain can be set to 0 dB and the exposure time can be set to the calculated value. If the calculated exposure time is greater than the threshold, the exposure time can be reduced and set to the threshold (e.g., 2 seconds), and the gain can be selected to the correct value to effectively multiply the reduced exposure time by the calculated exposure time. For example, the calculated exposure time may be 8 seconds, and the resulting camera capture parameters may be an exposure time of 2 seconds and a gain of 12 dB.

[0161] Thus, various embodiments of autonomous machine navigation in low light conditions are disclosed. Although reference is made herein to a set of drawings that form a part of the present disclosure, at least one person of ordinary skill in the art will understand that various adaptations and modifications to the embodiments described herein are within or do not depart from the scope of the present disclosure. For example, various aspects of the embodiments described herein may be combined with one another in a variety of ways. Therefore, it should be understood that within the scope of the appended claims, the claimed invention may be implemented in ways other than those explicitly described herein.

[0162] All references and publications cited herein are expressly incorporated herein by reference in their entirety to the fullest extent permitted by law, unless in any respect they directly contradict the present disclosure.

[0163] Unless otherwise specified, all scientific and technical terms used herein have the meaning commonly used in the art. The definitions provided herein are to facilitate the understanding of certain terms frequently used herein and are not meant to limit the scope of the present disclosure.

[0164] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims may be understood to be modified by the terms “exactly” or “approximately.” Therefore, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and the attached claims are approximations and may vary depending on the desired properties sought to be obtained by one skilled in the art utilizing the teachings disclosed herein or, for example, within the typical range of experimental error.

[0165] The term "coupled" or "connected" refers to elements being attached to each other directly (directly in contact with each other) or indirectly (having one or more elements between and attaching the two elements). Either term may be replaced with "coupleable" or "connectable" to describe elements configured to be coupled or connected. In addition, either term may be modified with "operable" and "operably" (which may be used interchangeably) to describe coupling or connection configurations that allow the components to interact to perform a function.

[0166] As used herein, the term "configured to" may be used interchangeably with the terms "adapted to" or "constructed to," unless the content of the present disclosure clearly dictates otherwise.

[0167] The phrases "at least one," "including at least one," and "one or more" followed by a list refers to any one of the items in the list and any combination of two or more items in the list.

[0168] As used herein, "having," "having," "including," "comprising," "includes," "comprising," and the like are used in their open-ended sense and generally mean "including but not limited to." It should be understood that "consisting essentially of," "consisting of," and the like are encompassed by "comprising," and the like.

[0169] In general, the terms "aspects" and "embodiments" can be used interchangeably to describe one or more examples of the present disclosure. References to "a single embodiment," "one embodiment," "certain embodiments," or "some embodiments," etc., mean that at least one embodiment of the present disclosure includes a particular feature, configuration, composition, or characteristic described in conjunction with the embodiment (or aspect). Therefore, such phrases appearing in various places throughout do not necessarily refer to the same embodiment of the present disclosure. In addition, in one or more embodiments, particular features, configurations, compositions, or characteristics may be combined in any suitable manner.

Claims

1. A method for autonomous machine navigation, comprising: capturing image data of a scene using one or more cameras configured with at least one initial camera capture parameter, the at least one initial camera capture parameter comprising one or more of: exposure time, gain, and active lighting intensity; comparing the pixel intensity of the image to a pixel intensity threshold to determine whether the image data is well exposed; generating irradiance image data based on the image data of the scene and the calibrated irradiance map data to estimate scene brightness; determining test camera capture parameters based on the irradiance image data; generating simulated image data based on an estimate of image data that would be captured if the image data were captured using the test camera capture parameters instead of the at least one initial camera capture parameter; as well as In response to determining that the simulated pixel intensities are well exposed, at least one updated camera capture parameter is generated based on the test camera capture parameter.

2. The method according to claim 1, wherein: A calibrated irradiance map is generated by a calibration test performed on one or more image sensors or camera boards and stored on the machine as camera configuration data, wherein the calibration test measurements are generally applicable to all cameras of the model of the one or more image sensors or camera boards.

3. The method of claim 1 or 2, wherein determining the test camera capture parameters comprises: The test camera capture parameter is also determined based on at least one previously stored camera capture parameter.

4. The method according to claim 1, further comprising: responsive to determining that the image data is well exposed, capturing a positioning image using the at least one initial camera capture parameter; as well as In response to determining that the image data is not well-exposed, the irradiance image data is generated.

5. The method according to claim 4, further comprising: In response to the initial camera capture parameters exceeding a captured scene parameter threshold, it is determined that the initial camera capture parameters are acceptable.

6. The method of claim 1, wherein the pixel intensity comprises one or more of: an average pixel intensity, a median pixel intensity, or a weighted sum of pixel intensities.

7. The method according to claim 1, further comprising: generating masked captured image data based on the image data of the scene in response to determining that a portion of the image data has excessively high or low pixel intensities; as well as The irradiance image data is generated based on the masked captured image data.

8. The method according to claim 1, further comprising: generating masked simulated image data based on the simulated image data of the scene; as well as Simulated scene parameters are determined based on the masked simulated image data.

9. The method according to claim 1, further comprising: In response to a simulated scene parameter exceeding a simulated scene parameter threshold, the simulated image data is determined to be well-exposed.

10. The method of claim 1, wherein capturing the image data while the machine is in motion results in motion blur in the image.

11. The method according to claim 1, further comprising: in response to determining that the image data is not well-exposed, updating the test camera capture parameters; as well as Additional simulated image data of the scene is generated based on the updated test camera capture parameters.

12. The method of claim 1, wherein the test camera capture parameters include one or more of: exposure time, gain, and active lighting intensity.

13. The method of claim 1, wherein the test camera capture parameters include a calculated exposure time, the method further comprising: In response to the calculated exposure time exceeding an exposure time threshold, using a reduced exposure time that does not exceed the exposure time threshold as a camera capture parameter, and Gain is used as a camera capture parameter based on the calculated exposure time and the reduced exposure time.

14. The method according to claim 1, further comprising: In subsequent captures of image data using the one or more cameras, the at least one updated camera capture parameter is used to configure the one or more cameras or to update the positioning.

15. The method according to claim 1, further comprising: capturing the image data of the scene during movement of the autonomous machine in low light conditions in a work area; determining whether to update a position of the autonomous machine during movement of the autonomous machine; reducing movement of the autonomous machine in response to determining to update the position; capturing an operational image of at least a portion of the working area in the low-light condition while reducing motion of the autonomous machine using the at least one updated camera capture parameter; updating a pose estimate of the autonomous machine based on the captured operational images; as well as Motion of the autonomous machine in the work area is restored based on the updated pose estimate.

16. The method according to claim 1, further comprising: determining an uncertainty parameter based on the current posture; determining whether the uncertainty parameter exceeds an uncertainty threshold; In response to the uncertainty parameter exceeding the uncertainty threshold, reducing the motion of the autonomous machine and determining to update the position of the autonomous machine; and An operational image of at least a portion of a work area is captured while reducing motion of the autonomous machine. The method of claim 16 , wherein the uncertainty threshold is determined based on a distance from a boundary of the working area.

18. An autonomous machine comprising: a housing coupled to a maintenance tool; a propulsion system comprising at least one motor; at least one camera adapted to record images under one or more lighting conditions; as well as A controller operatively coupled to the at least one camera and the propulsion system, the controller being adapted to perform a method according to any one of the preceding claims.

19. A computer readable medium comprising instructions stored on the computer readable medium, the instructions, when executed by a processing circuit, causing the processing circuit to perform the method according to any one of claims 1 to 17.

Images