Techniques for volumetric estimation

Abstract

To provide a method which performs volumetric estimation and dimensional estimation of a pile of material or other object by autonomous machinery.SOLUTION: A method includes: a step 1302 which acquires sensor data comprising 2D or 3D representations of a physical environment from a plurality of sensors; a step 1306 which generates a 3D representation of an object in the physical environment by using one or more machine learning models of an object detection module; a step 1308 which estimates a volume of the object by using the generated 3D representation; and a step 1310 which communicates the volume estimate to a planning subsystem for performing operation of autonomous machinery based upon the volume estimate.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is (a) U.S. Provisional Patent Application No. 62 / 862,633, filed June 17, 2019, entitled “TECHNIQUES FOR VOLUMETRIC ESTIMATION”; (b) U.S. Provisional Patent Application No. 62 / 863,211, filed June 18, 2019, entitled “TECHNIQUES FOR VOLUMETRIC ESTIMATION”; and (c) U.S. Nonprovisional Patent Application No. 16 / 657,921, filed October 18, 2019, entitled “TECHNIQUES FOR VOLUMETRIC ESTIMATION” Application No. 16 / 657,921 claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 862,633 and U.S. Provisional Patent Application No. 62 / 863,211. The entire contents of U.S. Provisional Patent Application No. 62 / 862,633, U.S. Provisional Patent Application No. 62 / 863,211, and U.S. Non-Provisional Patent Application No. 16 / 657,921 are hereby incorporated by reference for all purposes.

[0002]

[0002] This disclosure relates generally to operation of autonomous machines to perform various tasks in various industrial work sites, and more particularly to techniques for estimating the volume and size of piles of material or other objects, and the use of sensors for estimating the volume and size of piles of material or other objects in such work sites. [Background technology]

[0003]

[0003] Currently, tasks involving the movement of piles or chunks of material between points on a construction or industrial job site are handled by time-consuming methods, such as complex processes requiring human labor and / or expensive equipment. The use of robotic equipment, such as robotic arms for picking and sorting materials at some sites, is unable to solve the complex problems associated with autonomously picking materials at one point and autonomously maneuvering to another point to unload such materials. Such equipment is also limited by field of view. Additionally, current robotic equipment is unable to accurately sense or estimate materials in a timely manner to accurately perform and complete tasks at industrial sites. Summary of the Invention

[0004]

[0004] The present disclosure relates generally to operation of autonomous machines to perform various tasks in various industrial worksites, and more particularly to techniques for volumetric and dimensional estimation of piles of material or other objects, and the use of sensors for volumetric and dimensional estimation of piles of material or other objects in such worksites. A framework for volumetric and dimensional estimation of piles of material or other objects is provided for autonomous machines to facilitate autonomously performing various tasks in various industrial worksite environments, applicable to various industries, such as construction, mining, manufacturing, warehousing, logistics, sorting, packaging, agriculture, etc.

[0005]

[0005] Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media having stored thereon programs, code, or instructions executable by one or more processors.

[0006]

[0006] In certain embodiments, techniques are described for performing volumetric estimation of a pile of material using one or more sensors on board an autonomous machine. The autonomous machine may be configured to perform an operation autonomously. The autonomous machine may perform an operation using an output from performing the volumetric estimation. The autonomous machine may also be configured to communicate the output of the volumetric estimation to another machine to enable the other autonomous machine to perform an operation autonomously. For example, in some embodiments, an autonomous machine performing a volumetric estimation may be part of a fleet of autonomous machines deployed at a work site to handle different types of tasks or to expedite the performance of the same task, where the task(s) involve interaction between the fleet of autonomous machines and the same pile of material.

[0007] In a particular embodiment, the system includes a plurality of sensors and a controller system coupled to the plurality of sensors. The controller system is configured to receive sensor data from the plurality of sensors. The sensor data includes a first representation of a physical environment and a second representation of the physical environment. Each of the first representation and the second representation is a two-dimensional (2D) representation or a three-dimensional (3D) representation including data associated with a deposit of material in the physical environment. The controller system is further configured to generate a 3D representation of the deposit of material. Generating the 3D representation of the deposit of material includes identifying an area in the first representation as corresponding to the deposit of material, identifying an area in the second representation as corresponding to the deposit of material, and determining a boundary of the deposit of material based on the identified areas in the first representation and the second representation. The controller system is further configured to estimate a volume of the deposit of material based on the 3D representation of the deposit of material and communicate information regarding the deposit of material to a planning subsystem of the autonomous machine. The information regarding the deposit of material includes the estimated volume. The communication of information about the pile of material allows the autonomous machine to autonomously perform an action on the pile of material. The action can be based on the estimated volume.

[0008]

[0008] In certain embodiments, the first representation in the processing performed by the controller system described above is an image generated using at least one camera, and identifying portions of the first representation as corresponding to deposits of material includes inputting the first representation into a machine learning model of the controller system trained to detect multiple classes of objects including the deposit class, and receiving as an output of the machine learning model a segmented representation of the physical environment including segments corresponding to the deposit classes.

[0009]

[0009] In a particular embodiment, the machine learning model is trained prior to receiving the sensor data by the controller system. Training the machine learning model includes obtaining a training dataset, the training dataset including at least one of a 2D or 3D representation of a real-world environment and labeled with information indicative of a class of objects represented in the training data, generating additional training data by image processing performed on the training dataset, and augmenting the training dataset, the training dataset including inputting the augmented training dataset to the machine learning model, and adjusting the machine learning model based on a comparison of inferences generated by the machine learning model using the augmented training dataset to ground truth information corresponding to correct inferences for the augmented training dataset.

[0010]

[0010] In certain embodiments, augmenting the training dataset includes at least one of the following types of image processing: scaling, blurring, rotating, color adjusting, resizing, changing image resolution, adding noise artifacts, or adding visual effects to simulate occlusion or weather conditions.

[0011]

[0011] In certain embodiments, the order in which training data from the augmented training dataset is input into the machine learning model is determined such that the order maintains a ratio between the number of instances of the first class of objects and the number of instances of the second class of objects.

[0012]

[0012] In a particular embodiment, the first representation in the processing performed by the controller system described above is an image generated using one or more cameras, and the second representation is a point cloud generated using a LIDAR or radar sensor. In such an embodiment, identifying an area in the second representation as corresponding to a deposit of material may include inputting the point cloud into a machine learning model trained to detect deposit shapes from the point cloud, and receiving a voxel representation of the detected deposit shapes as an output of the machine learning model. Furthermore, generating a 3D representation of the deposit of material may include generating an additional voxel representation by combining the voxel representation of the detected deposit shapes with the segmented image generated from the first representation, and smoothing the additional voxel representation by fitting a geometric mesh over the voxel representation.

[0013] In certain embodiments, identifying regions in the second representation as corresponding to the deposit of material includes inputting the second representation into a voxel estimation model configured to generate a voxel representation of the physical environment by estimating voxel heights in the voxel representation based on a geometry of points in the point cloud. Additionally, generating the 3D representation of the deposit of material may include inputting the voxel representation into a Bayesian filter configured to determine, for each estimated voxel height, a probability that the estimated voxel height is correct based on data from the first representation, which is a depth image, and selecting, by the Bayesian filter, between information from the depth image and information from the voxel representation to generate the 3D representation of the deposit of material based on the determined probability.

[0014] In a particular embodiment, a Bayesian filter that receives the voxel representation generated by estimating the voxel heights is configured to determine, for each estimated voxel height, a probability that the estimated voxel height is correct based on data from the first representation, which is a depth image. The Bayesian filter is further configured to select between information from the depth image and information from the voxel representation to generate a 3D representation of the deposit of material based on the determined probability.

[0015]

[0015] In certain embodiments, the Bayesian filter communicates feedback to the voxel estimation model that causes the voxel estimation model to modify the estimated voxel heights to more closely match the voxel heights indicated by the depth image. The depth image can be generated based on disparity between the image from the first camera and the image from the second camera. Additionally, the depth image can be segmented by a machine learning model of the controller system for input to the Bayesian filter.

[0016]

[0016] In certain embodiments, estimating the volume of the pile of material includes determining a 3D shape corresponding to approximately half of the pile of material bounded by the ground, back, and boundary of the pile of material, calculating the volume of the 3D shape, and estimating a volume of the pile of material that is twice the volume of the 3D shape.

[0017] In certain embodiments, the controller system receives sensor data representing the pile of material viewed from different sides of the pile and uses the sensor data to generate a 3D representation of the pile of material.

[0018]

[0018] In a particular embodiment, the method includes a controller system of the autonomous machine receiving sensor data from a plurality of sensors, the sensor data including a first representation of the physical environment and a second representation of the physical environment. The first representation and the second representation are generated using different types of sensors, and each of the first representation and the second representation is a two-dimensional (2D) or three-dimensional (3D) representation including data associated with a deposit of material in the physical environment. The method further includes the controller system estimating a volume of the deposit of material. Estimating the volume of the deposit of material includes inputting the first representation and the second representation to a machine learning model trained to detect and infer a volume of the deposit of material, and receiving as an output of the machine learning model a value of the volume of the deposit of material inferred from the first representation and the second representation. The method further includes the controller system communicating information regarding the deposit of material to a planning subsystem of the autonomous machine or a planning subsystem of the second autonomous machine. The information regarding the deposit of material includes a value of the volume of the deposit of material. Upon communication of information regarding the pile of material, an action is autonomously performed on the pile of material by the autonomous machine or a second autonomous machine.

[0019]

[0019] In certain embodiments, training the machine learning model, which subsequently receives the first and second representations as input, includes inputting a training representation set to the machine learning model, receiving an estimated volume value based on the training representation set, comparing the estimated volume value with a ground truth volume value of the training representation set, and adjusting the machine learning model based on a difference between the ground truth volume value and the estimated volume value based on the training representation set. The machine learning model may include multiple subsystems, each configured to perform a different one of combining representations from different sensor types into a single representation, detecting a deposit of material in the single representation, and estimating a volume of the deposit of material. When the machine learning model includes such multiple subsystems, training the machine learning model may include training each subsystem separately before performing end-to-end training of the machine learning model as a whole.

[0020]

[0020] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0021]

[0021] The present disclosure can be better understood with reference to the following description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0022] [Figure 1A] FIG. 1 is a high-level simplified block diagram of an autonomous vehicle incorporating a controller system (referred to herein as an autonomous vehicle management system (AVMS)) in accordance with certain embodiments. [Figure 1B] FIG. 1 illustrates an exemplary autonomous vehicle management system implemented primarily in software, according to some embodiments. [Figure 2A] FIG. 2 is a simplified block diagram illustrating subsystems of an autonomous vehicle management system in accordance with certain embodiments. [Figure 2B] FIG. 1 illustrates software modules (e.g., programs, code, or instructions executable by one or more processors of an autonomous machine) that may be used to implement various subsystems of an autonomous vehicle management system according to certain embodiments. [Diagram 3] FIG. 1 illustrates an example set of sensors for an autonomous machine in accordance with certain embodiments. [Figure 4] FIG. 1 is a simplified block diagram of a perception subsystem in an autonomous machine in accordance with certain embodiments. [Diagram 5] FIG. 1 illustrates example training images that can be used to train a machine learning model to detect objects, according to certain embodiments. [Figure 6] FIG. 1 illustrates an example segmented image generated by a trained machine learning model, in accordance with certain embodiments. [Figure 7] FIG. 1 illustrates an exemplary method for detecting deposits of material and estimating properties thereof based on fusion of heterogeneous representations of a physical environment and / or objects within the physical environment, generated using different types of sensors, in accordance with certain embodiments. [Figure 8] FIG. 1 illustrates an exemplary method for estimating the volume of a deposit of material based on a 3D representation of the deposit, according to certain embodiments. [Figure 9] FIG. 1 illustrates an exemplary method for reconstructing a deposit of material, in accordance with certain embodiments. [Figure 10] FIG. 1 illustrates an exemplary method for detecting material deposits and estimating their properties using a Bayesian filter that generates feedback to adjust a machine learning model, according to certain embodiments. [Figure 11] FIG. 1 illustrates an exemplary method for detecting and estimating properties of material deposits using a single machine learning model, according to certain embodiments. [Figure 12] 1 is a flowchart illustrating a process for training a machine learning model to detect objects and / or make inferences related to other objects, in accordance with certain embodiments. [Figure 13] 1 is a flowchart illustrating a process for performing volumetric estimation, in accordance with certain embodiments. [Figure 14] FIG. 1 is a simplified block diagram illustrating an example computing system that can be used to implement one or more of the systems and subsystems described in this disclosure and / or to perform any of the processes or methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023]

[0038] Next, typical examples and embodiments of the present disclosure will be described in detail with reference to the drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. In particular, the following figures and examples are not intended to limit the scope of the present disclosure to a single embodiment, and other embodiments are possible by replacing or combining some or all of the elements described or illustrated. Where convenient, the same reference numbers will be used throughout the drawings to refer to the same or similar parts.

[0024]

[0039] In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular inventive embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." An embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0025]

[0040] Where certain elements of these implementations can be implemented partially or fully using known components, only those portions of such known components necessary to understand the present disclosure will be described, and detailed descriptions of other portions of such known components will be omitted so as not to obscure the present disclosure.

[0026]

[0041] The present disclosure relates generally to operation of autonomous machines to perform various tasks in various industrial work sites, and more particularly to volumetric and dimensional estimation of piles of material or other objects, and the use of sensors for volumetric and dimensional estimation of piles of material or other objects in such work sites. Applications and frameworks are disclosed for volumetric and dimensional estimation of piles of material or other objects using at least one sensor, preferably multiple sensors, on autonomous machines (e.g., robotic machines and autonomous vehicles) in various work site environments applicable to various industries, such as construction, mining, manufacturing, warehousing, logistics, sorting, packaging, agriculture, etc.

[0027]

[0042] As used herein, a "pile of material" refers to one or more types of materials or objects that are stacked, overlapping one another, or otherwise organized to form a mass, such as, for example, a pile of material. The materials that make up a pile can be of various shapes and sizes, e.g., dirt, rocks, construction debris, etc. Pile materials also vary in shape, size, density, etc. For example, berms are a type of pile commonly found at mining sites. Berms often act as guardrails to prevent vehicles from traveling beyond a certain area, and typically have a trapezoidal or triangular cross-section.

[0028]

[0043] This disclosure describes some embodiments in the context of autonomous vehicles. The use of the term "vehicle" and the description of a vehicle are not intended to be limiting or restrictive. The teachings described herein can be used and applied to any autonomous installation, including autonomous vehicles and other types of autonomous machines configured to perform one or more tasks or operations in an automated manner, substantially free of human intervention.

[0029]

[0044] FIG. 1A is a high-level simplified block diagram of an autonomous vehicle 120 incorporating a controller system (referred to herein as an autonomous vehicle management system (AVMS) 122) according to certain embodiments. For purposes of this disclosure, an autonomous vehicle, such as autonomous vehicle 120, is a vehicle that can perform one or more operations autonomously, substantially without a human user or manual input. For example, in certain embodiments, an autonomous operation can be the ability of vehicle 120 to autonomously sense its environment and autonomously steer or drive along a path substantially without a human user or manual input. Other examples of autonomous operations include, but are not limited to, scooping operations, moving materials or objects (e.g., moving dirt from one area to another), lifting, driving, rolling materials, spreading mud, excavating, transporting materials or objects from one point to another, and the like.

[0030]

[0045] The autonomous vehicle 120 may be of a variety of different types. For example, the autonomous vehicle 120 may be an automobile or mobile machine that may be used to transport people and / or cargo. The autonomous vehicle 120 may be a specialized vehicle for performing specialized operations such as compacting roads or paths, rolling, digging, lifting, etc. Examples of the autonomous vehicle 120 include, but are not limited to, a wagon, a bicycle, a motor vehicle (e.g., motorcycle, automobile, truck, bus), a rail vehicle (e.g., train, tram), a watercraft (e.g., ship, boat), an aircraft, a spacecraft, and / or a heavy equipment vehicle (e.g., dump truck, tractor, bulldozer, excavator, forklift, etc.). Because the environment of the autonomous vehicle 120 may include other vehicles, including other autonomous vehicles, for purposes of clarity, the autonomous vehicle 120 may also be referred to as a self-vehicle to distinguish the autonomous vehicle 120 from other vehicles in its environment.

[0031]

[0046] In the following, various features are described using an autonomous vehicle as an example. However, this is not intended to be limiting. The teachings described herein can be applied to any machine capable of performing one or more autonomous operations. For example, these teachings can be used by loaders, compactors, etc. The machines can be used in various industries such as manufacturing, mining, construction, medical applications, packaging, assembly, surveying, mapping technology, logistics, etc.

[0032]

[0047] 1A , autonomous vehicle 120 may include or be coupled to sensors 110 and vehicle systems 112, in addition to an autonomous vehicle management system 122. Autonomous vehicle management system 122 may be communicatively coupled to sensors 110 and vehicle systems 112 via wired or wireless links. One or more different communication protocols may be used to facilitate communication between autonomous vehicle management system 122 and sensors 110, and between autonomous vehicle management system 122 and vehicle systems 112.

[0033]

[0048] Vehicle system 112 may include various electromechanical systems, components, linkages, etc. that enable autonomous vehicle 120 to perform its intended function, such as traveling or steering along a particular path or course. Vehicle system 112 may include, for example, steering systems, throttling systems, braking systems, propulsion systems, etc. for driving the autonomous vehicle, electrical systems, auxiliary systems (e.g., systems for outputting information to a driver or passengers of autonomous vehicle 120), etc. Vehicle system 112 may be used to set the path and speed of autonomous vehicle 120. In autonomous vehicles configured to perform specialized operations (e.g., dump trucks, tractors, etc. specialized to perform lift and empty operations), vehicle system 112 may also include systems configured to perform such specialized operations.

[0034]

[0049] The sensors 110 may be located on or within the autonomous vehicle 120 ("on-board sensors") or may be located remotely from the autonomous vehicle 120 ("remote sensors"). The autonomous vehicle management system 122 may be communicatively coupled to the remote sensors via wireless links using wireless communication protocols. The sensors 110 may obtain environmental information for the autonomous vehicle 120. This sensor data may then be provided to the autonomous vehicle management system 122. FIG. 3 illustrates an exemplary set of sensors 110 for an autonomous vehicle, including, but not limited to, a LIDAR (Light Detection and Ranging) sensor 302, a radar 304, a camera 306 (different types of cameras with different sensing capabilities may be used), a global positioning system (GPS) and an inertial measurement unit (IMU) sensor 308, a vehicle-to-everything (V2X) sensor 310, audio sensors, and the like. The sensors 110 may obtain (e.g., sense, capture) environmental information for the autonomous vehicle 120 and communicate the sensed or captured sensor data to the autonomous vehicle management system 122 for processing. Other sensors may include proximity sensors, SONAR sensors, and other sensors.

[0035]

[0050] Examples of radar sensors 304 (e.g., long-range radar, short-range radar, imaging radar, etc.) may include sensors used to detect objects in the environment of the autonomous vehicle 120 and determine the speed of the detected objects. Examples of LIDAR sensors 302 include sensors that use a surveying technique to measure distances to targets by using light in the form of pulsed laser light. This is done by illuminating the target to be measured with pulsed laser light and measuring the reflected pulses using a sensor. Examples of V2X sensors include sensors that communicate with moving parts of a transportation system using V2X communication technology. For example, the autonomous vehicle 120 may use V2X sensors to pass and / or receive information from the vehicle to another entity around or near the autonomous vehicle. V2X communication sensors / systems may incorporate other more specific types of communication infrastructure such as V2I (Vehicle-to-Infrastructure), V2V (Vehicle-to-Vehicle), V2P (Vehicle-to-Pedestrian), V2D (Vehicle-to-device), V2G (Vehicle-to-grid), etc. An IMU sensor may be an electronic device that uses a combination of accelerometers, gyroscopes, magnetometers, etc. to measure and report the specific forces, angular velocities, and possibly magnetic fields surrounding a body. A GPS sensor uses a space-based satellite navigation system to determine geographic location and time information.

[0036]

[0051] As described below, in certain embodiments, data obtained from different types of sensors or multiple instances of the same type of sensor may be combined or fused to form a three-dimensional (3D) representation of a material deposit or other object of interest from which one or more properties of the material deposit / object (e.g., the volume of the deposit) can be estimated. A combination of various types of sensors may be used to obtain data for generating the 3D representation. There are certain advantages to combining different sensor types. For example, cameras can generate very detailed images of the environment an object is in, whereas LIDAR and radar provide better depth perception. LIDAR is generally more accurate than radar at detecting stationary objects, whereas radar is more accurate at detecting moving objects.

[0037]

[0052] The autonomous vehicle management system 122 (also referred to as a controller system) is configured to process data describing the state of the autonomous vehicle 120 and the state of the autonomous vehicle's environment and, based on the processing, control one or more autonomous functions or actions of the autonomous vehicle 120. For example, the autonomous vehicle management system 122 may issue instructions / commands to the vehicle system 112 to programmatically and autonomously control various aspects of the autonomous vehicle's movement, such as the propulsion, braking, steering or steering, and auxiliary behavior (e.g., turning on lights) functions of the autonomous vehicle 120. The autonomous vehicle management system 122 implements control and planning algorithms that enable the autonomous vehicle 120 to autonomously perform one or more actions.

[0038]

[0053] The autonomous vehicle management system 122 may be implemented using only software, only hardware, or a combination thereof. The software may be stored on a non-transitory computer-readable medium (e.g., on a memory device) and executed by one or more processors (e.g., by a computer system) to perform its functions. In the embodiment shown in FIG. 1A, the autonomous vehicle management system 122 is shown as being within or on the autonomous vehicle 120. However, this is not intended to be limiting. In alternative embodiments, the autonomous vehicle management system 122 may be remote from the autonomous vehicle 120.

[0039]

[0054] The autonomous vehicle management system 122 receives sensor data from the sensors 110 periodically or on-demand. The autonomous vehicle management system 122 uses the sensor data received from the sensors 110 to perceive the surroundings and environment of the autonomous vehicle. The autonomous vehicle management system 122 uses the sensor data received from the sensors 110 to generate and keep up to date a digital model that encapsulates information about the state of the autonomous vehicle as well as information about the space and environment surrounding the autonomous vehicle 120. This digital model may be referred to as an internal map, and encapsulates the current state of the autonomous vehicle 120 and its environment. The internal map, along with other information, is then used by the autonomous vehicle management system 122 to make decisions regarding actions to be taken by the autonomous vehicle 120 (e.g., steering, braking, accelerating, skimming, clearing, etc.). The autonomous vehicle management system 122 may send instructions or commands to the vehicle system 112 to cause the systems of the vehicle system 112 to take actions.

[0040]

[0055] As described above, the autonomous vehicle management system 122 may be implemented using only software, only hardware, or a combination thereof. FIG. 1B illustrates an exemplary autonomous vehicle management system in which the autonomous vehicle management system 122 is implemented primarily in software, according to some embodiments. The autonomous vehicle management system 122 may be implemented as a full autonomous vehicle software stack 100. The full autonomous vehicle software stack 100 may include a vehicle safety manager 102, a remote interface manager 114, applications 104, middleware 106, and an operating system 108. The full autonomous vehicle software stack 100 may be used to implement the functionality of the various systems and subsystems described above.

[0041]

[0056] 2A is a simplified block diagram illustrating subsystems of an autonomous vehicle management system 122 according to certain embodiments. The autonomous vehicle management system 122 may comprise multiple systems or subsystems communicatively coupled to one another via one or more communication channels. In the embodiment shown in FIG. 2A, the subsystems include a sensor interface subsystem 210, a position estimation subsystem 202, a perception subsystem 204, a planning subsystem 206, a control subsystem 208, and an intelligence subsystem 212.

[0042]

[0057] The embodiment of the autonomous vehicle management system 122 shown in FIG. 2A is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Those skilled in the art will recognize many variations, alternatives, and modifications. For example, in some implementations, the autonomous vehicle management system 122 may have more or fewer subsystems or components than those shown in FIG. 2A, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems. The subsystems may be implemented using only software, only hardware, or a combination thereof. In the embodiment shown in FIG. 2A, the autonomous vehicle management system 122 and all of its subsystems are shown as being within or on the autonomous vehicle 120. However, this is not intended to be limiting. In alternative embodiments, all of the subsystems of the autonomous vehicle management system 122 or certain subsystems of the autonomous vehicle management system 122 may be remote from the autonomous vehicle 120.

[0043]

[0058] The sensor interface subsystem 210 provides an interface that enables communication between the sensors 110 (including on-board and remote sensors) and the autonomous vehicle management system 122. The sensor interface subsystem 210 may receive sensor data from the sensors 110 and provide the data to one or more other subsystems of the autonomous vehicle management system 122. For example, as shown in FIG. 2A, the sensor data may be provided to the position estimation subsystem 202 and the perception subsystem 204 for further processing. The sensor data collected by the various sensors 110 allows the autonomous vehicle management system 122 to build a picture or image of the autonomous vehicle 120 and its surrounding environment.

[0044]

[0059] In certain embodiments, the autonomous vehicle management system 122 enables one or more subsystems of the autonomous vehicle management system 122 to send instructions or commands to one or more sensors 110 to control the operation of the one or more sensors. For example, instructions may be sent to a particular sensor to modify the behavior of the particular sensor. For example, instructions may be sent to a sensor to modify the information sensed or collected by the sensor and / or to modify the sensor data communicated from the sensor to the autonomous vehicle management system 122. Using these instructions, the autonomous vehicle management system 122 can dynamically control the sensor data communicated from the sensor 110 to the autonomous vehicle management system 122. Further details regarding this are provided below in the context of functions performed by the planning subsystem 206.

[0045]

[0060] The position estimation subsystem 202 is configured to receive sensor data from the sensors 110 and determine the location of the autonomous vehicle 120 in its surrounding environment (vehicle position estimation) based on the sensor data. The position estimation subsystem 202 provides current local position information of the own vehicle relative to its environment (e.g., a mine). The position of the own vehicle 120 may be determined with respect to a predefined map generated by the perception subsystem 204. In certain embodiments, the position estimation subsystem 202 is configured to broadcast the own vehicle's position information to other systems or subsystems of the autonomous vehicle 120. The other systems or subsystems may then use the position information for their own processing as needed.

[0046]

[0061] The position estimation subsystem 202 may implement various functions such as internal map management, map matching, visual odometry, dead reckoning, point history management, etc. For example, assume that the autonomous vehicle 120 is operating in a mine. The position estimation subsystem 202 may receive as input a map of the mine. A mine typically has a set route that includes drivable and non-drivable areas, and set roads for the mining vehicle to follow around the mine. The position estimation subsystem 202 may determine the location of the ego vehicle along the route. The position estimation subsystem 202 may determine the location of the ego vehicle by utilizing multiple inputs received from sensors and a map of the environment. The position estimation subsystem 202 may use GPS sensor data to determine a global position of the ego vehicle. The position estimation subsystem 202 may receive the GPS sensor data and convert it into a more useful form that can be used by one or more other subsystems of the autonomous vehicle management system 122. For example, the information, position estimation subsystem 202 may identify where the vehicle is located relative to a map of the environment, such as a mine map (also called a map master).

[0047]

[0062] The location estimation subsystem 202 may also be configured to perform map matching, where what the location estimation subsystem 202 perceives is matched against information the location estimation subsystem 202 has. Map matching may match recorded geographic coordinates (e.g., using a geographic information system (GPS) or the like) against a logical model of the real world. In one example, a map matching algorithm may take a set of recorded points (e.g., from a GPS) and associate them (e.g., as a network) with edges in an existing street graph, which may be in a sorted list representing the autonomous vehicle's travels. As part of map matching, the location estimation subsystem 202 is tracking its vehicle in its environment and estimating its position based on what it sees against a map, such as a real-world map.

[0048]

[0063] The position estimation subsystem 202 is also configured to perform visual odometry, which includes determining the orientation and position of the ego-vehicle based on sensor data, such as by analyzing images captured by one or more cameras.

[0049]

[0064] The position estimation subsystem 202 may also perform dead reckoning processing. Dead reckoning is the process of calculating its own current position using a previously determined position, i.e., a navigational position, and advancing that position based on elapsed time and known or estimated speed over a course. This may include calculating the position of the own vehicle by estimating the direction and distance traveled. For example, the autonomous vehicle management system 122 receives and knows certain information about the autonomous vehicle 120, such as its wheel speed, steering angle, where the autonomous vehicle 120 was a second ago, etc. Based on the past position information, in combination with the speed / steering angle, etc., the position estimation subsystem 202 can determine the next or current location of the vehicle. This provides a local understanding of the own vehicle's position as it travels along its path. The path may be a road, highway, rail system, runway, seaway, bike path, etc., according to various embodiments.

[0050]

[0065] The position estimation subsystem 202 may also perform local history keeping tracking, where historical information about the vehicle's path is analyzed and compared to the current path. For example, if the autonomous vehicle 120 drives around a particular path in a mine many times, this information can be compared and analyzed by the position estimation subsystem 202.

[0051]

[0066] The localization subsystem 202 may also implement a consistency module configured to perform rationality checks, fault checks, normalization of sensor data, and the like. For example, the localization subsystem 202 may receive information from different information sources regarding the location, point, and the like of the vehicle. The rationality checks may be used to perform validity checks to ensure that information from various sensors is consistent and robust. This helps reduce erroneous results. The rationality checks may include tests to evaluate whether the sensor data values ​​and / or the results of the calculations can possibly be true. The localization subsystem 202 may also normalize the sensor data received from the sensors 110 and then provide the normalized sensor data to the localization subsystem 202. The localization subsystem 202 may then utilize the normalized sensor data to generate and / or update a consistent internal map of the autonomous vehicle's real-time environment (e.g., assuming networking latency, processing latency, and the like).

[0052]

[0067] The perception subsystem 204 periodically or on-demand receives sensor data from the sensors 110 and builds and maintains a consistent internal map based on the received information. The perception subsystem 204 may also receive input from other sources, such as the localization subsystem 202, and use the received input to build and maintain the internal map. The internal map generated by the perception subsystem 204 includes all information, including information about the vehicle, the state of the vehicle and its environment, and information about objects in the vehicle's environment (e.g., information about dynamic and static objects around the vehicle). The consistent internal map may be a local map of sensed entities / objects in the autonomous vehicle's environment, e.g., around the autonomous vehicle. In certain embodiments, these sensed entities / objects are depicted in three dimensions (3D). In certain embodiments, the perception subsystem 204 receives location information from the localization subsystem 202 and incorporates the location information into the internal map. The internal map may be maintained even if a sensor goes offline.

[0053]

[0068] Reasonableness checks and normalization may be performed on the sensor data received by the perception subsystem 204. These checks may include tests to evaluate whether the sensor data values ​​and / or the results of the calculations may possibly be true. The sensor data received from the sensors 110 may also be normalized and then the normalized sensor data may be provided to the perception subsystem 204. The perception subsystem 204 may then utilize the normalized sensor data to generate and / or update a consistent internal map of the autonomous vehicle's real-time environment.

[0054]

[0069] The perception subsystem 204 may use a variety of different algorithms and techniques, including artificial intelligence (AI) and machine learning based techniques, to perform its functions. For example, the perception subsystem 204 may use a convolutional neural network (CNN) to perform object detection and object classification based on the sensor data. During the training phase, the CNN may be trained using labeled training data including sample images of the vehicle's environment and corresponding ground truth classifications. The labeled data generally includes a set of samples tagged with one or more labels, where the labels represent known outcomes (e.g., ground truth classifications, etc.) of the training input samples. Labeling may also be used to take a set of unlabeled data and augment each piece of that unlabeled data with informative, meaningful tags. The CNN model or other AI / machine learning model built based on the training may then be used in real time to identify and classify objects in the autonomous vehicle 120's environment based on new sensor data received from the sensors 110.

[0055]

[0070] The planning subsystem 206 is configured to generate a motion plan for the autonomous vehicle 120. The plan may include one or more planned actions or operations to be performed by the autonomous vehicle 120. For example, the plan may include information identifying a trajectory or path to be traversed by the autonomous vehicle 120. The path may be a road, highway, rail system, runway, waterway, bike path, etc., according to various embodiments. For example, the trajectory information may indicate how the vehicle should travel from point A to point B with a list of points between points A and B that describe the trajectory for the vehicle to follow from point A to point B. As another example, the plan generated by the planning subsystem 206 may include planned actions for accessories of the autonomous vehicle 120, such as turning on or off turn signals or lights, emitting one or more sounds (e.g., an alarm), etc. In situations where the autonomous vehicle 120 has specialized components customized to perform specialized operations, the plan generated by the planning subsystem 206 may also include planned operations to be performed by one or more of these specialized components. For example, if the autonomous vehicle is an excavation truck with a bucket and arm assembly to perform digging and moving materials, the plan generated by the planning subsystem 206 may include operations to be performed by the bucket and arm assembly to perform the digging. For example, the plan may include the angle at which the arm should be raised and / or the angle of the bucket relative to the arm. After the motion plan is generated, the planning subsystem 206 may communicate the motion plan to the control subsystem 208, which may then control one or more systems of the vehicle system 112 to cause the autonomous vehicle 120 to perform the planned operations in the motion plan in a safe manner.

[0056]

[0071] In addition to the internal map generated by the perception subsystem 204, the planning subsystem 206 may also receive various other inputs for use in generating a motion plan for the autonomous vehicle 120. These inputs may include, but are not limited to: (a) location or location estimate information received from the location estimation subsystem 202; (b) information identifying one or more goals for the autonomous vehicle 120 (e.g., information identifying an ultimate goal for the autonomous vehicle 120 for making a right turn may be received). The goals may be set by an end user or operator of the autonomous vehicle or machine. In an automobile example, a user may set a high level to drive from a current point of the autonomous vehicle 120 to a particular ultimate destination. The autonomous vehicle 120 may determine a GPS route plan based on the current point and the ultimate destination point, with the goal to drive autonomously from the current point to the ultimate destination according to the GPS route plan. In an example mining environment, a high level goal set by an operator may be to move 10 tons of material (e.g., sand, coal, etc.) from point A and empty the material at point B. Generally, one or more different goals may be provided. Examples of goal categories (some of which may overlap) include, but are not limited to, goals related to the autonomous vehicle performing autonomous operations (e.g., autonomous driving or steering along a path, skimming, moving materials or objects, lifting materials, driving, rolling, spreading mud, excavating, transporting materials or objects from one point to another, etc.), goals related to the operation of the vehicle, goals related to the vehicle's interaction with various actors, objects, etc. in the vehicle's environment, goals related to the general operation of the vehicle, etc. Examples of goals are changing lanes, driving from one point to another, driving to a destination as fast as possible, making turns, taking a series of steps in sequence, etc. (c) High-level route information regarding the path or route to be taken by the autonomous vehicle 120. This may be provided directly or indirectly by an end user or operator of the autonomous vehicle. (d) Information identifying safety considerations.These may also be provided to the autonomous vehicle, such as by an end user / operator using an API provided by the autonomous vehicle 120, or via metadata configured for the autonomous vehicle 120. Examples of these considerations include, but are not limited to: always stay in lane, always keep a certain distance from any object, dump truck not turn more than 30 degrees, loader B not climb grades more than 15 degrees, etc. (e) Information about how a particular operation was performed in the past. For example, for a particular autonomous vehicle, this may be the past history of how that particular autonomous vehicle performed that operation in the past, how a different autonomous vehicle performed that operation in the past, how that operation was performed manually using the vehicle in the past (e.g., how a driver / operator performed that operation in the past with a vehicle operating under the control of the driver / operator). For example, an autonomous vehicle has driven a route in the past, how a manual truck drove this route, or how a particular task was completed, etc. (f) Other inputs.

[0057]

[0072] The planning subsystem 206 generates a motion plan for the autonomous vehicle 120 based on one or more inputs. The planning subsystem 206 may update the plan periodically as the environment of the autonomous vehicle 120 changes, as the objectives to be accomplished by the autonomous vehicle 120 change, or generally in response to changes in any of the inputs to the planning subsystem 206.

[0058]

[0073] As part of generating and updating a motion plan, the planning subsystem 206 makes various decisions regarding which motions should be included in the plan to achieve a particular goal in a safe manner. The processes performed by the planning subsystem 206 as part of making these decisions may include behavior planning, overall planning, path planning, fail-safe paths, path history tracking, etc.

[0059]

[0074] The planning subsystem 206 may generate and update motion plans using various AI-based machine learning algorithms to achieve the goal of performing a function or action (e.g., autonomous driving or maneuvering, mining an area) to be performed by the autonomous vehicle 120 in a safe manner. For example, in certain embodiments, the planning subsystem 206 may use a model trained using reinforcement learning (RL) to generate and update motion plans. The autonomous vehicle management system 122 may use the RL model to select actions to be performed to control the autonomous operation of the autonomous vehicle 120. The RL model may be updated periodically to increase its coverage and accuracy. Reinforcement learning (RL) is a field of machine learning inspired by behaviorist psychology that is concerned with what actions an agent should take in an environment to maximize some notion of cumulative reward.

[0060]

[0075] In certain embodiments, in addition to generating a motion plan, the planning subsystem 206 can dynamically control the behavior of the sensors 110. For example, the planning subsystem 206 can send instructions or commands to particular sensors among the sensors 110 to dynamically control the sensor data captured by the particular sensors and / or to control the sensor data communicated from the sensors to the perception subsystem 204 (or other subsystems of the autonomous vehicle management system 122, such as the localization subsystem 202). Because the internal map built by the perception subsystem 204 is based on the sensor data received by the perception subsystem 204 from the sensors, being able to dynamically control the sensor data received from the sensors also allows the planning subsystem 206 to dynamically control the information included in and / or used by the perception subsystem 204 to build and maintain the internal map. The planning subsystem 206 can dynamically and on-demand directly instruct the sensors 110 to obtain particular types of information or to behave in a specified manner, e.g., to provide additional sensor data to update the internal map with consistency. For example, the planning subsystem 206 may instruct the LIDAR sensor to narrow its sensing range from a 360 degree (360°) field of view to a smaller range that includes a particular object to be sensed and / or tracked in more detail by the LIDAR system. In this manner, a consistent internal map is updated under the control of the planning subsystem based on feedback from the planning subsystem 206.

[0061]

[0076] The autonomous vehicle management system 122 provides an infrastructure that allows the planning subsystem 206 (or other subsystems of the autonomous vehicle management system 122) to send one or more instructions or commands to one or more sensors to control the behavior of those one or more sensors. In the embodiment shown in FIG. 2A, the sensor interface subsystem 210 provides an interface for interacting with the sensor 110. The planning subsystem 206 can send instructions or commands to the sensor interface subsystem 210 in the outbound direction (from the autonomous vehicle management system 122 to the sensor). The sensor interface subsystem 210 is then configured to communicate the received instructions to the intended destination sensor. In the inbound direction (from the sensor to the autonomous vehicle management system 122), the sensor interface subsystem 210 can receive sensor data from the sensor in response to the instruction sent from the planning subsystem 206. The sensor interface subsystem 210 can then communicate the received sensor data to the planning subsystem 206 (or the appropriate subsystem of the autonomous vehicle management system 122 that originated the instruction).

[0062]

[0077] The sensor interface subsystem 210 may be able to communicate with different sensors using one or more different communication protocols. In particular embodiments, for an instruction or command received in the outbound direction from the planning subsystem 206 (or from any other subsystem of the autonomous vehicle management system 122) to be sent to a particular sensor, the sensor interface subsystem 210 may translate the instruction into a format understandable by and suitable for communicating with that particular sensor, and then use the particular communication protocol applicable to that particular sensor.

[0063]

[0078] In certain embodiments, the autonomous vehicle management system 122 can access information identifying the sensors 110 and their capabilities. Subsystems of the autonomous vehicle management system 122 can then access and use this stored information to determine the possible capabilities and behavior of the sensor and send instructions to the sensor to change its behavior. In certain embodiments, the sensor must be registered with the autonomous vehicle management system 122 before communication is allowed between the sensor and the autonomous vehicle management system 122. As part of the registration process, information related to the sensor may be provided for the sensor to be registered. This information may include information identifying the sensor, the sensor's sensing capabilities and behavior, the communication protocol(s) usable by the sensor, and other information related to the sensor. The autonomous vehicle management system 122 can then use this information to communicate with the sensor and control the behavior of the sensor.

[0064]

[0079] As described above, the planning subsystem 206 may send instructions to the sensors to control and change their behavior. Changing the behavior of the sensors may include changing the sensor data communicated from the sensors to the autonomous vehicle management system 122 (e.g., sensor data communicated from the sensors to the perception subsystem 204 or other subsystems of the autonomous vehicle management system 122), changing the data collected or sensed by the sensors, or a combination thereof. For example, changing the sensor data communicated from the sensors to the autonomous vehicle management system 122 may include communicating more or less data than was communicated from the sensors to the autonomous vehicle management system 122 prior to receiving the instructions, and / or changing the type of sensor data communicated from the sensors to the autonomous vehicle management system 122. In some cases, the data sensed or collected by the sensors may remain the same, but the sensor data communicated from the sensors to the autonomous vehicle management system 122 may change. In other cases, the data sensed or collected by the sensors themselves may be changed in response to the instructions received from the autonomous vehicle management system 122. The planning subsystem 206 may also be able to turn sensors on or off by sending appropriate commands to the sensors.

[0065]

[0080] For example, the planning subsystem 206 may receive inputs including a current internal map generated by the perception subsystem 204, position information from the position estimation subsystem 202, and a goal for the autonomous vehicle 120 to turn within a certain time (e.g., turn right in the next 5 seconds). As part of determining what is the best set of actions to be taken by the autonomous vehicle 120 to achieve the goal in a safe manner, the planning subsystem 206 may determine that specific sensor data (e.g., additional imagery) is needed that shows the environment to the right of the autonomous vehicle 120. The planning subsystem 206 may then determine one or more sensors (e.g., cameras) that can provide the specific sensor data (e.g., images of the environment to the right of the autonomous vehicle 120). The planning subsystem 206 may then send commands to these one or more sensors to change their behavior so that the one or more sensors capture and communicate the specific sensor data to the autonomous vehicle management system 122 (e.g., to the perception subsystem 204). The perception subsystem 204 may use this specific sensor data to update the internal map. The updated interior map may then be used by planning subsystem 206 to make decisions regarding appropriate actions to be included in the motion plan of autonomous vehicle 120. After a right turn is successfully performed by autonomous vehicle 120, planning subsystem 206 may send another command to autonomous vehicle management system 122 instructing the same camera(s) to return to communicating a different, possibly reduced, level of sensor data. In this manner, the sensor data used to build the interior map can be dynamically changed.

[0066]

[0081] Examples of changes in sensor behavior caused by commands received by the sensor from the autonomous vehicle management system 122 may include, but are not limited to, the following: The planning subsystem 206 may then determine which sensor or sensors (e.g., cameras) are responsible for transmitting the sensor data to be reduced. The planning subsystem 206 may then send instructions to these one or more sensors to change their behavior so that the amount of sensor data transmitted from these sensors to the autonomous vehicle management system 122 (e.g., the perception subsystem 204) is reduced. As an example, the instructions sent from the planning subsystem 206 may do one or more of the following: By having the sensor change its field of view, for example by having a camera or LIDAR sensor zoom in to a smaller spot. Having the sensor transmit only partial information. For example, the sensor may transmit less than all of the information captured by the sensor. Cause the sensor to transmit information faster or slower than before or at a normal rate. Turn on the sensor. The sensors capture and / or transmit information to the autonomous vehicle management system 122 at a different resolution or granularity than before.

[0067]

[0082] 2B illustrates software modules (e.g., programs, codes, or instructions executable by one or more processors of the autonomous vehicle 120) that may be used to implement various subsystems of the autonomous vehicle management system 122 according to certain embodiments. The software modules may be stored on a non-transitory computer medium. If necessary, one or more of the modules or executable images of the modules may be loaded into a system memory (e.g., RAM) and executed by one or more processors of the autonomous vehicle 120. In the example illustrated in FIG. 2B, software modules for implementing a localization subsystem 202, a perception subsystem 204, a planning subsystem 206, and a control subsystem 208 are illustrated.

[0068]

[0083] FIG. 4 is a simplified block diagram of a perception subsystem 400 in an autonomous machine (e.g., an autonomous vehicle 120) according to certain embodiments. The perception subsystem 400 can be used to implement the perception subsystem 204 of FIG. 2A. As shown in FIG. 4, the perception subsystem 400 may include a pre-processing module 410, an object detection module 420, and an estimation module 430. Each of the modules 410, 420, and 430 may be implemented in software only, hardware only, or a combination thereof. The perception subsystem 400 shown in FIG. 4 is merely an example. Those skilled in the art will appreciate many variations, alternatives, and modifications. For example, in some implementations, the perception subsystem 400 may have more or fewer subsystems or components than those shown in FIG. 4, may combine two or more subsystems, or may have a different configuration or arrangement of subsystems. Additionally, certain components of the perception subsystem have been intentionally omitted so as not to obscure the description of aspects of the perception subsystem related to volume estimation.

[0069]

[0084] The pre-processing module 410 is configured to condition and / or reformat the acquired sensor data in preparation for further processing by the object detection module 420. Formatting may include converting data generated by one sensor and data generated by a second sensor into a shared format and / or a shared reference frame. For example, the sensors may capture data at different rates (e.g., two cameras capturing data at different frames per second or a radar sensor operating at a different frequency than a LIDAR sensor). Thus, as part of the processing performed by the pre-processing module 410, data captured from sensors operating at different rates may be reformatted to allow the sensor data to be subsequently combined in a coherent manner, e.g., merging or grouping together data captured by different sensors but corresponding to the same time period. As another example, the sensors may be located in different locations (e.g., different positions on the body of an autonomous vehicle) and / or oriented differently (e.g., two cameras pointed in slightly different directions to generate a stereo image). If a first sensor captures an object at a particular location and a second sensor captures the same object at a different location (e.g., due to a different viewpoint of the second sensor relative to the first sensor), the pre-processing module 410 may perform geometric correction so that the object is represented in the sensor data from both sensors as a single object rather than two separate objects.

[0070]

[0085] Adjustment of sensor data may include any number of operations that improve the quality of the sensor data. Adjustments may vary depending on the type of sensor. For example, camera pre-processing may include image size or resolution adjustments (e.g., to reduce large images to smaller sizes for faster downstream processing) and corrective image processing (e.g., lens correction, aberration correction, white balance, aperture correction, etc.). Camera pre-processing may also include combining different images into a single image (e.g., as an average of a set of images). Other types of adjustment operations include operations to remove noise or unwanted information (e.g., cropping images, removing LIDAR data captured outside a particular field of view, removing data corresponding to objects or areas not of interest (e.g., ground), etc.).

[0071]

[0086] The pre-processing module 410 may also be configured to perform sensor calibration to modify sensor behavior and / or compensate for non-ideal sensor behavior. Examples of modifying LIDAR or radar sensor behavior include adjusting reflectivity parameters to modify the operating range of the LIDAR / radar sensor (e.g., to prevent data capture beyond a certain distance from the vehicle when an object of interest, such as debris, is known to be less than that distance) and modifying the field of view captured by the LIDAR / radar sensor (e.g., from 360 degrees to 270 degrees). An example of corrective camera calibration is estimating parameters of the lens and / or image sensor in the camera, allowing the estimated parameters to be used to correct lens distortion during subsequent image capture. Thus, pre-processing may include operations performed before and after capturing sensor data. Calibration may include internal calibration (e.g., adjusting the sensor behavior based on data captured by the same sensor) and / or external calibration (e.g., adjusting the sensor behavior based on data from another sensor).

[0072]

[0087] In a particular embodiment, the calibration of the camera includes calculating the extrinsic matrix of the camera. The extrinsic matrix is ​​a transformation matrix that represents the pose of the camera and contains values ​​indicating the geometric transformation (e.g., translation and / or rotation) required to map the camera's reference frame to some other reference frame (e.g., the reference frame of a LIDAR sensor). The extrinsic matrix can be calculated as a 3×4 matrix using a checkerboard calibration method, in which a 3D calibration device featuring a checkerboard pattern is placed in the field of view of the camera and then captured to determine matrix parameters that map points or features in the checkerboard image to corresponding points or features in the other reference frame. For example, the corners of the calibration device represented in the checkerboard image can be mapped to the corners of the calibration device represented in the point cloud generated by the LIDAR sensor. The calculation of the extrinsic matrix can be performed as a one-time setup that includes the use of a perspective-n-point (PnP) algorithm that estimates the camera pose given a set of n 3D points in the camera image and their corresponding two-dimensional (2D) projections. Once computed, the extrinsic matrix can be used to combine data from a camera with data from another sensor, for example, to merge 2D camera images with 3D data from other sensors (e.g., LIDAR point clouds), or to merge 2D camera images from two different cameras to form a depth image based on the disparity between the camera images.

[0073]

[0088] The object detection module 420 receives the preprocessed sensor data from the preprocessing module 410 and is configured to determine which portions of the sensor data correspond to deposits of material or some other class of object. The object detection module 420 may divide the sensor data into segments, each segment represented by an enclosed 2D or 3D boundary. For example, segmenting a 2D image captured by a camera may include generating a border around a group of pixels based on determining that the pixels belong to the same object. Segmentation is usually performed simultaneously with classification (determining the class of each segment). The process of dividing an input representation into segments of one or more classes may be referred to as semantic segmentation. Semantic segmentation may be viewed as forming a mask for filtering the input representation, where the mask includes shapes labeled according to the type of object to which the shapes correspond. LIDAR or radar data (e.g., 3D point clouds) may also be segmented, for example, by generating a 3D surface (e.g., a geometric mesh) that represents the boundaries of the objects. Segmentation can be done algorithmically (e.g., using a software algorithm that performs geometric calculations to generate the polygonal surface as a geometric mesh) or using machine learning (ML) models trained to infer object boundaries from sensor data.

[0074]

[0089] Object detection does not necessarily include identifying all objects captured in the sensor data. Instead, the object detection module 420 can be configured to detect only objects of interest, such as deposits of material. In certain embodiments, detection of objects of interest can be performed by an AI or ML model, e.g., a CNN trained to detect objects in a deposit shape. The object detection module 420 can combine detection techniques, with different detection techniques being applied depending on the source of the sensor data. For example, as described below in connection with the embodiments of Figures 7 and 10, multiple ML models can be used to detect deposits of material from data captured using different types of sensors. In the embodiment of Figure 7, the CNN segmentation model 722 performs segmentation of the image data captured by one or more cameras to generate a segmented 2D image (e.g., an image representative of deposits and other objects in a physical environment), and the voxel CNN model 726 generates a voxel representation of the object of interest based on the LIDAR data (e.g., a voxel representation of the same deposit represented in the segmented image output by the CNN segmentation model 722).

[0075]

[0090] When performing segmentation, the object detection module 420 may classify the segments according to a set of pre-defined categories or classes of objects. Examples of different types of classes are shown in FIG. 5. The types of classes recognized by the object detection module 420 may vary depending on the environment in which the autonomous machine is deployed. For example, an object detection module designed for mining equipment may be configured or trained to recognize segments corresponding to mud piles, terrain (e.g., dirt and grass), sky, humans, mining vehicles, traffic cones, etc.

[0076]

[0091] The object detection module 420 uses the sensor data received from the pre-processing module 410 to generate one or more representations of the physical environment and / or objects of interest. For example, the object detection module 420 may generate a segmented 2D image as a replacement for the 2D camera image, where pixels of the original 2D image are replaced with pixels corresponding to shapes defined by the boundaries identified during the segmentation process. The color values ​​of the pixels in the segmented image may be set according to the class of the object. For example, the outline of a pile may be filled with a color uniquely assigned to the pile class. In this way, the original image may be simplified by removing extraneous visual information while making objects of different classes more distinguishable. In some embodiments, the color of the pixels in the segments may be kept substantially the same as the original image, except that an outline is drawn around the boundary of the segment. The replacement image may be augmented with a label describing the object (e.g., text or metadata describing the class of the object). As another example, the object detection module 420 may use the LIDAR point cloud to generate a voxel representation of the pile (e.g., a 3D stack of cubes that approximates the shape of the pile indicated by the points in the point cloud).

[0077]

[0092] The estimation module 430 is configured to receive the heterogeneous representations from the object detection module 420 and combine the heterogeneous representations into a single representation for analysis by the estimation module 430 and / or as output for downstream processing. For example, the estimation module 430 may generate a 3D representation of the pile from the segmented image and the voxel representation, where the 3D representation represents the geometry of the pile more accurately than the segmented image or the voxel representation alone. In certain embodiments, the 3D representation generated by the estimation module 430 is a smoothed voxel representation or a digital elevation model (DEM) representing the surface of the object of interest (e.g., the pile). The 3D representation may include a grid of boxes and, for each box in the grid, information indicating the height of the box (e.g., the total number of voxels occupied between the ground and the top of the object at a particular point along the surface of the object). In certain embodiments, the 3D representation generated by the estimation module 430 is used to update the internal map described above in connection with the embodiment of FIG. 2a as including information about objects in the environment of the ego vehicle and information about the state of the ego vehicle.

[0078]

[0093] The estimation module 430 may use the 3D representation to estimate one or more properties of the object of interest. In certain embodiments, the processing performed by the estimation module 430 includes estimating dimensions of the pile of material and estimating the volume of the pile based on the estimated dimensions. As part of estimating the properties of the object of interest, the estimation module 430 may, in some embodiments, determine the quality of the object. For example, at a mining site, a berm serves as a safety barrier and / or an indication of a drivable area (e.g., two berms placed opposite each other along a route). Berms are typically half the height of a mining vehicle and ideally are well compacted without rounding the top of the berm. Berms formed as loose piles of mud or piles of mud that are not tall enough may be a potential hazard to vehicles or persons coming into contact with the berm, for example, during an accidental collision or while attempting to get over the berm. Thus, the estimation module 430 may output an indication as to whether the berm meets minimum requirements (e.g., with respect to height or density). As another example, the estimation module 430 may include a machine learning model trained to distinguish between fresh and old sediment material based on differences in LIDAR reflectivity data. Thus, the estimation module 430 may output an indication of the age of the sediment or an area within the sediment. Thus, in addition to determining a value for the volume of the sediment, the estimation module 430 may also generate quantitative and / or qualitative estimates of any number of other sediment properties.

[0079]

[0094] As described above, volumetric and dimensional estimates of a pile of material can facilitate the autonomous machine performing various tasks. For example, the estimation module 430 can communicate an indication of the estimated characteristics of the pile and / or the quality of the pile to the planning subsystem 206 of FIG. 2 to enable the planning subsystem 206 to generate a motion plan that may include, for example, maneuvering around the pile, skimming a certain amount of material from the pile and moving the skimmed material to another location, adding more material onto the pile, etc. In some embodiments, the estimation module 430 communicates to the planning subsystem 206 a 3D representation based on which the pile characteristics were estimated. The 3D representation enables the planning subsystem to take into account the geometry and / or location of the pile when developing a motion plan. In certain embodiments, the estimation module 430 can be configured to communicate the estimated characteristics and / or the 3D representation to another autonomous machine (e.g., a perception subsystem of another nearby autonomous vehicle) to enable the other autonomous machine to perform one or more tasks based on such information.

[0080]

[0095] In certain embodiments, the object detection module 420 and the estimation module 430 may be implemented on a high-speed computing platform for efficient processing of sensor data with high throughput. For example, the sensors and various computing resources (processors, memory, etc.) forming the object detection module 420 and the estimation module 430 may be coupled by one or more high-speed communication networks (e.g., Gigabit Ethernet) and may perform processing in a pipelined manner so that the sensor data is collected and processed in real time.

[0081]

[0096] In certain embodiments, the object detection module 420 includes one or more machine learning models. Each ML model in the object detection module 420 receives preprocessed sensor data as input. The ML models can receive one or more types of sensor data. For example, in the embodiment of FIG. 7, separate ML models are provided for camera data and LIDAR data. In the embodiment of FIG. 11, a single ML model receives camera data, LIDAR data, and radar data. The sensor data received by the ML models includes a 2D or 3D representation of the physical environment captured by the sensor. For example, the sensor data may include a 2D image generated by a camera or a 3D point cloud generated by a LIDAR sensor. The ML models can be trained to generate inferences based on the sensor data. For example, the ML model may infer from the color values ​​of each pixel in the 2D image which areas of the image correspond to a deposit or other object of interest. The ML model may output inferences in the form of an object alone or additional 2D or 3D representations of objects in the environment. In certain embodiments, such as those of Figures 7 and 10, the ML model is trained to infer boundaries of different classes of objects based on color information from a camera image and output a segmented image showing the boundary of each object detected by the ML model. Training generally involves providing a set of training representations as input to the ML model, comparing the resulting guesses with ground truth information corresponding to correct guesses, and adjusting the behavior of the ML model based on the difference between the actual guesses and the correct guesses.

[0082]

[0097] FIG. 5 illustrates an example training image 500 (e.g., a photo of a real-world environment) that can be used to train a machine learning model to detect objects, according to certain embodiments. The training image 500 is an example of two-dimensional training data. However, it will be appreciated that if the machine learning model operates on 3D data, 3D training data (e.g., a point cloud) can also be used for training. For example, a point cloud representing multiple types of objects can be used to train the ML model to recognize those types of objects. The ML model can also be trained with data representing a single object. For example, to train a CNN to detect piles, the CNN can be provided with a point cloud of the shape of the pile.

[0083]

[0098] As shown in FIG. 5, the training image 500 includes labeled instances of various classes of objects, such as a first sky 502, a second sky 504, a building 506, water 508, a first terrain 510 (e.g., paved or unpaved ground), a second terrain 512, mud 514 (e.g., a first mud pile), mud 516 (e.g., a second mud pile), a traffic cone 518, a pole 520, and an unknown object 530. In the embodiment of FIG. 5, the mud object 514 and the mud object 516 are not explicitly labeled as being piles. However, in other embodiments, there may be a separate pile class, such that, for example, a mud pile is simultaneously labeled as being mud and a pile. This allows the ML model to distinguish between mud in pile form and mud that is not a pile form, such as unconsolidated mud around the base of a pile. The training data may be selected to include objects of types expected to be encountered by an autonomous machine on which the machine learning model will be deployed or used.

[0084]

[0099] The training images 500 may be accompanied by information indicating the boundary of each object. For example, as shown in FIG. 5, each object is annotated with a contour that surrounds the object. In certain embodiments, the boundary is represented as a 2D polygon. In a 3D representation, the boundary can be represented as a geometric mesh (e.g., a mesh of interconnected polygons) that defines a 3D surface. The boundary can also be indicated non-visually, for example, using metadata in a computer-readable file that encodes the training images 500, where the metadata lists the coordinates of pixels that correspond to the boundary of the object.

[0085]

[0100] The training images 500 are also accompanied by labeling information 550 that tags each object with a label indicating the object's class. As shown in FIG. 5, the labeling information 550 may also include statistical information about the objects represented in the training images 500, such as the number of each object class. The labeling information 550 may be provided, for example, as metadata in the training image 500 file or as a separate file. In certain embodiments, the granularity of the labeling of the training data is adjusted (e.g., labeling at the pixel level vs. labeling at the macroblock level). For example, coarse labeling can be applied to the training data to reduce the amount of processing and training data. Labeling that is too fine can potentially lead to overfitting of the machine learning model, while labeling that is too coarse can lead to overgeneralization and therefore poor fit.

[0086]

[0101] Machine learning models are typically trained on large training data sets (e.g., thousands of images). Each item of training data (e.g., training image) may be parsed for further processing by the machine learning model. For example, if the machine learning model is implemented as a neural network, relevant features from the items of training data may be extracted by the neural network or a separate machine learning model to generate a feature set for input to an input layer of the neural network. The machine learning model processes the values ​​of these features to generate inferences about what the features represent. For example, the machine learning model may infer the probability that a pixel in the training image 500 corresponds to a particular object class based on the pixel's color value. This classification result may then be compared to a ground truth classification (e.g., the labeled classes for the training image 500), and the behavior of the machine learning model may be adjusted according to the difference between the corresponding ground truth classification and the classification result generated by the machine learning model. For example, this difference may be backpropagated to adjust the weights of one or more mathematical functions used by the machine learning model to generate the classification result. The machine learning model may calculate the degree of adjustment to minimize a loss function that represents this difference. In this way, a machine learning model can be trained to classify objects, for example to recognize the presence of deposits of material in newly observed (untrained) images.

[0087]

[0102] The classification results generated by the machine learning model may be output in the form of a segmented representation, for example, the machine learning model may be configured to generate 2D images with a border around each object, for example as shown in FIG.

[0088]

[0103] Training may occur locally on a computing device or system (e.g., one or more processors forming the perception subsystem 204 of FIG. 2) on which the machine learning model will be deployed. The computing device or system may include multiple general-purpose processors (e.g., a cluster of central processing units (CPUs)). Alternatively or additionally, for the large amount of graphics-related calculations performed by the machine learning model, the computing device or system may include special-purpose processors (e.g., a cluster of graphic processing units (GPUs)). These processors may operate in parallel, e.g., by processing different portions of the training data simultaneously, to expedite the training of the machine learning model. In some embodiments, training may occur remotely, e.g., on a cloud server. Remote training may leverage additional computing resources beyond those available locally and is therefore useful when the amount of training data to be processed makes training locally impractical.

[0089]

[0104] In certain embodiments, training begins with the machine learning model being configured according to a pre-training model. This allows the machine learning model to leverage existing capabilities derived from previous training efforts, e.g., training performed by other computing systems using training data different from the training data on which the machine learning model will be trained. For example, machine learning models exist that are configured to detect certain classes of objects (e.g., people, certain types of vehicles, buildings, etc.) that may be related to the general operation of autonomous vehicles. Although such existing models are insufficient to provide the object detection and estimation capabilities described herein, such models can nevertheless be used as a starting point for training so that the training process can be completed more quickly. Thus, the use of a pre-training model can speed up the speed at which training converges to the final state of the machine learning model.

[0090]

[0105] In certain embodiments, the training data is presented to the machine learning model being trained in a particular order configured to achieve a balanced representation of object classes. For example, the training images can be selected such that there are at least a certain number of instances of a particular class of object in each image or across several images. Additionally, the training images can be selected to maintain a certain ratio of one object class to another. Maintaining such class balance helps to avoid the problem of the machine learning model "forgetting" a previously trained class due to a disproportionate amount of another object class present in the later training data. For example, a neural network initially trained exclusively on sediments will tend to perform well with respect to detecting sediments, but once the neural network is no longer provided with training instances of sediments or is provided with under-instances of sediments, the performance of the neural network with respect to sediments may begin to degrade as the neural network attempts to configure its behavior for other classes (e.g., landforms) represented in the later training data. Providing a balanced representation of object classes allows the machine learning model to perform well in a variety of settings, since real-world environments are rarely composed of only one type of object.

[0091]

[0106] To determine when to terminate training, the machine learning model can be tested by inputting validation data into the machine learning model and comparing the results generated by the machine learning model with the ground truth information of the validation data. For example, the validation data can include a subset of the pool of training data reserved for validation and not training purposes. Training and validation can be performed in separate phases. In certain embodiments, the training and validation phases are terminated based on determining that the machine learning model has reached a certain level of accuracy. For example, the accuracy of prediction / inference for a particular object class or an entire set of object classes can be specified as a threshold (e.g., average classification error). Thus, the threshold corresponds to a performance criterion. The threshold for validation can be, but need not be, the same as the threshold for training. When the results generated by the machine learning model meet this performance criterion, training can be terminated. If not, the machine learning model can be further trained using additional training data.

[0092]

[0107] FIG. 6 illustrates an exemplary segmented image 600 generated by a trained machine learning model, according to certain embodiments. The segmented image 600 represents a 2D image generated by a machine learning model trained using training data representing various object classes illustrated in FIG. 6. The segmented image 600 may be generated from an input image (e.g., an image captured by a camera and then preprocessed to correct and / or reformat the image). As illustrated in FIG. 6, the segmented image 600 is segmented into different regions corresponding to the terrain 602, the sky 604, a pile of mud 606, a person 608, a second instance of mud 610 (e.g., a mud hole), and a bucket 612 forming part of an autonomous vehicle in which a sensor is mounted that captured the input image. For each segment, the machine learning model may have determined (e.g., by inference) a probability (e.g., average probability) that the object represented by that segment belongs to a particular class. As mentioned above, the segmented image may be used as a surrogate for the image from which the segmented image was generated and may use a particular color scheme (e.g., having a unique color for each object class) that allows the objects to be easily distinguished. Segmentation may include semantic segmentation (e.g., such that objects of the same class are labeled without necessarily distinguishing between different instances of the same class) and / or instance segmentation (e.g., such that different instances of the same class are labeled separately, e.g., deposit #1, deposit #2, etc.).

[0093]

[0108] FIG. 7 illustrates an exemplary method for detecting material deposits and estimating properties based on fusion of heterogeneous representations of a physical environment and / or objects in a physical environment, generated using different types of sensors, according to certain embodiments. The method of FIG. 7 is illustrated with reference to a particular arrangement of components that can implement the method. As illustrated in FIG. 7, these components can include multiple sensors (e.g., a camera 702, a camera 704, a LIDAR sensor 706, and a radar sensor 708), a pre-processing module 710, an object detection module 720, and an estimation module 730. The number of sensors can vary. For example, in some embodiments, there can be only one camera (e.g., a single camera and a single LIDAR sensor, but no radar) or even no cameras (e.g., a LIDAR sensor + radar sensor or an imaging radar sensor with no LIDAR or camera sensors).

[0094]

[0109] The pre-processing module 710 may correspond to the pre-processing module 410 of FIG. 4. Each of the sensors is communicatively coupled to a respective pre-processing unit in the pre-processing module 710. For example, the camera 702 may be configured to provide image data to the pre-processing unit 712, the camera 704 may be configured to provide image data to the pre-processing unit 714, the LIDAR sensor 706 may be configured to provide LIDAR data to the pre-processing unit 716, and the radar sensor 708 may be configured to provide radar data to the pre-processing unit 718. As described above in connection with the embodiment of FIG. 4, pre-processing may include various post-capture and / or pre-capture operations to condition or format data from different sensors and to calibrate the sensors. For brevity, the discussion of FIG. 7 will not repeat the description of pre-processing.

[0095]

[0110] The object detection module 720 may correspond to the object detection module 420 of Figure 4. The object detection module 720 includes a CNN segmentation model 722, a geometric model 724, and a voxel CNN model 726. The CNN segmentation model 722 and the voxel CNN model 726 are examples of machine learning models implemented as neural networks. However, other types of machine learning models (e.g., models implemented using different types of neural networks) can be used in place of the CNN of Figure 7.

[0096]

[0111] The CNN segmentation model 722 receives preprocessed data generated by the preprocessing unit 712 using data from the camera 702. The preprocessed camera data may include 2D images. The CNN segmentation model 722 is configured (e.g., trained) to segment the 2D images and classify objects contained in the 2D images to provide a segmented version of the 2D images as input to the fusion module 732 of the estimation module 730. For example, the CNN segmentation model 722 may output an RGB (red, green, blue) formatted 2D image that is segmented and labeled with the class of each object identified in the 2D image received from the preprocessing unit 712. Similarly, the CNN segmentation model 722 may generate a segmented image based on an image derived from operations performed by the preprocessing unit 714 on the data from the camera 704.

[0097]

[0112] Both the geometric model 724 and the voxel CNN model 726 receive preprocessed data generated by the preprocessing unit 716 using the output of the LIDAR sensor 706. The output of the preprocessing unit 716 may include, for example, a point cloud represented in a 3D coordinate system. The geometric model 724 is configured to perform geometric processing to segment the preprocessed LIDAR data using computer vision techniques. The segmentation performed by the geometric model 724 may be implemented algorithmically, for example, using one or more software algorithms that distinguish between points associated with a volume (e.g., a 3D object) and points not associated with a volume (e.g., a flat terrain). Thus, the geometric model 724 does not need to be trained to perform the segmentation. Instead, the geometric model 724 may simply generate a geometric mesh, for example, by drawing a polygonal surface using polynomial fitting and Poisson surface reconstruction with points having a specific geometric relationship to each other. In a particular embodiment, the output of the geometric model 724 includes a geometric mesh that defines a 2D plane that corresponds to the ground in the 3D coordinate system. As described below in connection with FIG. 8, the ground surface can be used to determine a 3D volume (enclosed shape) from which the volume of the pile can be estimated.

[0098]

[0113] The voxel CNN model 726 receives the preprocessed data generated by the preprocessing unit 716 using data from the LIDAR sensor 706. The preprocessed LIDAR data may include a 3D representation of the physical environment (e.g., a point cloud in an x, y, z coordinate system). The voxel CNN model 726 is configured (e.g., trained) to detect any deposits represented in the preprocessed LIDAR data and provide a voxel representation of each deposit detected in the preprocessed LIDAR data as an input to the fusion module 732 of the estimation module 730. For example, the voxel CNN model 726 may have been trained using point cloud data representing objects of a deposit shape. Based on this training, the voxel CNN model 726 uses features extracted from the preprocessed LIDAR data to infer the general boundary of each object of the deposit shape. Thus, the voxel representation is a rough approximation of the shape of the deposit in 3D space. The voxel CNN model 726 may or may not be able to distinguish between the two pile-shaped objects (performing instance segmentation). For example, the voxel CNN model 726 may simply be trained to output a voxel representation of any pile-shaped object detected. In the case of multiple piles, the voxel representation output by the voxel CNN model 726 may be a composite of multiple pile-shaped voxels.

[0099]

[0114] Although not depicted as being coupled to a machine learning model, the output of a pre-processing unit 718 associated with the radar sensor 708 may in some embodiments be processed through a machine learning model trained to detect deposits and / or other objects from the pre-processed radar data. For example, a separate voxel CNN model may be provided as an input to the fusion module 732 to generate a voxel representation of deposits based on the point cloud generated using the radar sensor 708.

[0100]

[0115] The estimation module 730 may correspond to the estimation module 430 of Figure 4. In addition to the fusion module 732, the estimation module 730 includes a voxel reconstruction module 734 and a volume calculation module 736.

[0101]

[0116] The fusion module 732 is configured to combine the various 2D and 3D information provided by the CNN segmentation model 722, the geometric model 724, and the voxel CNN model 726 into a single 3D representation of the object, e.g., a colored voxel representation of the deposit. The 3D representation generated by the fusion module 732 provides more information about the object than any of the input representations provided to the fusion module. For example, the fusion module 732 may combine color information from the segmented 2D image with shape information from the voxel representation. The fusion may include determining the color of the entire surface of the object by projecting the 3D data onto the 2D data, e.g., from the plane of the LIDAR sensor 706 to the plane of the camera 702, and projecting back from the 2D data to the 3D data. This allows the fusion module 732 to better determine the deposit / object boundary based on the recognition that voxels that map to a certain color should not be considered part of the deposit / object. For example, a person standing next to a pile may be lumped into a set of voxels derived from the LIDAR point cloud, and voxels associated with the person may be filtered out from the voxel representation of the pile based on color differences between the person and the pile. The fusion module 732 may also incorporate the output of the geometric model 724 and / or preprocessed radar data into the combined 3D representation.

[0102]

[0117] The 3D representation generated by the fusion module 732 is refined by the voxel reconstruction module 734 to generate a more accurate 3D representation of the deposit or other object of interest. In certain embodiments, the voxel reconstruction module 734 reconstructs the object of interest (e.g., the deposit) based on the 3D representation generated by the fusion module 732 as a smooth surface with boundary edges, e.g., the deposit surface shown in FIG. 8 and FIG. 9. The reconstruction can be performed using techniques similar to those described above in connection with the geometric model 724. For example, the voxel reconstruction module 734 may reconstruct the deposit by fitting a mesh over the voxels in the 3D representation generated by the fusion module 732. The mesh operates to make the transition between adjacent points on the surface of the deposit more gradual. The mesh also fills any holes in the 3D representation (e.g., hollow areas on the surface of the deposit due to noise, sensor error, incorrect guesses by the CNN segmentation model 722 or the voxel CNN model 726, etc.). The output of the Voxel Reconstruction module 734 is a smooth, colored 3D representation of the object of interest.

[0103]

[0118] An exemplary method for reconstructing a pile, according to certain embodiments, is illustrated in Figure 9. Once the dimensions and boundaries of the pile 900 have been determined in the form of a voxel representation including a number of voxels 902 (only a portion of the voxels are shown in the figure for simplicity), as illustrated in Figure 9, a smooth surface 904 can be defined that better approximates the shape of the pile and fills in any holes in the voxel representation, as described above.

[0104]

[0119] In addition, as shown in FIG. 9, in certain embodiments, the pile is reconstructed based on sensor data captured from different perspectives. For example, sensor data may be captured from an autonomous vehicle 910 at different positions around the pile (e.g., positions P1, P2, and P3). By acquiring sensor data from different positions (e.g., positions over a range of 270 degrees or more), more information about the pile 900 becomes available, thus improving the accuracy of the volume estimation. For example, when the autonomous vehicle 910 is at position P3, the pile 900 may be determined to be hollow, asymmetric, or discontinuous on the side facing the sensor, but from the perspectives of positions P1 and P2, the pile 900 may appear to be completely solid, symmetric, and continuous. Various views of the pile 900 captured at different positions may be combined (e.g., by stitching together images of the pile, aggregating point clouds, etc.). In certain embodiments, such as the embodiment of FIG. 7, the combination of different views may be performed by an estimation module (e.g., fusion module 732 in estimation module 730). The reconstruction and multi-view data ingestion techniques shown in FIG. 9 may be applied to any of the estimation methods described herein, including the methods described below in connection with the embodiments of FIG. 10 and FIG. 11. In certain embodiments, the combination of sensor data from different viewpoints is performed using a long short-term memory (LSTM) neural network. The LSTM receives multiple sensor data sets, each captured from a particular viewpoint and including data from multiple sensors. The LSTM generates a 3D representation of the pile / object and updates the 3D representation each time a new sensor data set is input to the LSTM.

[0105]

[0120] 7, the volume calculation module 736 is configured to estimate a volume of the object of interest (e.g., the deposit 900 of FIG. 9) based on the 3D representation generated by the voxel reconstruction module 734. The volume estimation performed by the volume calculation module 736 may include, for example, slicing the 3D representation generated by the voxel reconstruction module 734 into a number of 3D slices and calculating a volume integral over each slice based on the number of voxels that fit within each slice, with each voxel representing a particular physical volume (e.g., a 2 cubic centimeter cube).

[0106]

[0121] 7, the volume calculation module 736 may output (e.g., to the planning subsystem 206 of FIG. 2) pile information 750. The pile information 750 may include an estimated volume of the pile along with other information determined by the estimation module 730 (e.g., estimated dimensions of the pile, a 3D representation generated by the voxel reconstruction module 734, etc.). The pile information 750 may be used to generate a motion plan for the pile, such as, for example, driving around the pile, removing a certain amount of material from the pile, etc.

[0107]

[0122] FIG. 8 illustrates an exemplary method for estimating the volume of a deposit of material based on a 3D representation of the deposit (e.g., a 3D representation generated by the voxel reconstruction module 734) according to certain embodiments. The method illustrated in FIG. 8 is not limited to the embodiment of FIG. 7 and can also be used to perform volume estimation in other embodiments, e.g., the embodiment of FIG. 10. As illustrated in FIG. 8, a 3D representation 800 of the deposit (e.g., a smooth 3D representation generated by the voxel reconstruction module 734) can be divided approximately in half by placing a plane through points A, B, C, and D. The method of FIG. 8 assumes that the deposit is continuous and symmetrical about the plane ABCD, such that the rear half of the deposit has the same volume as the front half (facing the sensor). Whether the deposit is symmetrical or has some irregularity on one side can be determined by collecting sensor data from different viewpoints, for example according to the method illustrated in FIG. 9. However, assuming that sensor data from only one side is available, the volume of the deposit can be estimated to be twice the volume of the "half".

[0108]

[0123] The boundary of the half volume can be determined as follows: First, a 2D plane 820 is determined that represents the ground surface. The ground surface 820 can be identified by a geometric process, for example as performed by the geometric model 724 of FIG.

[0109]

[0124] Next, a peak 810 (point A) along the top surface of the pile is determined. The peak 810 may correspond to a global maximum (e.g., the highest point) or a point defined by a set of highest points (e.g., the average value of a certain number of highest points). Since the 3D representation 800 describes the contour of the pile in detail, determining the peak 810 is relatively straightforward.

[0110]

[0125] The peak 810 is then projected onto the ground 820. For example, as shown in Figure 8, a line 830 perpendicular to the ground 820 can be determined to identify a point B corresponding to the intersection of the line 830 and the ground 820.

[0111]

[0126] Point B, where line 830 intersects ground surface 820, is then projected onto a sensor plane (e.g., the plane of a camera or LIDAR sensor) to determine where point B intersects with the sensor plane, thereby forming line 840 along ground surface 820.

[0112]

[0127] A line 850 is then determined that is parallel to line 840 by projecting back from the sensor plane onto the 3D representation 800 to identify point C inside the 3D representation 800. Line 850 is parallel to line 840 and is within the ground surface 820. The distance between points B and C can vary. In general, there are no restrictions on the placement of point C, as long as it is within the enclosed volume of the 3D representation of the pile.

[0113]

[0128] After point C is identified, another peak 860 (corresponding to point D) is identified by projecting point C upward to form a line 870. A plane is then determined that extends through all four points A, B, C, and D. This plane corresponds to the rear side of the observable half of the pile. The observable half is represented by a 3D volume bounded by (1) the ground surface 820, (2) the back surface ABCD, and (3) the 3D representation 800. These three elements define a convex hull or convex shell that corresponds to the exterior surface of the observable half. The partial volume of the pile can then be estimated, for example, by determining the total number of voxels between the ground surface 820 and the top of the pile at various points based on the contour of the convex hull and knowledge of the distances represented by the voxels, and then calculating a volume value based on the total number of voxels and multiplying the volume value by two. The total number of voxels between the ground surface 820 and the top of the pile at any given point represents the height of the pile, which can be expressed as a fraction or decimal because the top of the pile does not necessarily align with a voxel edge at all points along the top of the pile.

[0114]

[0129] 8 shows the back surface as being formed by four points ABCD, in practice the back surface may be determined by identifying multiple points C and D on either side of line AB. For example, point C' and corresponding peak D' located to the left of line 830 may be identified to extend the plane toward the left corner of the pile. This can be repeated to identify additional points until a plane is identified that cuts the entire length of the pile.

[0115]

[0130] 10 illustrates an exemplary method for detecting material deposits and estimating their properties using a Bayesian filter that generates feedback to adjust a machine learning model, according to certain embodiments. The method of FIG. 10 is illustrated with reference to a particular arrangement of components that can implement the method. As shown in FIG. 10, these components may include multiple sensors (e.g., a camera 1002, a camera 1004, a LIDAR sensor 1006, and a radar sensor 1008), a pre-processing module 1010, an object detection module 1020, and an estimation module 1030.

[0116]

[0131] Camera 1002 and camera 1004 are similar to camera 702 and camera 704, respectively, of FIG. 7. While the number of sensors can vary, there are at least two camera sensors (e.g., camera 1002 and camera 1004) in the embodiment of FIG. 10. The at least two camera sensors are positioned at different points, and possibly at different angles, to simultaneously capture the environment from different viewpoints. This allows images captured by the at least two cameras to be processed to determine depth information based on the difference in viewpoints of the camera sensors. In some embodiments, the two camera sensors are integrated into a single camera body to form a stereoscopic camera device.

[0117]

[0132] LIDAR sensor 1006 and radar sensor 1008 are similar to LIDAR sensor 706 and radar sensor 708, respectively, of FIG.

[0118]

[0133] Pre-processing module 1010 is similar to pre-processing module 710 of Figure 7 and includes pre-processing unit 1012, pre-processing unit 1014, pre-processing unit 1016, and pre-processing unit 1018, which are similar to pre-processing unit 712, pre-processing unit 714, pre-processing unit 716, and pre-processing unit 718, respectively. For the sake of brevity, the functions and operation of the sensors and pre-processing units shown in Figure 10 will not be described. Instead, it will be understood that these sensors and pre-processing units may incorporate the same features as previously described with respect to the corresponding sensors and processing units of Figure 7.

[0119]

[0134] The object detection module 1020 includes a CNN depth model 1022, a CNN segmentation model 1024, and a voxel estimation model 1026. The CNN depth model 1022 is configured to receive a first 2D image captured by the camera 1002 and a second 2D image captured by the camera 1004 from the pre-processing module 1010. The CNN depth model 1022 generates a 3D image based on the first 2D image and the second 2D image. The 3D image (e.g., an RGB+depth image) represents a composite image determined by matching pixel values ​​in the first 2D image with corresponding pixel values ​​in the second 2D image. The 3D image may include or be output together with depth information estimated by the CNN depth model 1022. The depth information may include a depth value for each pixel in the output image. To generate depth information and a 3D output image, the CNN depth model 1022 may be trained to infer depth values ​​using training data that includes training images from one or more pairs of cameras and ground truth depth information for the training images.

[0120]

[0135] The CNN segmentation model 1024 is similar to the CNN segmentation model 722 of FIG. 7. However, unlike the CNN segmentation model 722, the CNN segmentation model 1024 operates on 3D data (the 3D image output by the CNN depth model 1022). The CNN segmentation model 1024 is configured (e.g., trained) to segment the 3D image generated by the CNN depth model 1022 and classify each segment. Thus, the CNN segmentation model 1024 outputs a segmented 3D image that includes a separate segment for each object detected by the CNN segmentation model 1024. The training of the CNN segmentation model 1024 can be performed in a similar manner to the CNN segmentation model 722 of FIG. 7. However, like the CNN depth model 1022, the CNN segmentation model 1024 can be trained using 3D data. In some embodiments, the CNN segmentation model 1024 and the CNN depth model 1022 can be combined into a single machine learning model.

[0121]

[0136] The voxel estimation model 1026 is a non-CNN-based analog model that maps the LIDAR point cloud received from the pre-processing unit 1016 to a voxel grid based on estimates of the voxel heights of the points in the voxel grid. The voxel estimation model 1026 estimates the voxel heights based on the geometry of the points in the LIDAR point cloud and is not configured to recognize objects. Instead, for each grid point, the voxel estimation model 126 simply estimates the height, in number of voxels, of the feature located at the grid point. The feature may belong to a pile, another type of object, the ground, or some other body of material. The voxel estimation model 1026 outputs the voxel information to a Bayesian filter 1032 in the estimation module 1030. The voxel information includes a voxel representation of the physical environment captured by the LIDAR sensor 1006 and indicates the estimated voxel height as a function of spatial position according to the grid. The voxel estimation model 1026 also receives feedback from the Bayesian filter 1032 and updates itself to correct the height estimate according to the received feedback. Initially, the voxel estimation model 1026 may estimate the height based on certain assumptions (e.g., assuming that each voxel that contains a feature is fully occupied by that feature), and then the estimate is corrected based on the feedback.

[0122]

[0137] The estimation module 1030 comprises a Bayesian filter 1032 and a volume computation module 1034. The Bayesian filter 1032 receives the segmented 3D image from the CNN segmentation model 1024, the voxel information from the voxel estimation model 1026, and the radar data from the pre-processing unit 1018. The Bayesian filter 1032 selectively combines these disparate pieces of information to generate a 3D representation of the deposit or other object of interest (e.g., a DEM or voxel representation of the deposit) for input to the volume computation module 1034. The Bayesian filter 1032 is a histogram-based Bayesian filter that includes an array of Bayesian filters. Each filter in the array of Bayesian filters is configured to predict the correct voxel height of a corresponding grid point using the output of the CNN segmentation model 1024, the voxel estimation model 1026, and / or the radar pre-processing unit 1018.

[0123]

[0138] For example, in certain embodiments, the Bayesian filter 1032 determines a prior probability distribution of the deposit location based on the height estimated by the voxel estimation model 1026, and then determines a posterior probability distribution based on the output of the CNN segmentation model 1024 and / or the output of the radar pre-processing unit 1018. These two probability distributions essentially represent an answer to the question: "What is the probability that there is a deposit of material at this location given the observations derived from the sensor data?" The prior probability distribution represents an answer to the question based solely on information from the model to be corrected (e.g., the voxel estimation model 1026), whereas the posterior probability distribution represents an answer that takes into account additional observations provided by the CNN segmentation model 1024 and / or the segmented 3D image from the radar data. Thus, the posterior probability distribution indicates the likelihood that the height estimate from the voxel estimation model 1026 is correct or not.

[0124]

[0139] If the probability of a height value from the voxel estimation model 1026 is low compared to the probability of the height indicated by the segmented 3D image or the radar data for the same grid point, the Bayesian filter 1032 can communicate feedback to the voxel estimation model 1026 indicating the degree of error in the height estimate. Based on this feedback, the voxel estimation model 1026 can modify its estimate and send the modified estimate to the Bayesian filter 1032. This creates a feedback loop between the Bayesian filter 1032 and the voxel estimation model 1026, which iteratively refines the output of the voxel estimation model 1026 until a certain level of accuracy is reached (e.g., the probability of all estimated height values ​​exceeds a threshold probability value). Once the refinement of the estimate stops, the Bayesian filter 1032 can select the most accurate (e.g., most likely to be correct) input source of the three (segmented 3D image, voxel grid from the voxel estimation model 1026, or radar data) to generate the 3D representation of the deposit.

[0125]

[0140] In certain embodiments, the Bayesian filter 1032 is configured to communicate feedback to the CNN segmentation model 1024, for example, to correct the CNN segmentation model 1024 based on voxel information from the voxel estimation model 1026. The feedback path from the Bayesian filter 1032 to the CNN segmentation model 1024 is omitted from FIG. 10. The feedback communicated to the CNN segmentation model 1024 may indicate a degree of error in the classification of pixels in the segmented image, where the degree of error is determined based on a posterior probability distribution calculated taking into account the voxel information from the voxel estimation model 1026. The feedback to the CNN segmentation model 1024 may be used to adjust one or more weights used by the CNN segmentation model 1024, similar to the adjustments made during the initial training of the CNN segmentation model 1024.

[0126]

[0141] The volume calculation module 1034 is similar to the volume calculation module 736 of FIG. 7. As shown in FIG. 10, the volume calculation module 1034 can be directly coupled to the Bayesian filter 1032 to receive the 3D representation of the deposit / object from the Bayesian filter 1032. The volume calculation module 1034 is configured to estimate the volume of the object of interest (e.g., the deposit 900 of FIG. 9) based on the 3D representation output by the Bayesian filter 1032. The estimated volume can be determined in a manner similar to the volume estimation performed by the volume calculation module 736 (e.g., according to the method shown in FIG. 8) and is output as part of the deposit information 1050. In the embodiment of FIG. 10, the identification of the ground surface 820 of FIG. 8 can be performed by the Bayesian filter 1032 based on the output of the CNN segmentation model 1024 and the voxel estimation model 1026. The Bayesian filter 1032 determines, for each voxel in the voxel grid generated by the voxel estimation model 1026, a probability whether the voxel belongs, for example, to ground rather than to sediment, based on information provided about the voxel by the voxel estimation model 1026 and further based on information provided about the corresponding point in the segmented 3D image from the CNN segmentation model 1024. In this way, the ground can be determined by fitting a 2D plane to voxels classified as being part of the ground based on exceeding a threshold probability.

[0127]

[0142] FIG. 11 illustrates an exemplary method for detecting and estimating properties of material deposits using a single machine learning model, according to certain embodiments. The method of FIG. 11 is illustrated with reference to a particular arrangement of components that can implement the method. As illustrated in FIG. 11, these components may include multiple sensors (e.g., a camera 1102, a LIDAR sensor 1104, and a radar sensor 1106), a pre-processing module 1110, and a machine learning model 1120. Similar to the embodiments illustrated in FIG. 7 and FIG. 10, the pre-processing module 1110 may include a pre-processing unit for each sensor (e.g., a pre-processing unit 1112 that receives the output of the camera 1102, a pre-processing unit 1114 that receives the output of the LIDAR sensor 1104, and a pre-processing unit 1116 that receives the output of the radar sensor 1106).

[0128]

[0143] In the embodiment of FIG. 11, a single machine learning model (ML model 1120) is configured to combine data from different sensors, generate a 3D representation of the object of interest based on the combined data, and estimate the volume of the object of interest, possibly along with other properties of the object. The ML model 1120 outputs inferred information about the object (e.g., deposit information 1150 including an estimated volume of a deposit of material). The ML model 1120 can be implemented using a neural network, such as a CNN or an LSTM. Implementing the ML model 1120 using a CNN reduces computational complexity due to the reuse of convolution kernels, while also providing the advantage of preserving spatial information as the input of the ML model 1120 is processed through the various layers of the CNN. Implementing the ML model 1120 as an LSTM reduces the total number of layers while preserving temporal information (e.g., the time when the different images to be combined were captured).

[0129]

[0144] The ML model 1120 may be divided into sections (e.g., sub-networks or subsystems) that handle different parts of the tasks described above. For example, the ML model 1120 may include, in the following order: a first set of neural network layers that perform fusion (combining representations from different sensor types into a single representation, e.g., a 3D representation of a deposit); a second set of neural network layers that perform object detection (e.g., detecting deposits of material in the single representation); and a third set of neural network layers that perform volume estimation (e.g., estimating the volume of the detected deposits of material). In certain embodiments, the ML model 1120 is trained in two stages. In the first stage, each section of the ML model 1120 is trained separately, with any particular section being trained with training data appropriate for the type of task to be performed by that section. In a second stage, the ML model is trained as a whole using end-to-end training involving training data (e.g., camera, LIDAR, and radar data) representing the input to be processed by the ML model 1120, ground truth information for at least the last section (e.g., the section performing volume estimation), and optionally ground truth information for each of the previous sections. The end-to-end training may adjust the behavior of any particular section when the output of the ML model 1120 indicates that the section is generating an incorrect guess. For example, weights used by a neural network performing segmentation may be adjusted based on determining that the segmentation represented in the output of the ML model 1120 is incorrect compared to the ground truth segmentation information. Similarly, weights used by a neural network performing volume estimation may be adjusted based on the difference between the inferred volume values ​​and the ground truth volume values.

[0130]

[0145] FIG. 12 is a flow chart illustrating a process 1200 for training a machine learning model to detect objects and / or make inferences related to other objects, according to certain embodiments. The process illustrated in FIG. 12 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The method illustrated in FIG. 12 and described below is intended to be exemplary and non-limiting. FIG. 12 depicts various processing steps performed in a particular sequence or order, which is not intended to be limiting. In certain alternative embodiments, steps may be performed in a different order, certain steps may be omitted, or some steps may be performed in parallel. In certain embodiments, such as the embodiment illustrated in FIG. 4, the process illustrated in FIG. 12 may be performed by a machine learning model in a perception subsystem (e.g., an ML model implementing the object detection module 420 and / or the estimation module 430) in conjunction with a computing device or system configured to provide training inputs to the machine learning model.

[0131]

[0146] At 1202, training data is obtained that includes a representation of a physical environment that includes one or more classes of objects. More specifically, the obtained training data includes a 2D and / or 3D representation of the physical environment. For example, the training data may include a set of labeled training images, such as training images 500 of FIG. 5. When used to train an ML model to perform semantic segmentation, the training data includes a representation of a physical environment that includes multiple classes of objects. As described above in connection with the embodiment of FIG. 5, the training representation may include multiple objects of different classes. Alternatively, the ML model may be trained on a single class (e.g., sediment), in which case the training data may include a representation that includes only a single class of objects.

[0132]

[0147] In addition, ground truth information for the representation is obtained at 1202. The ground truth information can be provided as part of the representation itself, for example, by labeling different segments as shown in the embodiment of FIG. 5. Alternatively, the ground truth information can be provided separately from the representation. For example, when training an ML model to infer the volume of an object, ground truth volume values ​​can be provided to the object separately from the 2D or 3D representation of the object.

[0133]

[0148] At 1204, the training data obtained at 1202 is augmented by generating a permutation of the representation. For example, the training images may be subjected to one or more types of image processing to manipulate the training images at an image level (e.g., by applying a blur effect to the entire image) or at a sub-image level (e.g., by modifying objects within the image). Example manipulations include geometric transformations (e.g., scaling, blurring, rotation, hue, gamma or other color adjustments, resizing the image or objects within the image, changing image resolution, adding noise artifacts, and adding visual effects (e.g., to simulate dust on a camera lens, weather conditions such as rain, or other occlusions). Augmentation generates additional training data to supplement the training data originally obtained.

[0134]

[0149] If desired, the ground truth information obtained in 1202 can be used to generate additional ground truth information for the sorting generated in 1204. For example, if an object in an image is zoomed in, the object class labeling can be updated to reflect the change in classification as a result of the resulting image now having more pixels corresponding to the object.

[0135]

[0150] At 1206, the augmented training data (including the training data obtained at 1202 and the additional training data generated at 1204) is input to a machine learning model configured to detect objects represented in the augmented training data and / or make other object-related inferences for the objects. In certain embodiments, such as the embodiment of FIG. 11, the object-related inferences include inferring values ​​of the object's volume and / or other properties. As noted above, in certain embodiments, the training data is presented to the machine learning model in a particular order configured to achieve a balanced representation of object classes when training with multiple classes. Object detection may include classification of regions (e.g., pixels in an image or points in a point cloud) in the representation and determining that adjacent or nearby regions of the same class belong to the same object. As part of detection, the machine learning model may also segment the output representation, for example to generate an output representation that indicates boundaries between sediment objects and non-sediment objects.

[0136]

[0151] At 1208, the machine learning model is adjusted to minimize the error of the guess generated at 1206. The degree of error is determined based on the ground truth information of the training data obtained at 1202 and the ground truth information of the additional training data at 1204. For example, as shown in the embodiment of FIG. 5, the training representation can be labeled to indicate which regions correspond to specific object classes. As another example, when training the ML model to infer the volume of objects, a ground truth volume value can be provided for each object detected in the training data. As described above, adjusting the machine learning model can include modifying weights by backpropagation to minimize a loss function.

[0137]

[0152] At 1210, the accuracy of the machine learning model is verified using the validation data. As noted above, the validation data may be a subset of the training data set aside specifically for testing the accuracy of the machine learning model after the machine learning model has undergone some training. If the results produced by the machine learning model meet one or more criteria (e.g., a threshold accuracy value), the machine learning model is deemed sufficiently trained and the process proceeds to 1212. If not, the processes of 1202-1208 may be repeated for additional training data not previously used to train the machine learning model.

[0138]

[0153] At 1212, the machine learning model is deployed on the autonomous machine, e.g., by storing the machine learning model in a memory of the autonomous machine. In certain embodiments, a previous version of the machine learning model may already be deployed on the autonomous machine. If so, the deployment at 1212 may include updating the existing model, e.g., by communicating weights and other parameters of the machine learning model to the autonomous machine. In certain embodiments, the machine learning model is optimized before deployment. Optimization may include reducing the size of the machine learning model, e.g., by reducing the width (e.g., number of nodes) of one or more layers of a neural network that implements the machine learning model. Optimization may be determined by statistical analysis of the state of the machine learning model, e.g., by computing a histogram of weights in a layer of the neural network.

[0139]

[0154] FIG. 13 is a flow chart illustrating a process 1300 for performing volume estimation, according to certain embodiments. The process illustrated in FIG. 13 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, hardware, or combination thereof. The software may be stored in a non-transitory storage medium (e.g., memory device). The method illustrated in FIG. 13 and described below is intended to be exemplary and non-limiting. FIG. 13 illustrates various processing steps performed in a particular sequence or order, which is not intended to be limiting. In certain alternative embodiments, steps may be performed in a different order, certain steps may be omitted, or some steps may be performed in parallel. In certain embodiments, such as the embodiment illustrated in FIG. 4, the process illustrated in FIG. 13 may be performed by a perception subsystem (e.g., perception subsystem 400).

[0140]

[0155] At 1302, sensor data is acquired from a plurality of sensors. The acquired sensor data may include 2D and / or 3D representations of the physical environment, such as 2D images captured by one or more cameras and point clouds generated by LIDAR and / or radar sensors.

[0141]

[0156] At 1304, the sensor data acquired at 1302 is preprocessed (e.g., by preprocessing module 410) to prepare the sensor data for input to an object detection module (e.g., object detection module 420) at 1306. As noted above, preprocessing may include various reformatting and / or conditioning operations that vary depending on the source of the sensor data (e.g., camera-specific preprocessing, LIDAR-specific, or radar-specific preprocessing).

[0142]

[0157] At 1306, a 3D representation of the object in the physical environment (e.g., a DEM or voxel representation of the sediment or other object) is generated using one or more ML models of the object detection module. In certain embodiments, such as those of FIGS. 7 and 10, the 3D representation of the object is generated by combining information from multiple representations of the physical environment and / or object, including representations derived from different sources of sensor data. For example, in FIG. 10, the Bayesian filter 1032 generates the 3D representation of the sediment based on the segmented 3D representation of the environment received from the CNN segmentation model 1024 and further based on the voxel representation of the sediment received from the voxel estimation model 1026. Thus, as part of generating the 3D representation of the object, at least a portion of the preprocessed sensor data from 1304 may be input to an ML model configured to segment the input representation according to different classes, or more generally, to identify regions of the input representation as corresponding to the object.

[0143]

[0158] At 1308, the volume of the object is estimated using the 3D representation generated at 1306. The volume estimation may be performed, for example, by either of the estimation modules shown in Figures 7 and 10 (estimation module 730 and estimation module 1030) or by the machine learning model that generated the 3D representation at 1306 (e.g., ML model 1120 of Figure 11). As part of, or in addition to, estimating the volume of the object, other properties of the object may also be estimated at 1308. For example, if the volume is estimated according to the method shown in Figure 8, a maximum height of the object is determined. As another example, if the 3D representation is generated based at least in part on LIDAR data, the 3D representation may include reflectance values ​​based on which a metric indicative of the homogeneity of the material of the object is determined.

[0144]

[0159] At 1310, the volume estimate is communicated to a planning subsystem (e.g., planning subsystem 206) to perform an action based on the volume estimate. For example, the volume estimate may be communicated to a planning subsystem of the same autonomous machine performing process 1300 or to a planning subsystem of a different autonomous machine to enable a motion plan to be generated based on the estimated volume. For example, the motion plan may include adding or removing a certain amount of material from the pile to adjust the volume of the pile toward a target volume. As noted above, in certain embodiments, the 3D representation may also be output to the planning subsystem along with the volume estimate.

[0145]

[0160] FIG. 14 illustrates a simplified block diagram of an exemplary computing system 1400 that can be used to implement one or more of the systems and subsystems described in this disclosure and / or to perform any of the processes or methods described herein. For example, in an embodiment in which the autonomous vehicle management system 122 is implemented in software, the software may be executed by a computing system such as the computing system 1400 illustrated in FIG. 14. The computing system 1400 may include, for example, a processor, memory, storage, and I / O devices (e.g., a monitor, keyboard, disk drives, Internet connection, etc.). In some cases, the computing system 1400 may also include other components, circuits, or other specialized hardware for performing specialized functions. In some operational settings, the computing system 1400 may be configured as a system including one or more units, each configured to perform some aspects of a process, either in software only, in hardware only, or some combination thereof. The computing system 1400 may be configured to include additional systems to perform various functions.

[0146]

[0161] As shown in the embodiment of FIG. 14, computing system 1400 includes one or more processing units 1408, a memory set (including system memory 1410, computer readable media 1420, and disk storage 1416), and an I / O subsystem 1406. These components may be communicatively coupled to one another via a bus subsystem that provides a mechanism for the various systems and subsystems of computing system 1400 to communicate with one another as intended. The bus subsystem may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, and the like. In some embodiments, component 1406, component 1408, and component 1410 may be located on a motherboard 1404.

[0147]

[0162] The processing unit 1408 may include one or more processors. The processor may be a single processor single or multi-core processor. The processing unit 1408 may also be implemented using customized circuitry, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor is configured to execute instructions (e.g., programs, codes, etc.) stored in various memories, such as the system memory 1410, the computer readable storage medium 1420, or on the disk 1416. The programs or processes may be executed sequentially or in parallel. In certain embodiments, the computing system 1400 may provide a virtualized computing environment that runs one or more virtual machines. In such embodiments, one or more processors or processor cores may be allocated to each virtual machine. In some embodiments, the processing unit 1408 may include dedicated co-processors, such as graphics processors (GPUs), digital signal processors (DSPs), and the like.

[0148]

[0163] The memory set may include one or more non-transitory memory devices, including volatile and non-volatile memory devices. Software (programs, code modules, instructions) that, when executed by one or more processors of the processing unit(s) 1408, provide the functionality described herein may be stored in one or more of the memories. In certain embodiments, flash memory 1412 may also be included. The system memory 1410 may include several memories, including volatile main random access memory (RAM) (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) for storing instructions and data during program execution, and non-volatile read-only memory (ROM) or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computing system 1400, such as during start-up, may typically be stored in the ROM. The RAM typically contains data and / or program modules currently being operated on and executed by the processing unit(s) 1408.

[0149]

[0164] Executable code, program instructions, applications, and program data may be loaded into the system memory 1410 and executed by the one or more processors of the processing unit(s) 1408. One or more operating systems may also be loaded into the system memory 1410. Examples of operating systems include, but are not limited to, various versions of Microsoft Windows®, Apple Macintosh®, Linux® operating systems, and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, Palm® OS operating systems, and others.

[0150]

[0165] In particular embodiments, programming modules and instructions, data structures, and other data (collectively 1422) used to provide functionality of some embodiments may be stored on a computer readable medium 1420. A media drive 1418 connected to the computing system 1400 may be provided to read information from and / or write information to the computer readable medium 1420. The computer readable medium 1420 may include non-volatile memory such as magnetic disk drives, optical disk drives such as CD-ROMs, DVDs, Blu-Ray® disks, or other optical media, Zip® drives, various types of memory cards and drives (e.g., USB flash drives, SD cards), DVD disks, digital video tape, solid state drives (SSD), and the like.

[0151]

[0166] The I / O subsystem 1406 may include devices and mechanisms for inputting information into the computing system 1400 and / or outputting information from or through the computing system 1400. In general, use of the term input device is intended to include all possible types of devices and mechanisms for inputting information into the computing system 1400. Input mechanisms may include, for example, keyboards, pointing devices such as mice or trackballs, touch pads or touch screens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, bar code readers, and the like. In general, use of the term output device is intended to include all possible types of devices and mechanisms for outputting information from the computing system 1400 to a user or to another computer. Such output devices may include one or more types of displays, indicator lights, or non-visual displays, such as audio output devices, printers, speakers, headphones, audio output devices, and the like. The I / O subsystem 1406 may also include an interface to input and / or output devices external to the I / O subsystem 1406 , such as a display 1414 .

[0152]

[0167] The computing system 1400 may include a communications subsystem 1424 that provides an interface for the computing system 1400 to communicate (e.g., receive data, transmit data) with other computer systems and networks. The communications subsystem 1424 may support both wired and / or wireless communications protocols. For example, the communications subsystem 1424 may enable the computing system 1400 to be communicatively coupled with remote sensors, networks such as the Internet, and the like. A variety of different communications protocols and formats may be used for communication, such as Wi-Fi, Bluetooth (and / or other standards for exchanging data over short distances, including those using short-wavelength radio transmissions), USB, Ethernet, cellular, ultrasonic local area communications protocols, and the like.

[0153]

[0168] The computing system 1400 can be one of a variety of types, including a mobile device (e.g., a cell phone, a tablet, a PDA, etc.), a personal computer, a workstation, or any other data processing system. Due to the ever-changing nature of computers and networks, the description of the computer system 1400 shown in Figure 14 is intended as a specific example only. Many other configurations are possible having more or fewer components than the system shown in Figure 14.

[0154]

[0169] At least some values ​​based on the results of the above-mentioned processes can be stored for subsequent use. In addition, a computer-readable medium can be used to store (e.g., tangibly embody) one or more computer programs for performing any one of the above-mentioned processes by a computer. The computer programs can be written, for example, in a general-purpose programming language (e.g., Pascal, C, C++, Java, Python) and / or some specialized application-specific language (PHP, JavaScript, XML). It is noted that JavaScript is used as an example in some embodiments. However, in other embodiments, other scripting languages ​​and / or JavaScript variants can be utilized as well.

[0155]

[0170] The described features, structures, or characteristics described in this disclosure may be combined in any suitable manner in one or more embodiments. In the description herein, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, and the like, to provide a thorough understanding of various embodiments. However, those skilled in the art will recognize that these features may be implemented without one or more of these specific details, or with other methods, components, materials, and the like. In other instances, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring novel aspects.

[0156]

[0171] The schematic flow chart diagrams included herein are generally described as logical flow chart diagrams. Thus, the illustrated order and labeled steps represent one embodiment of the presented method. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more steps, or portions thereof, of the illustrated method. In addition, it should be understood that the format and symbols used are provided to describe the logical steps of the method and do not limit the scope of the method. Although various types of arrows and lines may be used in the flow chart diagrams, they should be understood not to limit the scope of the corresponding method. In fact, some arrows or other connectors may be used to indicate only the logical flow of the method. For example, arrows may indicate waiting or monitoring periods of unspecified duration between the enumerated steps of the illustrated method. In addition, the order in which a particular method is performed may or may not strictly adhere to the order of the corresponding steps illustrated.

[0157]

[0172] While specific embodiments have been described, various modifications, variations, alternative constructions, and equivalents are possible. The embodiments are not limited to operating in a particular specific data processing environment, but can freely operate in multiple data processing environments. In addition, while specific embodiments are described using a particular sequence of transactions and steps, it should be apparent to one skilled in the art that this is not intended to be limiting. Although some flow charts describe operations as a sequential process, many of the operations can occur in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may have additional steps not included in the figures. Various features and aspects of the above-described embodiments may be used individually or together.

[0158]

[0173] Additionally, while certain embodiments are described using particular combinations of hardware and software, it should be recognized that other combinations of hardware and software are possible. Certain embodiments may be implemented exclusively in hardware, exclusively in software, or using a combination thereof. The various processes described herein may be performed on the same processor or any combination of different processors.

[0159]

[0174] Where a device, system, component or module is described as being configured to perform a particular operation or function, such configuration may be achieved, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, such as by executing computer instructions or code, or by a processor or core programmed to execute code or instructions stored on a non-transitory memory medium, or any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques from time to time.

[0160]

[0175] Specific details are shown in this disclosure to provide a thorough understanding of the embodiments. However, the embodiments may be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques are shown without unnecessary detail to avoid obscuring the embodiments. This description is only intended to provide exemplary embodiments and is not intended to limit the scope, applicability, or configuration of other embodiments. Rather, the foregoing description of the embodiments provides those skilled in the art with enablement requirements for various embodiments. Various changes may be made in the function and arrangement of elements.

[0161]

[0176] The specification and drawings are therefore to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope as set forth in the following claims. Thus, while specific embodiments have been described, these are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

Claims

[Claim 1] a controller system of an autonomous machine receiving sensor data from a plurality of sensors, the sensor data including a first representation of a physical environment and a second representation of the physical environment, each of the first representation and the second representation being a two-dimensional (2D) or three-dimensional (3D) representation including data associated with a deposit of material within the physical environment; and generating a 3D representation of the deposit of material, the controller system comprising: identifying an area within the first representation as corresponding to the deposit of material; identifying an area within the second representation as corresponding to the deposit of material; determining a boundary of the deposit of material based on the identified regions in the first representation and in the second representation; and the controller system estimating a volume of the deposit of material based on the 3D representation of the deposit of material; the controller system communicating information regarding the pile of material to a planning subsystem of the autonomous machine or a planning subsystem of a second autonomous machine, the information regarding the pile of material including the estimated volume, and communicating the information regarding the pile of material results in an action being autonomously performed by the autonomous machine or the second autonomous machine on the pile of material; A method comprising:

Images