Remote processing of sensor data for machine operation

Abstract

The controller may receive sensor data of an environment, including the ground on which the work machine is located, from one or more sensor devices of the work machine. The controller may determine whether a wireless connection is possible. The controller may selectively transmit the sensor data to one or more remote computing devices for processing or process the sensor data locally by the work machine based on whether a wireless connection is possible. If a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices and has the one or more remote computing devices process the sensor data to provide first processed data. If a wireless connection is not possible, the sensor data is processed locally by the work machine to generate second processed data.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates generally to processing machine sensor data, for example, to remote processing of machine sensor data. [Background technology]

[0002] A work machine (e.g., an excavator) may be equipped with stereoscopic ("stereo") cameras that capture images of the environment surrounding the machine. The images may be processed to generate information that can be used in various operations performed by the work machine. Processing the images (e.g., to generate such information) can be computationally complex and time consuming.

[0003] Processing the images using the typical computational resources of a work machine consumes excessive amounts of computational resources that may be required for various operations of the work machine, and thus processing the images using the typical computational resources may adversely affect the ability of the work machine to perform such operations. Moreover, work machines may be used in rocky and harsh conditions that may cause typical computational resources to prematurely fail.

[0004] U.S. Patent No. 6,856,879 (the '879 patent) discloses a system that enables reliable management of the time each operator operates a construction machine. The '879 patent also discloses that a manager can accurately learn in real time, without delay, that a mechanic has performed maintenance on the construction machine. The '879 patent further discloses that work instructions are issued to the mechanic, facilitating and accurately managing the mechanic's performance and labor. The '879 patent further discloses that the ability to accurately learn the installation status of the construction machine in real time enables quick response to customer requests. However, as discussed above, the '879 patent does not address the problems associated with having a construction machine perform computationally complex and time-consuming tasks that require high-performance computing resources.

[0005] The present disclosure solves one or more of the above problems and / or other problems in the art. Summary of the Invention

[0006] A method performed by a controller of a work machine includes receiving sensor data of an environment including a ground surface on which the work machine is located from one or more sensor devices of the work machine; determining whether a wireless connection is possible; selectively transmitting the sensor data to one or more remote computing devices for processing or processing the sensor data locally by the work machine based on determining whether a wireless connection is possible, where if a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices and caused the one or more remote computing devices to process the sensor data to provide first processed data, the first processed data including at least one of first surface data regarding one or more portions of the ground surface or first volumetric data regarding a volume of material moved or to be moved by the work machine; and where if a wireless connection is not possible, the sensor data is processed locally by the work machine to generate second processed data, the second processed data including at least one of second surface data regarding one or more portions of the ground surface or second volumetric data regarding a volume of material moved or to be moved by the work machine; and providing the first processed data or the second processed data to facilitate operation of the work machine.

[0007] 1. A system comprising: one or more sensor devices configured to acquire sensor data of an environment including a ground surface on which a work machine is located; and a controller, wherein the controller is configured to: determine whether a wireless connection is possible; transmit, using a wireless communication component, the sensor data to one or more remote computing devices for processing, wherein if a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices causing the one or more remote computing devices to filter the sensor data and generate processed data; receive the processed data from the one or more remote computing devices, the processed data including at least one of surface data regarding one or more portions of the ground surface or volumetric data regarding a volume of material moved or to be moved by the work machine; and provide the processed data to facilitate operation of the work machine.

[0008] 1. A machine comprising: an implement; one or more sensor devices configured to acquire sensor data of an environment including a ground surface on which the machine is located; a wireless communication component; and a controller, wherein the controller is configured to: determine whether a wireless connection is possible; use the wireless communication component to send a request to one or more remote computing devices based on determining that a wireless connection is possible, the request including the sensor data being sent to the one or more remote computing devices causing the one or more remote computing devices to filter the sensor data and generate processed data after filtering the sensor data; receive processed data from the one or more remote computing devices, the processed data including at least one of surface data regarding one or more portions of the ground surface or volumetric data regarding a volume of material moved or to be moved by the implement; and provide the processed data to facilitate at least one of movement of the machine within the environment or operation of an implement of the machine. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of an exemplary implementation described herein. [Figure 2] FIG. 1 is a diagram of an exemplary system described herein. [Figure 3] 1 is a flowchart of an exemplary process for remote processing of sensor data. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure relates to a machine that selectively transmits sensor data (e.g., from a stereo camera) to one or more computing devices for processing or processes the sensor data locally based on whether a wireless connection is available. A wireless connection may be available when the machine establishes a wireless connection with one or more computing devices, or when the machine has established a wireless connection and the rate of data transmission (of the wireless connection) meets a transmission rate threshold. The sensor data may be transmitted to one or more computing devices for processing, for example, when a wireless connection is available. The sensor data may be transmitted via a satellite device configured for high-speed communication or via a cellular network (e.g., a 5G network) configured for high-speed communication. The one or more computing devices may include one or more cloud computing devices (e.g., one or more high-performance remote computing devices).

[0011] One or more computing devices may process the sensor data to generate processed data. The processed data may be transmitted to the machine (e.g., via a satellite device or cellular network) to facilitate machine operation (e.g., to facilitate material movement operations, navigation operations, among other examples). The machine may suspend transmission of sensor data and buffer the sensor data when wireless communication is lost or when a wireless connection is established and the rate of data transmission (of the wireless connection) does not meet a transmission rate threshold. The machine may resume transmitting sensor data when wireless communication is re-established or when the rate of data transmission (of the wireless connection) meets a transmission rate threshold. In some cases, when wireless connection is not available, the machine may suspend operations (of the machine) that depend on the processed data (e.g., suspend autonomous navigation operations, suspend autonomous material movement operations, among other examples). In some examples, when wireless connection is not available, sensor data may be processed locally on the machine. A wireless connection may not be possible if the machine has not established a wireless connection with one or more computing devices, or if the machine has established a wireless connection and the rate of data transmission does not meet a transmission rate threshold. The machine may provide a notification (e.g., to a machine operator) indicating that it is unable to transmit sensor data to one or more remote computing devices and / or that the sensor data is being processed locally on the machine.

[0012] The term "machine" may refer to a machine that performs an operation involved in industries such as mining, construction, agriculture, transportation, etc. Additionally, one or more implements may be connected to the machine. By way of example, the machine may include a construction vehicle, a work vehicle, or a similar vehicle involved in the above industries.

[0013] 1 is a diagram of an example implementation 100 described herein. As shown in FIG. 1, the example implementation 100 includes a machine 105. The machine 105 is embodied as an earthmoving machine, such as an excavator. Alternatively, the machine 105 may be another type of machine, such as a dozer.

[0014] As shown in FIG. 1 , machine 105 includes ground engaging members 110, machine body 115, operator cabin 120, and swing element 125. Ground engaging members 110 may include tracks (as shown in FIG. 1 ), wheels, and / or rollers, etc., for propulsion machine 105. Ground engaging members 110 are mounted to a rotating frame (not shown) and are driven by one or more engines and drivetrains (not shown). Machine body 115 is mounted to the rotating frame (not shown). Operator cabin 120 is supported by machine body 115. Operator cabin 120 includes an integrated display (not shown) and operator controls 124, such as, for example, an integrated joystick. Operator controls 124 may include one or more input components.

[0015] In the case of an autonomous machine, operator controls 124 may not be designed for use by an operator, but rather may be designed to operate independently of an operator. In this case, operator controls 124 may include one or more input components that, for example, provide input signals used by another component without any operator input. Pivoting elements 125 may include one or more components that allow the rotating frame (and machine body 115) to rotate (or pivot). For example, pivoting elements 125 may allow the rotating frame (and machine body 115) to rotate (or pivot) relative to ground engaging members 110.

[0016] As shown in FIG. 1 , machine 105 includes boom 130, stick 135, and mechanical work tool 140. Boom 130 is pivotally mounted to its proximal end on machine body 115 and articulates relative to machine body 115 by one or more fluid-actuated cylinders (e.g., hydraulic or pneumatic cylinders), electric motors, and / or other electromechanical components. Stick 135 is pivotally mounted to a distal end of boom 130 and articulates relative to boom 130 by one or more fluid-actuated cylinders, electric motors, and / or other electromechanical components. Boom 130 and / or stick 135 may be referred to as a linkage. Mechanical work tool 140 is mounted to the distal end of stick 135 and may articulate relative to stick 135 by one or more fluid-actuated cylinders, electric motors, and / or other electromechanical components. Mechanical work tool 140 may be a bucket (as shown in FIG. 1 ) or another type of tool that may be mounted to stick 135. The machine work tool 140 may be referred to as an implement.

[0017] As shown in FIG. 1 , machine 105 includes a controller 145 (e.g., an electronic control module (ECM), a computer vision controller, an autonomy controller, among other examples), one or more inertial measurement units (IMUs) 150 (referred to herein individually as “IMU 150” and collectively as “IMUs 150”), a stereo camera 155, a wireless communication component 160, and one or more sensor devices 165. Controller 145 may control and / or monitor operation of machine 105. For example, controller 145 may control and / or monitor operation of machine 105 based on signals from operator controls 124, signals from IMU 150, signals from stereo camera 155, signals from wireless communication component 160, and / or signals from one or more sensor devices 165.

[0018] 1 , IMUs 150 are mounted at various locations on components or portions of machine 105, such as machine body 115, boom 130, stick 135, and machine work tool 140. IMUs 150 include one or more devices capable of receiving, generating, storing, processing, and / or providing signals indicative of the position and orientation of the component of machine 105 on which IMU 150 is mounted. For example, IMU 150 may include one or more accelerometers and / or one or more gyroscopes. The one or more accelerometers and / or one or more gyroscopes generate and provide signals that can be used to determine the position and orientation of IMU 150 relative to a frame of reference, and thus the position and orientation of the component. While the examples discussed herein refer to IMUs 150, the present disclosure is applicable to using one or more other types of sensor devices that can be used to determine the position and orientation of the component of machine 105.

[0019] The stereo camera 155 may include one or more devices capable of acquiring and providing sensor data of the environment, including the ground on which the machine 105 is located. The sensor data may include image data (e.g., three-dimensional (3D) image data) of the environment. Although the examples discussed herein refer to the stereo camera 155, the present disclosure is applicable to using one or more other types of devices, such as light detection and ranging (LIDAR) devices and / or radio detection and ranging (RADAR) devices, among other examples of devices (e.g., sensor devices) configured to provide data about the environment. Unless otherwise specified, any reference hereinafter to the stereo camera 155 should be understood to encompass these other devices.

[0020] 1 illustrates a single stereo camera 155 as being included on the machine 105. In practice, the machine 105 may include one or more additional stereo cameras 155 and / or otherwise positioned stereo cameras 155 (e.g., stereo cameras 155 mounted on other portions of the machine 105). The stereo cameras 155 may be mounted on, for example, the boom 130, the stick 135, and the machine work tool 140.

[0021] Wireless communication component 160 includes one or more devices that enable machine 105 to communicate with other devices, as described herein. Wireless communication component 160 may include a transceiver, separate transmitters and receivers, and antennas, among other examples. Wireless communication component 160 may transmit sensor data (e.g., from stereo camera 155) to one or more computing devices and receive processed data from one or more computing devices, as described herein.

[0022] Wireless communication component 160 may be configured to transmit sensor data as a continuous flow of data from stereo camera 155. In some examples, wireless communication component 160 may transmit sensor data and receive processed data via a satellite device, via a base station, and / or via another device that may enable wireless communication component 160 to transmit data to and receive data from one or more computing devices, as described below.

[0023] The wireless communication component 160 may communicate with one or more computing devices over a network including one or more wired and / or wireless networks, such as a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, a Code Division Multiple Access (CDMA) network, a 3G network, a 4G network, or another type of cellular network), the Internet, a cloud computing network, a wireless local area network (LAN), an intranet, a fiber optic-based network, a wide area network (WAN), a public land mobile network (PLMN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, and an ad hoc network, and / or a combination of these or other types of networks, among other examples.

[0024] Sensor devices 165 include one or more devices capable of receiving, generating, storing, processing, and / or providing signals related to one or more components of machine 105. For example, sensor devices 165 may include hydraulic displacement sensors, pressure sensors (e.g., for hydraulic cylinders), machine speed sensors, engine RPM sensors, and / or global positioning system (GPS) devices, among other examples.

[0025] 1 , exemplary implementation 100 includes satellite device 170, which includes one or more computing devices 175-1 through 175-N (hereinafter collectively referred to as computing devices 175 and individually as computing device 175), and also includes base station 180. Satellite device 170 may include one or more devices configured for high-speed communications (e.g., geostationary satellites configured for high-speed communications). Satellite device 170 may be configured to provide, for example, high-speed Internet access (e.g., to machine 105 via wireless communications component 160).

[0026] Computing device (or remote computing device) 175 may include one or more devices configured to process sensor data to generate processed data, as described herein. Computing device 175 may be implemented as a single computing device (e.g., a single server) or multiple computing devices (e.g., multiple servers), such as multiple computing devices in one or more data centers. In some examples, computing device 175 may be included in a cloud computing environment. Additionally or alternatively, computing device 175 may be included in a back-office system.

[0027] The base station 180 may be connected to a cellular network (e.g., as described above). The base station 180 may be connected to, for example, a 5G network. The base station 180 may include a base transceiver station, a radio base station, a Node B, an evolved Node B (eNB), a next generation Node B (gNB), a base station subsystem, a cellular site, a cellular tower (e.g., a cell phone tower, a mobile phone tower, etc.), an access point, a transmit / receive point (TRP), a radio access node, a macrocell base station, a microcell base station, a picocell base station, and / or a femtocell base station, or a similar type of device.

[0028] As noted above, Figure 1 is provided as an example. Other examples may differ from the example described with reference to Figure 1.

[0029] 2 is a diagram of an example system 200 described herein. As shown in FIG. 2, system 200 includes controller 145, IMU 150, stereo camera 155, wireless communication component 160, one or more sensor devices 165, one or more computing devices 175, along with satellite device 170 and base station 180. Controller 145 may be configured to cause sensor data from stereo camera 155 to be processed by one or more computing devices 175 or to be processed locally on machine 105, as described in further detail below.

[0030] The IMUs 150 may include one or more first IMUs 150, one or more second IMUs 150, and one or more third IMUs 150. The one or more first IMUs 150 may be located on the boom 130 and / or stick 135. The one or more first IMUs 150 may generate signals that can be used to determine the position and / or orientation of the boom 130 and / or stick 135, which may be used to generate linkage pose data indicative of the position and / or orientation of the boom 130 and / or stick 135. The one or more second IMUs 150 may be located on the machine work tool 140. The one or more second IMUs 150 may generate signals that can be used to determine the position and / or orientation of the machine work tool 140, which may be used to generate implement pose data indicative of the position and / or orientation of the machine work tool 140. The one or more third IMUs 150 may be located on the machine body 115. The one or more third IMUs 150 may generate signals that can be used to determine the position and / or orientation of the mechanical body 115, and the signals may be used to generate machine pose data indicative of the position and / or orientation of the mechanical body 115.

[0031] IMU 150 may provide linkage pose data, implement pose data, and / or machine pose data to controller 145 periodically (e.g., every 30 seconds, every minute, every 5 minutes, among other examples). Additionally or alternatively, IMU 150 may provide linkage pose data, implement pose data, and / or machine pose data to controller 145 based on a trigger event. Trigger events may include, among other examples, a request from controller 145, detection of movement of machine body 115, detection of movement of machine work tool 140, detection of movement of machine body 115, detection of movement of boom 130 and / or stick 135, a request from an operator, a request from a back office system, initiation of a new task by machine 105, relocation of machine 105 to a different geographic location, after machine 105 completes a digging cycle and / or after machine work tool 140 completes digging, and during swing before unloading material. The linkage pose data, instrument pose data, and / or machine pose data may be transmitted to the one or more remote computing devices at the same rate that the sensor data was transmitted to the one or more remote computing devices. The linkage pose data, instrument pose data, and / or machine pose data may be transmitted in this manner to synchronize the sensor data with the linkage pose data, instrument pose data, and / or machine pose data. Alternatively, the linkage pose data, instrument pose data, and / or machine pose data may be time-stamped so that images (from the sensor data) can be registered to world coordinates using data from IMU 150 and correlated with other data from machine 105. If precise synchronization of camera frames and sensor time samples is not possible, time interpolation of the sensor data may be performed to temporally align the sensor data with the linkage pose data, instrument pose data, and / or machine pose data.

[0032] The stereo camera 155 may be configured to acquire sensor data of the environment, including the terrain surrounding the machine 105 and including the ground on which the machine 105 is located. The sensor data may include image data of the environment (e.g., images of the environment). In some cases, the sensor data may include 3D image data of the environment. The stereo camera 155 may provide the sensor data to the controller 145 periodically (every second, every other second, among other examples). Additionally or alternatively, the stereo camera 155 may provide the sensor data to the controller 145 based on a trigger event (e.g., a request from the controller 145).

[0033] Wireless communication component 160 may be configured to establish a wireless connection with one or more computing devices 175 (e.g., establish a wireless connection with satellite device 170 or with base station 180, and thus with one or more computing devices) via satellite device 170 or via base station 180. Wireless communication component 160 may be configured to attempt to establish the wireless connection periodically (e.g., every 30 seconds, every minute, among other examples) and / or based on a trigger event (e.g., based on a request from controller 145, based on receiving an explicit indication that machine 105 is operating in an autonomous mode, based on initiation of wireless communication component 160, among other examples).

[0034] The wireless communication component 160 may transmit the sensor data (captured by the stereo camera 155) over a wireless connection to one or more computing devices 175 for processing. The one or more computing devices 175 may process the sensor data to generate processed data, and the wireless communication component 160 may receive the processed data from the one or more computing devices 175 over a wireless connection.

[0035] Wireless communication component 160 may be configured to prioritize establishing a wireless connection with satellite device 170 (e.g., over establishing a wireless connection with base station 180). Wireless communication component 160 may be configured to establish a wireless connection with satellite device 170, for example, if the signal strength of base station 180 does not exceed the signal strength of satellite device 170 by more than a threshold difference. As another example, wireless communication component 160 may be configured to establish a wireless connection with satellite device 170 if the signal strength of satellite device 170 meets a signal strength threshold regardless of the signal strength of base station 180.

[0036] Wireless communication component 160 may be pre-configured with information identifying the threshold difference, may receive information identifying the threshold difference from a device of an operator of machine 105, or may receive information identifying the threshold difference from a back office system, among other examples. As an alternative to prioritizing establishing a wireless connection with satellite device 170, wireless communication component 160 may be configured to prioritize establishing a wireless connection with base station 180 (e.g., over establishing a wireless connection with satellite device 170) in a manner similar to that described above with respect to satellite device 170.

[0037] In some implementations, wireless communication component 160 may be configured to establish a wireless connection based on the relative signal strength of satellite device 170 and the signal strength of base station 180. Wireless communication component 160 may be configured to establish a wireless connection with satellite device 170, for example, when the signal strength of satellite device 170 exceeds the signal strength of base station 180. Alternatively, wireless communication component 160 may be configured to establish a wireless connection with base station 180 when the signal strength of base station 180 exceeds the signal strength of satellite device 170.

[0038] Wireless communication component 160 (and / or controller 145) may periodically (e.g., every 15 seconds, every 30 seconds, among other examples) determine the signal strength of satellite device 170 and the signal strength of base station 180. In this regard, wireless communication component 160 may periodically alternate between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180 based on the periodic determination of the signal strength of satellite device 170 and the signal strength of base station 180.

[0039] As an example of periodically alternating the establishment of a wireless connection, assume that wireless communication component 160 has established a wireless connection with satellite device 170. Wireless communication component 160 may terminate the wireless connection with satellite device 170 and establish a wireless connection with base station 180 when wireless communication component 160 (and / or controller 145) determines that the signal strength of base station 180 exceeds the signal strength of satellite device 170 (and vice versa).

[0040] In addition to or as an alternative to periodically determining signal strength, wireless communication component 160 (and / or controller 145) may determine the signal strength of satellite device 170 and the signal strength of base station 180 based on a trigger event (e.g., based on a request from controller 145, based on an indication of a change in operation of machine 105, among other examples). Wireless communication component 160 may alternate between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180 based on the trigger event.

[0041] By alternating (e.g., periodically and / or based on a trigger event) between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180, wireless communication component 160 may maintain communication with one or more computing devices 175 and may attempt to minimize interruptions in the transmission of sensor data and / or the reception of processed data. By preventing interruptions in the transmission of sensor data and / or the reception of processed data, wireless communication component 160 and / or controller 145 may prevent any interruptions in the operations performed by machine 105.

[0042] In some examples, wireless communication component 160 may determine that a wireless connection with satellite device 170 cannot be established if the signal strength of satellite device 170 does not meet a signal strength threshold. Wireless communication component 160 may similarly determine that a wireless connection with base station 180 cannot be established if the signal strength of base station 180 does not meet a signal strength threshold. Controller 145 may determine that sensor data is to be processed locally on machine 105 if the signal strength of satellite device 170 and the signal strength of base station 180 do not both meet the signal strength threshold. Wireless communication component 160 may be pre-configured with information identifying the signal strength threshold, receive information identifying the signal strength threshold from an operator's device, or receive information identifying the signal strength threshold from a back-office system, among other examples.

[0043] Wireless communication component 160 may be configured to establish a wireless connection based on the output of the machine learning model. Wireless communication component 160 (and / or controller 145) may use the machine learning model to predict, for example, whether wireless communication component 160 should establish a wireless connection with satellite device 170, whether wireless communication component 160 should establish a wireless connection with base station 180, or whether wireless communication component 160 should not establish a wireless connection with satellite device 170 or base station 180.

[0044] Controller 145 may provide as input to the machine learning model information identifying the current geographic location and / or current geographic region of machine 105, information identifying one or more obstacles at the current geographic location and / or current geographic region (e.g., one or more images of the one or more obstacles), information identifying the time of day, and / or information identifying expected weather conditions, among other examples. The input may be obtained from an operator's device, stereo camera 155, and / or one or more sensor devices 165, among other examples. Based on the input, the machine learning model may provide as output predicted connection information indicating whether wireless communication component 160 should establish a wireless connection with satellite device 170, whether to establish a wireless connection with base station 180, or whether to not establish a wireless connection with satellite device 170 or base station 180.

[0045] The machine learning model may be trained using historical data including, among other examples, historical data of geographic locations and / or geographic areas, historical data (e.g., images) of obstacles at past geographic locations and / or geographic areas, historical data of past times, weather conditions at past geographic locations and / or geographic areas, historical signal strengths of satellite devices 170 and base stations 180 at past geographic locations and / or geographic areas (e.g., in light of past obstacles, past times, past weather conditions, among other examples), historical latency of satellite devices 170 and base stations 180 at past geographic locations and / or geographic areas, and / or historical data indicative of established wireless connections (e.g., satellite devices 170 or base stations 180).

[0046] The machine learning models may be generated and trained by a trainer device, which may be a separate hardware or software component (not shown). The training device may be included in a back office system or included in one or more computing devices 175, among other examples. The training device may provide the machine learning models to the machine 105 for use by the controller 145 and / or the wireless communication component 160. The training device may update and provide the machine learning models to the machine 105 (e.g., based on a schedule, upon demand, per trigger, or periodically, among other examples). In some cases, the controller 145 may obtain additional training data (e.g., additional past data similar to the past data described above) and cause the machine learning models to be retrained based on the additional training data. The controller 145 may, for example, provide the additional training data to the training device to retrain the machine learning models. The machine learning models may be retrained periodically and / or based on a trigger event.

[0047] When training a machine learning model, the training device may divide the training data into a training set (e.g., a dataset for training the machine learning model), a validation set (e.g., a dataset used to evaluate the fit of the machine learning model and / or fine-tune the machine learning model), and / or a test set (e.g., a dataset used to evaluate the final fit of the machine learning model), etc. The training device may preprocess and / or perform dimensionality reduction to reduce the training data to a minimal feature set. By training the machine learning model on this minimal feature set, the training device may reduce processing for training the machine learning model and may apply classification techniques to the minimal feature set.

[0048] The training device may use classification techniques, such as logistic regression, random forest, and / or gradient boosting machine learning (GBM), to determine a classification outcome (e.g., whether to establish a wireless connection with satellite device 170 or base station 180). The training device may use a naive Bayesian classifier technique in addition to or as an alternative to using a classification technique. In this case, the device may perform binary recursive partitioning to divide the training data of a minimum feature set into partitions and / or branches and use the partitions and / or branches to make a prediction (e.g., whether to establish a wireless connection with satellite device 170 or base station 180). Based on the use of recursive partitioning, the training device may reduce the use of computational resources associated with manual linear classification and analysis of data items, thereby enabling models to be trained using thousands, millions, or billions of data items, resulting in a more accurate model than would be obtained using fewer data items.

[0049] The training device may train the machine learning model using a supervised training procedure that includes receiving input to the machine learning model from subject matter experts (e.g., one or more operators associated with the machine 105 and / or one or more machines), which may reduce the time and / or amount of processing resources, etc., to train the machine learning model compared to an unsupervised training procedure. The training device may use one or more other model training techniques, such as neural network techniques and / or latent semantic indexing techniques.

[0050] For example, the training device may implement artificial neural network processing techniques (e.g., using a two-layer feed-forward neural network architecture and / or a three-layer feed-forward neural network architecture, etc.) to perform pattern recognition on patterns establishing wireless connections with satellite device 170 or base station 180. In this case, the use of artificial neural network processing techniques may improve the accuracy of the machine learning models generated by the training device by being more robust to noisy, inaccurate, or incomplete data and by enabling the training device to detect patterns and / or trends that cannot be detected by human analysts or systems using less complex techniques.

[0051] In the following examples, it is assumed that stereo camera 155 provides sensor data to controller 145. Stereo camera 155 may, for example, provide the sensor data to controller 145 in a manner similar to that described above. It is assumed that stereo camera 155 receives a request for sensor data from controller 145, for example. Based on receiving the request, stereo camera 155 may provide the sensor data to controller 145. Controller 145 may receive the sensor data from stereo camera 155 and determine whether to cause the sensor data to be sent to one or more computing devices 175 for processing or whether to cause the sensor data to be processed locally on machine 105.

[0052] The controller 145 may determine whether a wireless connection is available as part of determining whether to have the sensor data transmitted to one or more computing devices 175 for processing or to have the sensor data processed locally on the machine 105. A wireless connection may be available if the wireless communication component 160 establishes a wireless connection with a satellite device 170 or a base station 180, or if the wireless communication component 160 has established a wireless connection and the rate of data transmission meets a transmission rate threshold. The transmission rate threshold may be determined by an operator, determined by a back office system, or stored in memory of the machine 105, among other examples. The controller 145 may cause the wireless communication component 160 to transmit the sensor data to one or more computing devices 175 via a wireless connection if a wireless connection is available. Alternatively, the controller 145 may cause the sensor data to be processed locally on the machine 105 if a wireless connection is not available. A wireless connection may not be possible if the wireless communication component 160 has not established a wireless connection with the satellite device 170 or base station 180, or if the wireless communication component 160 has established a wireless connection and the rate of data transmission does not meet the transmission rate threshold.

[0053] In some examples, controller 145 may receive wireless connection information from wireless communication component 160 indicating whether a wireless connection is possible, and controller 145 may determine whether a wireless connection is possible based on the wireless connection information. In some examples, controller 145 may determine whether wireless communication component 160 needs to establish a wireless connection (e.g., with satellite device 170 or base station 180) and may provide instructions to wireless communication component 160 to establish the wireless connection based on determining whether wireless communication component 160 establishes a wireless connection with satellite device 170 or base station 180.

[0054] Controller 145 may receive wireless connection information from wireless communication component 160 after providing the instructions and may determine whether a wireless connection is possible based on the wireless connection information. Controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection without determining whether wireless communication component 160 can establish a wireless connection, as an alternative to providing instructions based on determining whether wireless communication component 160 needs to establish a wireless connection. Controller 145 may provide instructions based on receiving sensor data, for example.

[0055] As part of determining whether wireless communication component 160 establishes a wireless connection, controller 145 may determine whether machine 105 is in an idle state or an active state, determine whether machine work tool 140 is in a locked or unlocked position, determine whether the signal strength of satellite device 170 or the signal strength of base station 180 meets a signal strength threshold, determine whether machine 105 is operating in a manual mode or an autonomous mode, and determine the geographic location of machine 105, among other examples. As an example, controller 145 may obtain machine speed data identifying the ground speed of machine 105 and / or engine speed data identifying the engine speed of machine 105 from one or more sensor devices 165. Controller 145 may determine whether machine 105 is in an idle state or an active state based on the ground speed and / or engine speed.

[0056] For example, the controller 145 may determine that the machine 105 is in an idle state if the ground speed does not meet the ground speed threshold and / or the engine speed does not meet the engine speed threshold. Conversely, the controller 145 may determine that the machine 105 is in an active state if the ground speed meets the ground speed threshold and / or the engine speed meets the engine speed threshold. The controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection if the machine 105 is in an active state or if the machine 105 has been in an idle state for a time that does not meet the threshold time. Alternatively, the controller 145 may provide instructions to the wireless communication component 160 not to establish a wireless connection if the machine 105 has been in an idle state for a time that meets the threshold time.

[0057] The controller 145 may determine whether the mechanical work tool 140 is in a locked or unlocked position based on input from a lever (or button) in the operator cabin 120. The operator may lock (or secure) or unlock the mechanical work tool 140 using, for example, a lever. The operator may lock or unlock a hydraulic system associated with the mechanical work tool 140 using, for example, a lever. When the hydraulic system is locked, hydraulic fluid may not flow to the hydraulic system's linkage hydraulic cylinders, swing pump motor, or track pump motor. When the hydraulic system is unlocked, hydraulic fluid may flow to the hydraulic system's linkage hydraulic cylinders, swing pump motor, or track pump motor. The lever may generate a value indicating whether the mechanical work tool 140 is locked or unlocked. The controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection when the mechanical work tool 140 is in the unlocked position. The controller 145 may obtain information identifying the signal strength of the satellite device 170 or information identifying the signal strength of the base station 180 from the wireless communication component 160. Controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection if the signal strength of satellite device 170 or the signal strength of base station 180 meets the signal strength threshold. Conversely, if the signal strength of satellite device 170 and the signal strength of base station 180 do not both meet the signal strength threshold, controller 145 may determine that wireless communication component 160 cannot establish a wireless connection (e.g., with satellite device 170 or base station 180). Controller 145 may determine that the sensor data (from stereo camera 155) will be processed locally based on a determination that wireless communication component 160 cannot establish a wireless connection.

[0058] Controller 145 may obtain (e.g., from a memory of machine 105) operational information indicating whether machine 105 is operating in a manual mode or an autonomous mode. If the operational information indicates that machine 105 is operating in an autonomous mode, controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection.

[0059] Controller 145 may obtain location data (e.g., from one or more sensor devices 165) identifying the geographic location of machine 105. If controller 145 determines that the geographic location is an underground location, a building, or a covered structure that may prevent wireless communication component 160 from communicating with satellite device 170 or base station 180, among other locations, controller 145 may prevent wireless communication component 160 from attempting to establish a wireless connection. Alternatively, if controller 145 determines that the geographic location is not a location that may prevent wireless communication component 160 from communicating with satellite device 170 or base station 180, controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection. Controller 145 may perform similar action with respect to information regarding predicted weather conditions at a geographic location, such that wireless communication component 160 is prevented from establishing a wireless communication. As described above, by preventing wireless communication component 160 from attempting to establish a wireless connection, controller 145 may conserve computational resources that would otherwise be used in a failed attempt to establish a wireless connection (e.g., due to geographic location, predicted weather conditions, among other examples).

[0060] Assume that controller 145 provides instructions to wireless communication component 160 to establish a wireless connection. Based on the instructions, controller 145 may establish a wireless connection (e.g., with satellite device 170 or base station 180) in a manner similar to that described above and provide an indication that the wireless connection has been established. Assume that controller 145 determines (e.g., based on an indication from wireless communication component 160) that a wireless connection is possible. Based on determining that a wireless connection is possible, controller 145 may cause wireless communication component 160 to transmit sensor data (from stereo camera 155) over the wireless connection to one or more computing devices 175 for processing.

[0061] The sensor data may include images acquired by the stereo camera 155 and / or a disparity map generated based on the images. In some cases, the machine 105 may include a LIDAR device. In such cases, the sensor data may include a 3D point cloud of the environment based on data acquired by the LIDAR device. One or more computing devices 175 may receive the sensor data and process the sensor data to generate processed data.

[0062] When processing the sensor data, the one or more computing devices 175 may perform one or more operations (e.g., filtering operations) on the sensor data. The one or more computing devices 175 may, for example, among other examples, process the sensor data to remove or reduce an amount of noise (e.g., outlier data) from the sensor data and / or remove items that may be distorted in the sensor data or distort the rendering of the sensor data. The one or more computing devices 175 may, for example, perform filtering on the sensor data (e.g., perform filtering on a disparity map) to remove or reduce the amount of noise (e.g., outlier data) and / or remove items, among other examples.

[0063] With respect to performing filtering, the one or more computing devices 175 may process the sensor data using spatial noise reduction filtering, temporal / time filtering, and / or digital signal processing techniques, among other examples. As a result of processing the sensor data in this manner, the one or more computing devices 175 may increase the resolution of the sensor data (e.g., increase the measure of smoothness of the sensor data).

[0064] In addition to or as an alternative to removing or reducing noise and / or removing items, the one or more computing devices 175 may process the sensor data to remove image occlusions from the sensor data. The one or more computing devices 175 may, for example, use one or more computer vision techniques to identify and remove one or more items that may have obstructed the line of sight of the stereo camera 155. The one or more computing devices 175 may, for example, analyze the sensor data using one or more object detection techniques (e.g., single-shot detector (SSD) techniques and / or YOLO (You Only Look Once) techniques, etc.) to identify one or more items. In addition to or as an alternative to removing image occlusions, the one or more computing devices 175 may interpolate data in one or more portions of the sensor data. As an example, the one or more computing devices 175 may perform image processing on the sensor data (e.g., using one or more image processing techniques) to interpolate the sensor data.

[0065] By performing one or more of the operations described above, one or more computing devices 175 may increase a measure of quality of the sensor data and increase the probability of identifying useful information that may facilitate operation of machine 105. One or more computing devices 175 may generate filtered data based on performing one or more of the operations described above. One or more computing devices 175 may process the filtered data (e.g., using semantic segmentation techniques) to determine information about one or more holes on the ground, information about one or more piles of material on the ground, information about the material, and / or information about the material within machine work tool 140, among other examples.

[0066] The information about the one or more holes may include, among other examples, the location of the one or more holes, the depth of the one or more holes, and / or the geometry (or shape) of the one or more holes. The information about the one or more peaks may include, among other examples, the location of the one or more peaks, the height of the one or more peaks, the geometry (or shape) of the one or more peaks, and / or the volume of the one or more peaks. The information about the material may include, among other examples, the type of material, such as sand, clay, and / or hard rock. The information about the material in the machine work tool 140 may include, among other examples, the volume of the material, the geometry (or shape) of the material, and / or the fill level of the machine work tool 140.

[0067] Information regarding one or more holes, one or more mountains, and / or material may be included in the surface data. The surface data may be used by the machine 105 to facilitate navigation of the machine 105 within an environment. The surface data may be used by the machine 105, for example, to identify one or more navigable portions of the environment and to identify one or more non-navigable portions of the environment. As an example, the one or more navigable portions may include one or more portions of the ground that do not include one or more holes and / or do not include one or more mountains, among other examples of items that may interfere with navigation operations of the machine 105. The one or more non-navigable portions may include one or more portions of the ground that include one or more holes and / or one or more mountains, among other examples of items that may interfere with navigation operations of the machine 105. In this regard, the surface data may be used for autonomous navigation operations of the machine 105.

[0068] The surface data may be used by the machine 105 to facilitate material movement operations (e.g., excavation operations) of the machine 105. The surface data may be used by the machine 105, for example, to identify one or more locations in the ground (other than the location of one or more holes and / or one or more mounds) for material movement operations. In this regard, the surface data may be used for autonomous material movement operations of the machine 105.

[0069] Information regarding the material and / or information regarding the material in the machine work tool 140 may be included in the volumetric data. The volumetric data may be used (e.g., by the machine 105 and / or a back office system, among other examples) to determine, among other examples, a measure of the productivity of an operator of the machine 105. The volumetric data may include, for example, information identifying a volume of material moved by the machine 105 over a period of time and / or information indicative of a measure of completion of a material movement task, among other examples.

[0070] In some implementations, one or more computing devices 175 may process the filtered data using one or more machine learning models. The one or more computing devices 175 may, for example, provide the filtered data as input to one or more machine learning models, which may provide information about one or more holes, information about one or more peaks, information about the material, and / or information about the material in the machine work tool 140 as output. For example, a first machine learning model may provide information about one or more holes (as output), a second machine learning model may provide information about one or more peaks (as output), and a third machine learning model may provide information about the material (as output).

[0071] The one or more computing devices 175 may aggregate the various outputs into processed data, including source data and volumetric data. The wireless communication component 160 may receive the processed data from the one or more computing devices 175 via a wireless connection and provide the processed data to the controller 145 to facilitate machine operations (e.g., navigation operations and / or material movement operations, among other examples). In some cases, the processed data may be provided to a site manager's device and / or a back-office system, among other examples.

[0072] The one or more machine learning models may be trained using historical data (e.g., historical images, historical disparity maps, and / or historical point clouds) that identify, among other examples, different holes (e.g., different sizes, shapes, and / or depths), different peaks (e.g., different sizes, shapes, and / or heights), different types of material, different volumes of material, different mechanical work tools, and / or different volumes of material within the mechanical work tools. In some cases, PointNet may be an example of one or more machine learning models used with respect to point clouds. The one or more machine learning models may be trained in a manner similar to that described above.

[0073] The controller 145 may receive additional data for processing in conjunction with the sensor data from one or more sensor devices 165. The additional data may include implement pose data, machine pose data, linkage pose data, hydraulic displacement data, and / or pressure data (e.g., for hydraulic cylinders), among other examples. The controller 145 may cause the wireless communication component 160 to transmit the additional data via a wireless connection to one or more computing devices 175 for processing along with the sensor data to generate processed data.

[0074] For example, when determining the volume of material within the machine work tool 140, the one or more computing devices 175 may determine the coordinates of the machine work tool 140 within the disparity map based on the implement pose data. The coordinates of the machine work tool 140 may enable the one or more computing devices 175 (and / or one or more machine learning models) to identify a portion of the sensor data (e.g., a portion of the disparity map) that includes the machine work tool 140 when determining the volume of material within the machine work tool 140. By identifying a portion of the sensor data, the one or more computing devices 175 (and / or one or more machine learning models) may conserve computational resources that would be used to process the entire sensor data for purposes of determining the volume of material within the machine work tool 140. In some examples, when determining the volume of material within the machine work tool 140, the one or more computing devices 175 may determine the geometry of the machine work tool 140. In some cases, the one or more computing devices 175 may receive information regarding the geometry of the machine work tool 140 from the machine 105 (e.g., from the controller 145 via the wireless communication component 160). Alternatively, one or more computing devices 175 may receive information identifying machine 105 (e.g., from controller 145 via wireless communication component 160) and may use the information identifying machine 105 to obtain information regarding the geometry of machine work tool 140 (e.g., from a data structure). One or more computing devices 175 may determine the geometry of the interior surface of machine work tool 140 based on the information regarding the geometry of machine work tool 140. One or more computing devices 175 may determine the portion of the interior surface that is in contact with material within machine work tool 140 based on analyzing sensor data. One or more computing devices 175 may, for example, determine how high up on machine work tool 140 is material on all sides of machine work tool 140. Additionally, one or more computing devices 175 may determine the contour shape of material above the top surface of machine work tool 140.One or more computing devices 175 may combine all surfaces and calculate one single enclosed volume.

[0075] In some implementations, controller 145 may cause wireless communication component 160 to transmit data (e.g., sensor data, additional data, among other examples) via a wireless connection based on determining whether mechanical work tool 140 is embedded in the ground or has been removed from the ground. Controller 145 may determine whether mechanical work tool 140 is embedded in the ground or has been removed from the ground based on, for example, implement pose data and / or linkage pose data. If mechanical work tool 140 is removed from the ground, controller 145 may cause wireless communication component 160 to transmit the data.

[0076] By determining whether mechanical work tool 140 is embedded in the ground or removed from the ground, controller 145 may prevent wireless communication component 160 from transmitting data that would have caused one or more computing devices 175 (and / or one or more machine learning models) to attempt to identify mechanical work tool 140 if mechanical work tool 140 is embedded in the ground during an excavation operation. As such, controller 145 may conserve computational resources that would have been used to transmit data that may not be useful to one or more computing devices 175 and / or to attempt to identify mechanical work tool 140 if mechanical work tool 140 was embedded in the ground.

[0077] In some implementations, the controller 145 may cause the wireless communication component 160 to transmit the processing information over a wireless connection to one or more computing devices 175, causing the one or more computing devices 175 to process the sensor data based on the processing information. The processing information may include filtering information that identifies how to filter the sensor data (from the stereo camera 155) to obtain filtered data.

[0078] The filtering information may include, for example, information identifying an order for performing the one or more filtering operations described above, and / or information identifying different computing devices (of the one or more computing devices 175) for performing different filtering operations, among other examples. In addition to or as an alternative to including filtering information, the processing information may include information identifying the one or more machine learning models described above, information indicating that a first machine learning model is used to determine information regarding one or more holes, information indicating that a second machine learning model is used to determine information regarding one or more mountains, etc.

[0079] In some implementations, after establishing a wireless connection, controller 145 (and / or wireless communication component 160) may determine the signal strength of satellite device 170 and the signal strength of base station 180. Based on determining the signal strength of satellite device 170 and the signal strength of base station 180, controller 145 may periodically cause wireless communication component 160 to establish a wireless connection with satellite device 170, establish a wireless connection with base station 180, terminate the wireless connection, or not attempt to establish a wireless connection. When controller 145 causes wireless communication component 160 to terminate the wireless connection or not attempt to establish a wireless connection, controller 145 may cause sensor data (from stereo camera 155) to be processed locally on machine 105, as described below.

[0080] In some implementations, after establishing a wireless connection, the controller 145 may determine whether the machine 105 is in an idle or active state, as described above, and / or may determine whether the machine work tool 140 is in a locked or unlocked position, and / or may determine whether the machine 105 is operating in a manual or autonomous mode, and / or may determine the geographic location of the machine 105. The controller 145 may perform such actions periodically and / or based on a trigger event, as described above.

[0081] If controller 145 determines that machine 105 is idle, machine work tool 140 is in a locked position, machine 105 is operating in a manual mode, and / or the geographic location prevents wireless communication component 160 from communicating with satellite device 170 or base station 180, controller 145 may cause wireless communication component 160 to terminate the wireless connection or not attempt to establish a wireless connection. If controller 145 causes wireless communication component 160 to terminate the wireless connection or not attempt to establish a wireless connection, controller 145 may cause sensor data (from stereo camera 155) to be processed locally on machine 105, as described below.

[0082] In some implementations, when a wireless connection is not possible, controller 145 stores sensor data received from stereo camera 155. For example, assume that controller 145 determines that a wireless connection is not possible as described above, and that controller 145 receives sensor data after determining that a wireless connection is not possible. Based on determining that a wireless connection is not possible, controller 145 may cause the sensor data to be stored (or buffered) locally for a threshold time. In some examples, controller 145 may cause the sensor data to be buffered locally using a circular buffer (or ring buffer), a single buffer, or a double buffer, among other examples.

[0083] After the threshold time has expired, controller 145 may determine whether a wireless connection is available (as described above). Assume that controller 145 determines that a wireless connection is available after the threshold time has elapsed. Based on determining that a wireless connection is available after the threshold time has elapsed, controller 145 may cause sensor data to be transmitted to one or more computing devices 175 (as described above). In some cases, if controller 145 determines that a wireless connection is not available after the threshold time has elapsed, controller 145 may cause the sensor data to be processed locally, as described below.

[0084] In some implementations, if a wireless connection is not available, controller 145 may cause machine 105 to pause an operation being performed or initiated by machine 105. For example, assume controller 145 determines (as described above) that a wireless connection is not available and that processed data is not being received from computing device 175. Controller 145 may determine whether machine 105 is performing or initiating an operation that uses the processed data.

[0085] For example, assume that the controller 145 determines (e.g., based on information from the memory of the machine 105) that the machine 105 is performing or initiating an autonomous operation (e.g., an autonomous navigation operation, an autonomous material movement operation, among other examples). Further assume that the processed data (generated by the one or more computing devices 175) includes surface data and / or volumetric data, and that the surface data may be used by the machine 105 for the autonomous navigation operation and the volumetric data may be used by the machine 105 for the autonomous material movement operation. The controller 145 may cause the machine 105 to pause operation based on determining that a wireless connection is not possible before the processed data is received, and based on determining that the machine 105 is performing or initiating an operation that uses the processed data.

[0086] The operation may be paused until the wireless communication component 160 re-establishes the wireless connection and receives process data from the computing device 175 via the wireless connection. In some cases, the controller 145 may cause the machine 105 to pause operation if the controller 145 determines that the wireless connection is not possible for a threshold time (e.g., after transmitting sensor data). In some cases, the controller 145 may provide a notification indicating that the wireless connection is not possible after the threshold time has elapsed. The notification may be provided via the operator cabin 120 (e.g., via an integrated display) and / or provided to the operator's device and / or provided to the back office, among other examples. In some cases, the controller 145 may allow the machine 105 to continue performing the operation if the controller 145 determines that the operation is not an operation that uses the processed data.

[0087] In some implementations, controller 145 may cause the sensor data (from stereo camera 155) to be processed locally for one or more of the reasons described above. Controller 145 may process the sensor data to generate machine-processed data. Controller 145 may process the sensor data in a manner similar to that described above with respect to one or more computing devices 175 processing the sensor data. Controller 145 may, for example, perform one or more operations (described above) to remove noise from the sensor data, reduce noise in the sensor data, remove distorted items from the sensor data, and remove image occlusions from the sensor data, among other examples.

[0088] The controller 145 may generate the machine-filtered data based on performing one or more operations. The one or more operations performed by the controller 145 may be less computationally complex than the one or more operations performed by the one or more computing devices 175. As a result, the machine-filtered data may differ from the data filtered by the one or more computing devices 175 (e.g., the resolution of the machine-filtered data may be less than the resolution of the filtered data from the computing devices 175).

[0089] Controller 145 may process the machine-filtered data using one or more machine learning models (similar to the one or more machine learning models described above) in a manner similar to that described above. As a result of processing the machine-filtered data, controller 145 may generate machine-processed data including machine surface data (similar to the surface data described above) and / or machine volume data (similar to the volume data described above). In some cases, due to computational constraints of machine 105, one or more machine learning models (used by controller 145) may provide output that is less accurate than the output provided by one or more machine learning models used by one or more computing devices 175. In this regard, the machine-processed data may be less accurate than the processed data provided by one or more computing devices 175.

[0090] The controller 145 may determine the volume of the region of interest (e.g., the volume of a pile of material or the volume of material within the mechanical work tool 140, among other examples). In some circumstances, the controller 145 may determine the volume of material within the mechanical work tool 140 based on the geometry of an inner surface of the mechanical work tool 140 in a manner similar to that described above. In some circumstances, the controller 145 may analyze sensor data (e.g., images) to identify the region of interest. The controller 145 may, for example, analyze the images using one or more object detection techniques (e.g., single-shot detector (SSD) techniques and / or YOLO (You Only Look Once) techniques, etc.) to identify the region of interest.

[0091] Controller 145 may generate a graphical representation (e.g., a three-dimensional (3D) graphical representation or a 2.5D graphical representation) based on the image. The graphical representation may represent an area surrounding machine 105 (or a portion of an area surrounding machine 105), including a region of interest. In this regard, the graphical representation may enable controller 145 to determine characteristics of the area surrounding machine 105, including characteristics of the region of interest (e.g., the volume of the region of interest). The graphical representation may be generated based on a disparity map of the image.

[0092] Controller 145 may determine the position and / or orientation of one or more parts of machine 105 based on the instrument pose data and / or machine pose data. Controller 145 may determine coordinates of the region of interest relative to machine 105 based on the position and / or orientation of the one or more parts. Controller 145 may, for example, determine the coordinates of the region of interest relative to a particular part of machine 105. Controller 145 may, for example, consider a particular part of machine 105 to be a center point of a graphical representation (e.g., coordinates "0,0,0" in a 3D graphical representation), and the coordinates of the region of interest may correspond to the coordinates of the graphical representation relative to the center point.

[0093] Controller 145 may identify a portion of the 3D graphical representation (hereinafter referred to as the "3D portion") that corresponds to the region of interest based on the coordinates. Controller 145 may identify the 3D portion to conserve computational resources that would be used to process the entire 3D graphical representation to determine the volume of the region of interest. Controller 145 may determine the volume of the 3D portion using one or more computational models, one or more computational algorithms, and / or other machine algorithms that may be used to determine the volume of the 3D graphical representation.

[0094] In some examples, the controller 145 may determine whether the wireless communication component 160 can establish a wireless connection in a manner similar to that described above (e.g., periodically and / or based on a trigger event). For example, after determining that a wireless connection is not possible, the controller 145 may determine (e.g., periodically and / or based on a trigger event) whether the wireless communication component 160 can re-establish the wireless connection. The controller 145 may cause the wireless communication component 160 to establish the wireless connection based on determining that the wireless communication component 160 can re-establish the wireless connection. The wireless communication component 160 may establish the wireless connection as described above.

[0095] In some circumstances, algorithms (used by one or more computing devices 175 to perform the actions described above) may be updated on one or more computing devices 175 with no or minimal downtime to the machine 105. Algorithms may be updated with new features, tools, and / or information, for example, to gather new information from existing sensors on the machine 105. In this regard, additional value (or functionality) may be added to the machine 105 without retrofitting the machine 105 with new hardware to change how data is processed or with no or minimal downtime to the machine 105.

[0096] With respect to algorithm updates, as new algorithms (e.g., for noise / artifact removal, feature extraction, machine learning model training, among other examples) are developed and validated on non-production machines, the new algorithms may be introduced to production machines in various manners. In one example, a new algorithm may be replaced (on one or more computing devices 175) with the same identifier(s) as the corresponding old algorithm(s) so that existing machines that transmit data (to one or more computing devices 175 for processing) can immediately recognize the improvements upon request. The process is a continuous integration-style update that occurs only on the server side (e.g., only on one or more computing devices 175) and affects all machines that use one or more computing devices 175 for remote processing.

[0097] As another example, a new algorithm can be substituted (on one or more computing devices 175) under new identifier(s), and existing machines can have their on-board software flashed by updates that transmit requests to the new identifier(s). The process is a continuous integration style update that occurs only on the server side (e.g., only on one or more computing devices 175) and a software distribution update that occurs on the client side (e.g., on the machine 105). No hardware changes are required, but new software must be pushed to the machines. This process can be used to manage which machines are upgraded with minimal downtime.

[0098] As yet another example, new algorithms can be provided (on one or more computing devices 175) under new identifier(s), and existing machines can have their existing on-board software configured to use the new identifier(s), for example, through a manual change made through a display. This process is a continuous integration-style update that occurs on the server side and a software configuration change that occurs on the client side. No hardware changes are required to the machines 105. This process can be used to manage which machines are upgraded with little to no downtime.

[0099] The number and arrangement of devices shown in Figure 2 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged devices compared to those shown in Figure 2. Furthermore, two or more of the devices shown in Figure 2 may be implemented within a single device, or a single device shown in Figure 2 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of system 200 may perform one or more functions described as being performed by another set of devices in system 200.

[0100] 3 is a flowchart of an example process 300 for remote processing of sensor data. One or more process blocks of FIG. 3 may be performed by a controller (e.g., controller 145). One or more process blocks of FIG. 3 may be performed by another device or devices separate from or including the device, such as a stereo camera (e.g., stereo camera 155), a wireless communication component (e.g., wireless communication component 160), a satellite device (e.g., satellite device 170), one or more computing devices (e.g., one or more computing devices 175), and / or a base station (e.g., base station 180).

[0101] 3, process 300 may include receiving sensor data of an environment, including the ground on which the work machine is located, from one or more sensor devices on the work machine (block 310). For example, the controller may receive sensor data of the environment, including the ground on which the work machine is located, from one or more sensor devices on the work machine, as described above.

[0102] The one or more sensor devices are one or more first sensor devices, the sensor data is first sensor data, and the method further includes receiving second sensor data from one or more second sensor devices regarding at least one of a position or an orientation of the one or more parts of the work machine.

[0103] 3, the process 300 may include determining whether a wireless connection is available (block 320). For example, the controller may determine whether a wireless connection is available, as described above.

[0104] Process 300 includes determining whether the work machine is in an idle state or an active state or determining whether an implement on the work machine is in a locked position or an unlocked position, and determining whether a wireless connection is possible further includes determining whether a wireless connection is possible when the work machine is in an active state or when the implement is in an unlocked position.

[0105] As further shown in FIG. 3 , process 300 may include selectively transmitting the sensor data to one or more computing devices for processing or processing the sensor data locally by the work machine based on whether a wireless connection is available (block 330).

[0106] The controller may, for example, selectively transmit the sensor data to one or more remote computing devices for processing or process the sensor data locally by the work machine based on whether a wireless connection is available; if a wireless connection is available, the sensor data is transmitted to the one or more remote computing devices and processed by the one or more remote computing devices to provide first processed data, the first processed data including at least one of first surface data regarding one or more portions of the ground surface or first volumetric data regarding a volume of material moved or to be moved by the work machine; and if a wireless connection is not active, the sensor data is processed locally by the work machine to generate second processed data, as described above, the second processed data including at least one of second surface data regarding one or more portions of the ground surface or second volumetric data regarding a volume of material moved or to be moved by the work machine.

[0107] Transmitting the sensor data includes transmitting one or more of: one or more images of the environment, one or more disparity maps of the environment, and / or one or more point clouds of the environment.

[0108] Transmitting the sensor data includes transmitting the sensor data via a satellite device configured for high-speed communication or a cellular network configured for high-speed communication to one or more computing devices for processing.

[0109] Transmitting the sensor data includes transmitting the sensor data to cause one or more remote computing devices to perform filtering operations on the sensor data, the filtering operations including two or more of removing noise from the sensor data, removing image occlusions from the sensor data, or removing information about one or more obstacles in the environment from the sensor data.

[0110] Transmitting the sensor data includes transmitting the sensor data and processing information and causing one or more remote computing devices to process the sensor data based on the processing information to generate processed data, wherein the processing information includes at least one of information identifying a manner in which the sensor data is filtered to obtain filtered sensor data or information identifying one or more machine learning models that process the sensor data or the filtered sensor data to generate the processed data.

[0111] 3, process 300 may include providing the first processed data or the second processed data to facilitate operation of the work machine (block 340). The controller may provide the first processed data or the second processed data to facilitate operation of the work machine, for example, as described above.

[0112] The first volume data or the second volume data includes at least one of information identifying a first volume of material in a bucket of an implement of the work machine or information identifying a second volume of material located in one or more portions of the ground, and providing the first processed data or the second processed data includes providing at least one of information identifying the first volume or information identifying the second volume.

[0113] The first surface data or the second surface data includes at least one of information identifying one or more holes located in one or more portions of the ground or information identifying one or more piles of material located in one or more portions of the ground, and providing the first processed data or the second processed data includes providing at least one of information identifying the one or more holes or information identifying the one or more piles to facilitate at least one of movement of a work machine or operation of an implement of the work machine over the ground.

[0114] The one or more sensor devices are one or more first sensor devices, the sensor data is first sensor data, and the method further includes: selectively transmitting or processing the sensor data includes selectively transmitting or processing the first sensor data and the second sensor data to obtain first processed data or second processed data, respectively.

[0115] Process 300 may include determining that the wireless communication component is unable to receive the processed data after transmitting the sensor data, and preventing the work machine from performing an action based on the processed data if the wireless communication component is unable to receive the processed data after transmitting the sensor data.

[0116] Process 300 may include causing the machine to autonomously navigate a terrain based on surface data regarding one or more portions of the terrain, or causing the machine to autonomously operate an implement to move material based on volumetric data regarding a volume of material moved or to be moved by the implement.

[0117] Although Figure 3 illustrates example blocks of process 300, process 300 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than the blocks depicted in Figure 3. Additionally or alternatively, two or more of the blocks of process 300 may be performed in parallel or combined. [Industrial Applicability]

[0118] The present disclosure relates to machines that selectively transmit sensor data (e.g., from a stereo camera) to one or more remote computing devices for processing or process the sensor data locally based on whether a wireless connection is available. For example, if a wireless connection is available, the sensor data may be transmitted to one or more remote computing devices for processing to generate processed data. Alternatively, if a wireless connection is not available, the sensor data may be processed locally on the machine.

[0119] Processing images from a stereo camera to generate useful information for a machine is a computationally complex and time-consuming task that benefits from the use of high-performance computing resources. Processing images using a machine's typical computing resources presents several challenges. For example, when images are processed by a machine's typical computing resources, information with lower accuracy and / or quality than information generated by high-performance computer resources may be generated. Processing images using a work machine's typical computing resources consumes an excessive amount of computing resources. Such computing resources may be required for various operations of the work machine. Therefore, processing images using typical computing resources may adversely affect the work machine's ability to perform various operations. Moreover, work machines may be used in rocky and harsh conditions that may cause typical computing resources to fail prematurely. Additionally, a machine's typical computing resources may generate information in an inefficient manner.

[0120] The present disclosure solves the above problems. For example, by transmitting sensor data to one or more remote computing devices for processing, the processed data of the present disclosure has higher accuracy and quality than data generated using typical computational resources of a machine. In this regard, the processed data may improve the accuracy of operations performed by the machine, improve the precision of operations performed by the machine, improve the speed of operations performed by the machine, and / or improve the efficiency of operations performed by the machine, among other examples. In addition, by transmitting sensor data to one or more remote computing devices for processing, the processed data is generated in an efficient manner. Furthermore, by communicating with one or more computing devices using a satellite device configured for high-speed communication or a cellular network configured for high-speed communication, the machine may receive the processed data in a timely manner.

[0121] Furthermore, the one or more computing devices are no longer exposed to the rocky and harsh conditions common in many job sites and therefore are less likely to fail prematurely. Additionally, by transmitting sensor data to one or more remote computing devices for processing, the machine conserves computational resources that would otherwise be consumed by processing images locally. Furthermore, by transmitting sensor data to one or more remote computing devices for processing, the machine reduces power consumption, which in turn conserves excessive amounts of power that would otherwise be consumed by typical computational resources when processing images. Additionally, using one or more remote computing devices for processing provides flexibility with respect to equipment upgrades for the machine.

[0122] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations may be made in light of the above disclosure or acquired from practice of the implementations. Furthermore, any implementations described herein may be combined unless a reason why one or more implementations cannot be combined is explicitly provided in the foregoing disclosure. Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of various implementations. While each dependent claim listed below may depend directly on only one claim, the disclosure of various implementations includes each dependent claim in combination with all other claims in the claim group.

[0123] As used herein, the terms "a," "an," and "set" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the term "the" is intended to include one or more items referenced in conjunction with "the" and may be used interchangeably with "one or more." Furthermore, the phrase "based on" is intended to mean "based at least in part on," unless expressly stated otherwise. Also, as used herein, the term "or" is intended to be inclusive when used in a series and may be used interchangeably with "and / or" unless expressly stated otherwise (e.g., when used in combination with "either" or "only one of"). Additionally, as used herein, the term "eliminate" encompasses the partial reduction of a property or characteristic, as well as the elimination of that property or characteristic.

Claims

1. A method performed by a controller (145) of a work machine (105), comprising: receiving sensor data of an environment including a ground surface on which the work machine (105) is located from one or more sensor devices (155) of the work machine (105); determining whether a wireless connection between the work machine (105) and one or more remote computing devices (175) is possible; selectively transmitting the sensor data to the one or more remote computing devices (175) for processing or processing the sensor data locally by the work machine (105) based on determining whether the wireless connection is available; If the wireless connection is available, the sensor data is transmitted to the one or more remote computing devices (175), causing the one or more remote computing devices (175) to process the sensor data to provide first processed data, the first processed data including at least one of first surface data regarding one or more portions of the ground surface, or first volumetric data regarding a volume of material moved or to be moved by the work machine (105); if the wireless connection is not possible, the sensor data is processed locally by the work machine (105) to generate second processed data, the second processed data including at least one of second surface data relating to one or more portions of the ground surface or second volumetric data relating to a volume of material moved or to be moved by the work machine (105); The first volume data or the second volume data is information identifying a first volume of material within an implement of said work machine (105); information identifying a second volume of material located in one or more portions of the ground surface; and providing the first processed data or the second processed data includes providing at least one of the information identifying the first volume or the information identifying the second volume. And, providing the first processed data or the second processed data to facilitate operation of the work machine (105); A method comprising:

2. the one or more sensor devices (155) (150) are one or more first sensor devices (155), and the sensor data is first sensor data; The method comprises: receiving second sensor data relating to at least one of a position or an orientation of one or more portions (130, 135, and 140) of the work machine from one or more second sensor devices (150); Selectively transmitting or processing the sensor data includes selectively transmitting or processing the first sensor data and the second sensor data to obtain the first processed data or the second processed data, respectively. The method of claim 1.

3. transmitting the sensor data transmitting the sensor data to the one or more remote computing devices (175) for processing; a satellite device (170) configured for high speed communications; or A cellular network (180) configured for high speed communication To send via 3. The method of claim 1 or 2, comprising:

4. The second volume data is including information identifying a third volume of material loaded in front of the device; The first surface data or the second surface data is information identifying one or more holes located in one or more portions of said ground; or information identifying one or more piles of material located on one or more portions of the ground; and Providing the first processed data or the second processed data includes: providing at least one of information identifying the one or more holes or information identifying the one or more peaks; movement of the work machine over the ground; or Operation of the implements of the work machine To facilitate at least one of the following: The method of claim 2 , comprising:

5. determining whether the work machine (105) has been idle or active for a threshold time; or Determining whether an implement (130, 135, 140) of said work machine (105) is in a locked or unlocked position. further comprising Determining whether the wireless connection is possible includes: Whether the wireless connection is possible, when the work machine (105) is in the active state; or When the device (130, 135, 140) is in the unlocked position If at least one of the following is true, the judgment shall be made. The method of claim 1 , comprising:

6. 1. A system comprising: one or more sensor devices (155) configured to acquire sensor data of an environment including a ground surface on which the work machine (105) is located; a controller (145); Equipped with The controller (145) determining whether a wireless connection between the work machine (105) and one or more remote computing devices (175) is possible; transmitting, if the wireless connection is available, the sensor data to the one or more remote computing devices (175) for processing using a wireless communication component; the sensor data is transmitted to the one or more remote computing devices (175), causing the one or more remote computing devices (175) to filter the sensor data and generate processed data; And, receiving the processed data from the one or more remote computing devices (175); the processed data includes at least one of surface data relating to one or more portions of the ground surface or volumetric data relating to a volume of material moved or to be moved by the work machine (105); The surface data is information identifying one or more holes located in one or more portions of said ground; or information identifying one or more piles of material located on one or more portions of the ground; and providing the processed data includes providing at least one of information identifying the one or more holes or information identifying the one or more peaks to facilitate operation of the work machine (105). And, A system configured to:

7. the controller (145) is configured to provide the volumetric data to facilitate material movement operations to provide the processed data; The volume data is information identifying a first volume of material within an implement of said work machine (105); information identifying a second volume of material located in one or more portions of the ground surface; or Information identifying the volume of material moved by said work machine (105) over a period of time. at least one of: The system of claim 6.

8. The controller (145) transmits the sensor data by: configured to transmit the sensor data and processing information to the one or more remote computing devices (175) and cause the one or more remote computing devices (175) to process the sensor data based on the processing information to generate the processed data; The processing information is information identifying how to filter the sensor data to obtain filtered sensor data; or Information identifying one or more machine learning models that process the sensor data or the filtered sensor data to generate the processed data at least one of:

8. The system according to claim 6 or 7.

9. The controller (145) determining that the wireless connection is not possible; based on determining that the wireless connection is not possible, causing the sensor data to be stored for a threshold time; determining whether the wireless connection is possible after the threshold time has elapsed; and configured to: The controller (145) transmits the sensor data to the one or more computing devices (175): and causing the sensor data to be transmitted to the one or more remote computing devices (175) based on determining that the wireless connection is available after the threshold time has elapsed. The system of claim 6.

10. The controller (145) determining that the wireless communication component is unable to receive the processed data after transmitting the sensor data; preventing the work machine (105) from performing an action based on the processed data if the wireless communication component is unable to receive the processed data after transmitting the sensor data; configured to: The system of claim 6.

Images