Remote processing of sensor data for machine operation

Abstract

The controller may receive sensor data of an environment, including the ground on which the work machine is located, from one or more sensor devices of the work machine. The controller may determine whether a wireless connection is possible. The controller may selectively transmit the sensor data to one or more remote computing devices for processing or process the sensor data locally by the work machine based on whether a wireless connection is possible. If a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices and has the one or more remote computing devices process the sensor data to provide first processed data. If a wireless connection is not possible, the sensor data is processed locally by the work machine to generate second processed data.

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to processing machine sensor data, for example, to remote processing of machine sensor data. [Background technology]

[0002] A work machine (e.g., an excavator) may be equipped with stereoscopic ("stereo") cameras that capture images of the environment surrounding the machine. The images may be processed to generate information that may be used for various operations performed by the work machine. Processing the images (e.g., to generate such information) may be computationally complex and time consuming.

[0003] Processing the images using the typical computational resources of a work machine consumes excessive amounts of computational resources that may be required for various operations of the work machine, and thus may adversely affect the ability of the work machine to perform such operations. Moreover, the work machine may be used in rocky and harsh conditions that may cause premature failure of the typical computational resources.

[0004] U.S. Patent No. 6,856,879 (the '879 patent) discloses that the time each operator operates the construction machine can be reliably managed. The '879 patent also discloses that the maintenance performed by the mechanic on the construction machine can be accurately learned by the manager in real time without delay. The '879 patent further discloses that the mechanic is given a work instruction, and the performance and labor management of the mechanic can be easily and accurately performed. The '879 patent further discloses that the installation status of the construction machine can be accurately learned in real time, thereby enabling a rapid response to customer requests. However, the '879 patent does not address the problems associated with having the construction machine perform computationally complex and time-consuming tasks that require high performance computing resources, as discussed above.

[0005] The present disclosure solves one or more of the above problems and / or other problems in the art. Summary of the Invention

[0006] A method performed by a controller of the work machine includes receiving sensor data of an environment including a ground surface on which the work machine is located from one or more sensor devices of the work machine; determining whether a wireless connection is possible; selectively transmitting the sensor data to one or more remote computing devices for processing or processing the sensor data locally by the work machine based on determining whether a wireless connection is possible, where if a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices and has the one or more remote computing devices process the sensor data to provide first processed data, the first processed data including at least one of first surface data regarding one or more portions of the ground surface or first volumetric data regarding a volume of material moved or to be moved by the work machine; and where if a wireless connection is not possible, the sensor data is processed locally by the work machine to generate second processed data, the second processed data including at least one of second surface data regarding one or more portions of the ground surface or second volumetric data regarding a volume of material moved or to be moved by the work machine; and providing the first processed data or the second processed data to facilitate operation of the work machine.

[0007] 1. A system comprising: one or more sensor devices configured to acquire sensor data of an environment including a ground surface on which a work machine is located; and a controller, the controller configured to: determine whether a wireless connection is possible; transmit, using a wireless communication component, the sensor data to one or more remote computing devices for processing, where if a wireless connection is possible, the sensor data is transmitted to the one or more remote computing devices causing the one or more remote computing devices to filter the sensor data and generate processed data; receive the processed data from the one or more remote computing devices, the processed data including at least one of surface data regarding one or more portions of the ground surface or volumetric data regarding a volume of material moved or to be moved by the work machine; and provide the processed data to facilitate operation of the work machine.

[0008] 1. A machine comprising: an implement; one or more sensor devices configured to obtain sensor data of an environment including a ground surface on which the machine is located; a wireless communication component; and a controller, the controller configured to: determine whether a wireless connection is possible; use the wireless communication component to send a request to one or more remote computing devices based on determining that a wireless connection is possible, the request including the sensor data being sent to the one or more remote computing devices causing the one or more remote computing devices to filter the sensor data and generate processed data after filtering the sensor data; receive the processed data from the one or more remote computing devices, the processed data including at least one of surface data regarding one or more portions of the ground surface or volumetric data regarding a volume of material moved or to be moved by the implement; and provide the processed data to facilitate at least one of movement of the machine within an environment or operation of an implement of the machine. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram of an example implementation described herein. [Diagram 2] FIG. 1 is a diagram of an exemplary system described herein. [Diagram 3] 1 is a flowchart of an exemplary process for remote processing of sensor data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present disclosure relates to a machine that selectively transmits sensor data (e.g., from a stereo camera) to one or more computing devices for processing or processes the sensor data locally based on whether a wireless connection is available. A wireless connection may be available when the machine establishes a wireless connection with one or more computing devices, or when the machine has established a wireless connection and the rate of data transmission (of the wireless connection) meets a transmission rate threshold. The sensor data may be transmitted to one or more computing devices for processing, for example, when a wireless connection is available. The sensor data may be transmitted via a satellite device configured for high-speed communication or via a cellular network (e.g., a 5G network) configured for high-speed communication. The one or more computing devices may include one or more cloud computing devices (e.g., one or more high-performance remote computing devices).

[0011] One or more computing devices may process the sensor data to generate processed data. The processed data may be transmitted to the machine (e.g., via a satellite device or cellular network) to facilitate operation of the machine (e.g., to facilitate material movement operations, navigation operations, among other examples). The machine may suspend transmission of sensor data and buffer the sensor data when wireless communication is stopped or when a wireless connection is established and the rate of data transmission (of the wireless connection) does not meet a transmission rate threshold. The machine may resume transmission of sensor data when wireless communication is re-established or when the rate of data transmission (of the wireless connection) meets a transmission rate threshold. In some cases, when wireless connection is not possible, the machine may suspend operations (of the machine) that depend on the processed data (e.g., suspend autonomous navigation operations, suspend autonomous material movement operations, among other examples). In some examples, when wireless connection is not possible, the sensor data may be processed locally on the machine. A wireless connection may not be possible if the machine has not established a wireless connection with one or more computing devices, or if the machine has established a wireless connection and the rate of data transmission does not meet a transmission rate threshold. The machine may provide a notification (e.g., to a machine operator) indicating that the sensor data cannot be transmitted to one or more remote computing devices and / or that the sensor data is being processed locally on the machine.

[0012] The term "machine" may refer to a machine that performs an operation involved in industries such as mining, construction, agriculture, transportation, etc. Additionally, one or more implements may be connected to the machine. By way of example, the machine may include a construction vehicle, a work vehicle, or a similar vehicle involved in the above industries.

[0013] 1 is a diagram of an example implementation 100 described herein. As shown in FIG. 1, the example implementation 100 includes a machine 105. The machine 105 is embodied as an earthmoving machine, such as an excavator. Alternatively, the machine 105 may be another type of machine, such as a dozer.

[0014] As shown in FIG. 1, the machine 105 includes ground engaging members 110, a machine body 115, an operator cabin 120, and a pivoting element 125. The ground engaging members 110 may include tracks (as shown in FIG. 1), wheels, and / or rollers, etc., for the propulsion machine 105. The ground engaging members 110 are mounted to a rotating frame (not shown) and driven by one or more engines and drive trains (not shown). The machine body 115 is mounted to the rotating frame (not shown). The operator cabin 120 is supported by the machine body 115. The operator cabin 120 includes an integrated display (not shown) and operator controls 124, such as, for example, an integrated joystick. The operator controls 124 may include one or more input components.

[0015] In the case of an autonomous machine, the operator controls 124 may not be designed for use by an operator, but rather may be designed to operate independently of an operator. In this case, the operator controls 124 may include one or more input components that provide input signals that are used by another component without any operator input, for example. The pivoting element 125 may include one or more components that allow the rotating frame (and machine body 115) to rotate (or pivot). For example, the pivoting element 125 may allow the rotating frame (and machine body 115) to rotate (or pivot) relative to the ground engaging members 110.

[0016] As shown in FIG. 1, the machine 105 includes a boom 130, a stick 135, and a machine work tool 140. The boom 130 is pivotally mounted to its proximal end on the machine body 115 and articulates relative to the machine body 115 by one or more fluid actuated cylinders (e.g., hydraulic or pneumatic cylinders), electric motors, and / or other electromechanical components. The stick 135 is pivotally mounted to a distal end of the boom 130 and articulates relative to the boom 130 by one or more fluid actuated cylinders, electric motors, and / or other electromechanical components. The boom 130 and / or the stick 135 may be referred to as a linkage. The machine work tool 140 is mounted to a distal end of the stick 135 and may articulate relative to the stick 135 by one or more fluid actuated cylinders, electric motors, and / or other electromechanical components. The machine work tool 140 may be a bucket (as shown in FIG. 1) or another type of tool that may be mounted to the stick 135. The machine work tool 140 may be referred to as an implement.

[0017] As shown in FIG. 1 , the machine 105 includes a controller 145 (e.g., an electronic control module (ECM), a computer vision controller, an autonomy controller, among other examples), one or more inertial measurement units (IMUs) 150 (referred to herein individually as “IMU 150” and collectively as “IMUs 150”), a stereo camera 155, a wireless communication component 160, and one or more sensor devices 165. The controller 145 may control and / or monitor the operation of the machine 105. For example, the controller 145 may control and / or monitor the operation of the machine 105 based on signals from the operator controls 124, signals from the IMUs 150, signals from the stereo camera 155, signals from the wireless communication component 160, and / or signals from the one or more sensor devices 165.

[0018] 1, the IMU 150 is mounted at various locations on components or portions of the machine 105, such as, for example, the machine body 115, the boom 130, the stick 135, and the machine work tool 140. The IMU 150 includes one or more devices capable of receiving, generating, storing, processing, and / or providing signals indicative of the position and orientation of the component of the machine 105 on which the IMU 150 is mounted. For example, the IMU 150 may include one or more accelerometers and / or one or more gyroscopes. The one or more accelerometers and / or one or more gyroscopes generate and provide signals that can be used to determine the position and orientation of the IMU 150 relative to a reference frame, and thus the position and orientation of the component. Although the examples discussed herein refer to the IMU 150, the disclosure is applicable to using one or more other types of sensor devices that can be used to determine the position and orientation of the component of the machine 105.

[0019] The stereo camera 155 may include one or more devices capable of acquiring and providing sensor data of the environment, including the ground on which the machine 105 is located. The sensor data may include image data (e.g., three-dimensional (3D) image data) of the environment. Although the examples discussed herein refer to the stereo camera 155, the present disclosure is applicable to using one or more other types of devices, such as light detection and ranging (LIDAR) devices and / or radio detection and ranging (RADAR) devices, among other examples of devices (e.g., sensor devices) configured to provide data about the environment. Unless otherwise specified, any reference hereinafter to the stereo camera 155 should be understood to encompass these other devices.

[0020] 1 illustrates a single stereo camera 155 as being included on the machine 105. In practice, the machine 105 may include one or more additional stereo cameras 155 and / or otherwise positioned stereo cameras 155 (e.g., stereo cameras 155 provided on other portions of the machine 105). The stereo cameras 155 may be provided on the boom 130, the stick 135, and the machine work tool 140, for example.

[0021] Wireless communication component 160 includes one or more devices that enable machine 105 to communicate with other devices, as described herein. Wireless communication component 160 may include a transceiver, separate transmitters and receivers, and antennas, among other examples. Wireless communication component 160 may transmit sensor data (e.g., from stereo camera 155) to one or more computing devices and receive processed data from one or more computing devices, as described herein.

[0022] Wireless communication component 160 may be configured to transmit the sensor data as a continuous flow of data from stereo camera 155. In some examples, wireless communication component 160 may transmit the sensor data and receive processed data via a satellite device, via a base station, and / or via another device that may enable wireless communication component 160 to transmit data to and receive data from one or more computing devices, as described below.

[0023] The wireless communication component 160 may communicate with one or more computing devices over a network, including one or more wired and / or wireless networks, such as a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, a Code Division Multiple Access (CDMA) network, a 3G network, a 4G network, or another type of cellular network), the Internet, a cloud computing network, a wireless local area network (LAN), an intranet, a fiber optic based network, a wide area network (WAN), a public land mobile network (PLMN), a metropolitan area network (MAN), a telephone network (e.g., the Public Switched Telephone Network (PSTN)), a private network, and an ad-hoc network, and / or a combination of these or other types of networks, among other examples.

[0024] Sensor devices 165 include one or more devices that can receive, generate, store, process, and / or provide signals related to one or more components of machine 105. For example, sensor devices 165 may include hydraulic displacement sensors, pressure sensors (e.g., for hydraulic cylinders), machine speed sensors, engine RPM sensors, and / or Global Positioning System (GPS) devices, among other examples.

[0025] 1, exemplary implementation 100 includes satellite device 170, including one or more computing devices 175-1 through 175-N (hereinafter collectively referred to as computing devices 175 and individually as computing device 175), and also includes base station 180. Satellite device 170 may include one or more devices configured for high-speed communications (e.g., geostationary satellites configured for high-speed communications). Satellite device 170 may be configured to provide, for example, high-speed Internet access (e.g., to machine 105 via wireless communications component 160).

[0026] Computing device (or remote computing device) 175 may include one or more devices configured to process sensor data to generate processed data, as described herein. Computing device 175 may be implemented as a single computing device (e.g., a single server) or multiple computing devices (e.g., multiple servers), such as multiple computing devices in one or more data centers. In some examples, computing device 175 may be included in a cloud computing environment. Additionally or alternatively, computing device 175 may be included in a back office system.

[0027] The base station 180 may be connected to a cellular network (e.g., as described above). The base station 180 may be connected to a 5G network, for example. The base station 180 may include a base transceiver station, a radio base station, a Node B, an evolved Node B (eNB), a next generation Node B (gNB), a base station subsystem, a cellular site, a cellular tower (e.g., a cell phone tower, a mobile phone tower, etc.), an access point, a transmit / receive point (TRP), a radio access node, a macro cell base station, a micro cell base station, a pico cell base station, and / or a femto cell base station, or a similar type of device.

[0028] As noted above, Figure 1 is provided as an example. Other examples may differ from the example described with reference to Figure 1.

[0029] Figure 2 is a diagram of an example system 200 described herein. As shown in Figure 2, system 200 includes controller 145, IMU 150, stereo camera 155, wireless communication component 160, one or more sensor devices 165, one or more computing devices 175, along with satellite device 170 and base station 180. Controller 145 may be configured to cause sensor data from stereo camera 155 to be processed by one or more computing devices 175 or to be processed locally on machine 105, as described in further detail below.

[0030] The IMUs 150 may include one or more first IMUs 150, one or more second IMUs 150, and one or more third IMUs 150. The one or more first IMUs 150 may be located on the boom 130 and / or stick 135. The one or more first IMUs 150 may generate signals that may be used to determine a position and / or orientation of the boom 130 and / or stick 135, which may be used to generate linkage pose data indicative of the position and / or orientation of the boom 130 and / or stick 135. The one or more second IMUs 150 may be located on the machine work tool 140. The one or more second IMUs 150 may generate signals that may be used to determine a position and / or orientation of the machine work tool 140, which may be used to generate implement pose data indicative of the position and / or orientation of the machine work tool 140. The one or more third IMUs 150 may be located on the machine body 115. The one or more third IMUs 150 may generate signals that can be used to determine the position and / or orientation of the mechanical body 115, and the signals may be used to generate machine pose data indicative of the position and / or orientation of the mechanical body 115.

[0031] The IMU 150 may provide the linkage pose data, the implement pose data, and / or the machine pose data to the controller 145 periodically (e.g., every 30 seconds, every minute, every 5 minutes, among other examples). Additionally or alternatively, the IMU 150 may provide the linkage pose data, the implement pose data, and / or the machine pose data to the controller 145 based on a trigger event. The trigger event may include, among other examples, a request from the controller 145, detection of movement of the machine body 115, detection of movement of the machine work tool 140, detection of movement of the machine body 115, detection of movement of the boom 130 and / or stick 135, a request from an operator, a request from a back office system, initiation of a new task by the machine 105, relocation of the machine 105 to a different geographic location, after the machine 105 completes a digging cycle and / or after the machine work tool 140 completes digging, and during the swing before unloading material. The linkage pose data, instrument pose data, and / or machine pose data may be transmitted to the one or more remote computing devices at the same rate that the sensor data was transmitted to the one or more remote computing devices. The linkage pose data, instrument pose data, and / or machine pose data may be transmitted in this manner to synchronize the sensor data with the linkage pose data, instrument pose data, and / or machine pose data. Alternatively, the linkage pose data, instrument pose data, and / or machine pose data may be time-stamped so that images (from the sensor data) can be registered to world coordinates using data from the IMU 150 and correlated with other data from the machine 105. In cases where the camera frames and sensor time samples cannot be precisely synchronized, a time interpolation of the sensor data may be performed to time-align the sensor data with the linkage pose data, instrument pose data, and / or machine pose data.

[0032] The stereo camera 155 may be configured to acquire sensor data of the environment, including the terrain surrounding the machine 105 and including the ground on which the machine 105 is located. The sensor data may include image data of the environment (e.g., images of the environment). In some cases, the sensor data may include 3D image data of the environment. The stereo camera 155 may provide the sensor data to the controller 145 periodically (every second, every second, among other examples). Additionally or alternatively, the stereo camera 155 may provide the sensor data to the controller 145 based on a trigger event (e.g., a request from the controller 145).

[0033] Wireless communication component 160 may be configured to establish a wireless connection with one or more computing devices 175 via satellite device 170 or via base station 180 (e.g., establish a wireless connection with satellite device 170 or with base station 180 and thus with one or more computing devices). Wireless communication component 160 may be configured to attempt to establish a wireless connection periodically (e.g., every 30 seconds, every minute, among other examples) and / or based on a trigger event (e.g., based on a request from controller 145, based on receiving an explicit indication that machine 105 is operating in an autonomous mode, based on initiation of wireless communication component 160, among other examples).

[0034] The wireless communication component 160 may transmit the sensor data (captured by the stereo camera 155) over the wireless connection to one or more computing devices 175 for processing. The one or more computing devices 175 may process the sensor data to generate processed data, and the wireless communication component 160 may receive the processed data from the one or more computing devices 175 over the wireless connection.

[0035] Wireless communication component 160 may be configured to prioritize establishing a wireless connection with satellite device 170 (e.g., over establishing a wireless connection with base station 180). Wireless communication component 160 may be configured to establish a wireless connection with satellite device 170, for example, if the signal strength of base station 180 does not exceed the signal strength of satellite device 170 by more than a threshold difference. As another example, wireless communication component 160 may be configured to establish a wireless connection with satellite device 170 if the signal strength of satellite device 170 meets a signal strength threshold regardless of the signal strength of base station 180.

[0036] Wireless communication component 160 may be pre-configured with information identifying the threshold difference, receive information identifying the threshold difference from a device of an operator of machine 105, or receive information identifying the threshold difference from a back office system, among other examples. As an alternative to prioritizing establishing a wireless connection with satellite device 170, wireless communication component 160 may be configured to prioritize establishing a wireless connection with base station 180 (e.g., over establishing a wireless connection with satellite device 170) in a manner similar to that described above with respect to satellite device 170.

[0037] In some implementations, wireless communication component 160 may be configured to establish a wireless connection based on the relative signal strength of satellite device 170 and the signal strength of base station 180. Wireless communication component 160 may be configured to establish a wireless connection with satellite device 170, for example, when the signal strength of satellite device 170 exceeds the signal strength of base station 180. Alternatively, wireless communication component 160 may be configured to establish a wireless connection with base station 180 when the signal strength of base station 180 exceeds the signal strength of satellite device 170.

[0038] Wireless communication component 160 (and / or controller 145) may periodically (e.g., every 15 seconds, every 30 seconds, among other examples) determine the signal strength of satellite device 170 and the signal strength of base station 180. In this regard, wireless communication component 160 may periodically alternate between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180 based on the periodic determination of the signal strength of satellite device 170 and the signal strength of base station 180.

[0039] As an example of periodically alternating the establishment of a wireless connection, assume that wireless communication component 160 has established a wireless connection with satellite device 170. Wireless communication component 160 may terminate the wireless connection with satellite device 170 and establish a wireless connection with base station 180 when wireless communication component 160 (and / or controller 145) determines that the signal strength of base station 180 has exceeded the signal strength of satellite device 170 (and vice versa).

[0040] In addition to or as an alternative to periodically determining signal strength, wireless communication component 160 (and / or controller 145) may determine the signal strength of satellite device 170 and the signal strength of base station 180 based on a trigger event (e.g., based on a request from controller 145, based on an indication of a change in operation of machine 105, among other examples). Wireless communication component 160 may alternate between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180 based on the trigger event.

[0041] By alternating (e.g., periodically and / or based on a trigger event) between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180, wireless communication component 160 may maintain communication with one or more computing devices 175 and may attempt to minimize interruptions in the transmission of sensor data and / or reception of processed data. By preventing interruptions in the transmission of sensor data and / or reception of processed data, wireless communication component 160 and / or controller 145 may prevent any interruptions in operations performed by machine 105.

[0042] In some examples, wireless communication component 160 may determine that a wireless connection with satellite device 170 cannot be established if the signal strength of satellite device 170 does not meet the signal strength threshold. Wireless communication component 160 may similarly determine that a wireless connection with base station 180 cannot be established if the signal strength of base station 180 does not meet the signal strength threshold. Controller 145 may determine that sensor data is processed locally on machine 105 if the signal strength of satellite device 170 and the signal strength of base station 180 do not both meet the signal strength threshold. Wireless communication component 160 may be preconfigured with information identifying the signal strength threshold, receive information identifying the signal strength threshold from an operator's device, or receive information identifying the signal strength threshold from a back office system, among other examples.

[0043] The wireless communication component 160 may be configured to establish a wireless connection based on the output of the machine learning model. The wireless communication component 160 (and / or the controller 145) may use the machine learning model, for example, to predict whether the wireless communication component 160 should establish a wireless connection with a satellite device 170, whether the wireless communication component 160 should establish a wireless connection with a base station 180, or whether the wireless communication component 160 should not establish a wireless connection with a satellite device 170 or a base station 180.

[0044] Controller 145 may provide as input to the machine learning model information identifying a current geographic location and / or current geographic region of machine 105, information identifying one or more obstacles in the current geographic location and / or current geographic region (e.g., one or more images of one or more obstacles), information identifying a time of day, and / or information identifying expected weather conditions, among other examples. The input may be obtained from an operator's device, stereo camera 155, and / or one or more sensor devices 165, among other examples. Based on the input, the machine learning model may provide as output expected connection information indicating whether wireless communication component 160 should establish a wireless connection with satellite device 170, whether to establish a wireless connection with base station 180, or whether to not establish a wireless connection with satellite device 170 or base station 180.

[0045] The machine learning model may be trained using historical data including, among other examples, historical data of geographic locations and / or geographic areas, historical data (e.g., images) of obstacles at past geographic locations and / or geographic areas, historical data of past times, weather conditions at past geographic locations and / or geographic areas, historical signal strengths of satellite devices 170 and base stations 180 at past geographic locations and / or geographic areas (e.g., in light of past obstacles, past times, past weather conditions, among other examples), historical latency of satellite devices 170 and base stations 180 at past geographic locations and / or geographic areas, and / or historical data indicative of established wireless connections (e.g., satellite devices 170 or base stations 180).

[0046] The machine learning models may be generated and trained by a trainer device, which may be a separate hardware or software component (not shown). The trainer device may be included in a back office system or included in one or more computing devices 175, among other examples. The trainer device may provide the machine learning models to the machine 105 for use by the controller 145 and / or the wireless communication component 160. The trainer device may update and provide the machine learning models to the machine 105 (e.g., based on a schedule, upon request, per trigger, periodically, among other examples). In some cases, the controller 145 may obtain additional training data (e.g., additional past data similar to the past data described above) so that the machine learning models are retrained based on the additional training data. The controller 145 may, for example, provide the additional training data to the trainer device to retrain the machine learning models. The machine learning models may be retrained periodically and / or based on a trigger event.

[0047] In training the machine learning model, the training device may separate the training data into a training set (e.g., a dataset for training the machine learning model), a validation set (e.g., a dataset used to evaluate the fit of the machine learning model and / or fine-tune the machine learning model), and / or a test set (e.g., a dataset used to evaluate the final fit of the machine learning model), etc. The training device may preprocess and / or perform dimensionality reduction to reduce the training data to a minimum feature set. By training the machine learning model on this minimum feature set, the training device may reduce processing for training the machine learning model and may apply classification techniques to the minimum feature set.

[0048] The training device may use classification techniques, such as logistic regression, random forest, and / or gradient boosting machine learning (GBM) techniques, to determine a classification outcome (e.g., whether to establish a wireless connection with the satellite device 170 or the base station 180). The training device may use a naive Bayesian classifier technique as an addition or alternative to using classification techniques. In this case, the device may perform binary recursive partitioning to divide the training data of a minimum feature set into partitions and / or branches and perform predictions (e.g., whether to establish a wireless connection with the satellite device 170 or the base station 180) using the partitions and / or branches. Based on the use of recursive partitioning, the training device may reduce the use of computational resources associated with manual linear classification and analysis of data items, thereby allowing models to be trained using thousands, millions, or billions of data items, resulting in a more accurate model than would be obtained using fewer data items.

[0049] The training device may train the machine learning model using a supervised training procedure that includes receiving input to the machine learning model from a subject matter expert (e.g., one or more operators associated with the machine 105 and / or one or more machines), which may reduce the time and / or amount of processing resources, etc., to train the machine learning model as compared to an unsupervised training procedure. The training device may use one or more other model training techniques, such as neural network techniques and / or latent semantic indexing techniques.

[0050] For example, the training device may execute artificial neural network processing techniques (e.g., using a two-layer feedforward neural network architecture and / or a three-layer feedforward neural network architecture, etc.) to perform pattern recognition on patterns of establishing wireless connections with satellite devices 170 or base stations 180. In this case, the use of artificial neural network processing techniques may improve the accuracy of the machine learning models generated by the training device by being more robust to noisy, inaccurate, or incomplete data and by enabling the training device to detect patterns and / or trends that cannot be detected by human analysts or systems using less complex techniques.

[0051] In the examples that follow, it is assumed that stereo camera 155 provides sensor data to controller 145. Stereo camera 155 may, for example, provide sensor data to controller 145 in a manner similar to that described above. It is assumed that stereo camera 155 receives a request for sensor data from controller 145, for example. Stereo camera 155 may provide the sensor data to controller 145 based on receiving the request. Controller 145 may receive the sensor data from stereo camera 155 and determine whether to cause the sensor data to be sent to one or more computing devices 175 for processing or to cause the sensor data to be processed locally on machine 105.

[0052] The controller 145 may determine whether a wireless connection is possible as part of determining whether to have the sensor data transmitted to one or more computing devices 175 for processing or to have the sensor data processed locally on the machine 105. A wireless connection may be possible if the wireless communication component 160 establishes a wireless connection with a satellite device 170 or a base station 180, or if the wireless communication component 160 has established a wireless connection and the rate of data transmission meets a transmission rate threshold. The transmission rate threshold may be determined by an operator, determined by a back office system, or stored in memory of the machine 105, among other examples. The controller 145 may cause the wireless communication component 160 to transmit the sensor data to one or more computing devices 175 via a wireless connection if a wireless connection is possible. Alternatively, the controller 145 may cause the sensor data to be processed locally on the machine 105 if a wireless connection is not possible. A wireless connection may not be possible if the wireless communication component 160 has not established a wireless connection with the satellite device 170 or base station 180, or if the wireless communication component 160 has established a wireless connection and the rate of data transmission does not meet the transmission rate threshold.

[0053] In some examples, controller 145 may receive wireless connection information from wireless communication component 160 indicating whether a wireless connection is possible, and controller 145 may determine whether a wireless connection is possible based on the wireless connection information. In some examples, controller 145 may determine whether wireless communication component 160 needs to establish a wireless connection (e.g., with satellite device 170 or base station 180) and provide instructions to wireless communication component 160 to establish a wireless connection based on determining whether wireless communication component 160 will establish a wireless connection with satellite device 170 or will establish a wireless connection with base station 180.

[0054] The controller 145 may receive wireless connection information from the wireless communication component 160 after providing the instructions and may determine whether a wireless connection is possible based on the wireless connection information. The controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection without determining whether the wireless communication component 160 can establish a wireless connection, as an alternative to providing instructions based on determining whether the wireless communication component 160 needs to establish a wireless connection. The controller 145 may provide instructions based on receiving sensor data, for example.

[0055] As part of determining whether the wireless communication component 160 establishes a wireless connection, the controller 145 may determine whether the machine 105 is in an idle or active state, determine whether the machine work tool 140 is in a locked or unlocked position, determine whether the signal strength of the satellite device 170 or the signal strength of the base station 180 meets a signal strength threshold, determine whether the machine 105 is operating in a manual or autonomous mode, and determine the geographic location of the machine 105, among other examples. As an example, the controller 145 may obtain machine speed data identifying a ground speed of the machine 105 and / or engine speed data identifying an engine speed of the machine 105 from one or more sensor devices 165. The controller 145 may determine whether the machine 105 is in an idle or active state based on the ground speed and / or the engine speed.

[0056] The controller 145 may determine that the machine 105 is in an idle state, for example, if the ground speed does not meet the ground speed threshold and / or if the engine speed does not meet the engine speed threshold. Conversely, the controller 145 may determine that the machine 105 is in an active state if the ground speed meets the ground speed threshold and / or if the engine speed meets the engine speed threshold. The controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection if the machine 105 is in an active state or if the machine 105 has been in an idle state for a time that does not meet the threshold time. Alternatively, the controller 145 may provide instructions to the wireless communication component 160 not to establish a wireless connection if the machine 105 has been in an idle state for a time that meets the threshold time.

[0057] The controller 145 may determine whether the mechanical work tool 140 is in a locked or unlocked position based on an input from a lever (or button) in the operator cabin 120. The operator may lock (or secure) or unlock the mechanical work tool 140, for example, using a lever. The operator may lock or unlock a hydraulic system associated with the mechanical work tool 140, for example, using a lever. When the hydraulic system is locked, hydraulic fluid may not flow to the linkage hydraulic cylinders, swing pump motor, or track pump motor of the hydraulic system. When the hydraulic system is unlocked, hydraulic fluid may flow to the linkage hydraulic cylinders, swing pump motor, or track pump motor of the hydraulic system. The lever may generate a value indicative of whether the mechanical work tool 140 is locked or unlocked. The controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection when the mechanical work tool 140 is in the unlocked position. The controller 145 may obtain information identifying the signal strength of the satellite device 170 or information identifying the signal strength of the base station 180 from the wireless communication component 160. Controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection if the signal strength of satellite device 170 or the signal strength of base station 180 meets the signal strength threshold. Conversely, if the signal strength of satellite device 170 and the signal strength of base station 180 do not both meet the signal strength threshold, controller 145 may determine that wireless communication component 160 is unable to establish a wireless connection (e.g., with satellite device 170 or base station 180). Controller 145 may determine that the sensor data (from stereo camera 155) is to be processed locally based on a determination that wireless communication component 160 is unable to establish a wireless connection.

[0058] Controller 145 may obtain (e.g., from a memory of machine 105) operational information indicating whether machine 105 is operating in a manual mode or an autonomous mode. If the operational information indicates that machine 105 is operating in an autonomous mode, controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection.

[0059] The controller 145 may obtain location data (e.g., from one or more sensor devices 165) that identifies a geographic location of the machine 105. If the controller 145 determines that the geographic location is an underground location, a building, a covered structure, among other locations, that may prevent the wireless communication component 160 from communicating with the satellite device 170 or the base station 180, the controller 145 may prevent the wireless communication component 160 from attempting to establish a wireless connection. Alternatively, if the controller 145 determines that the geographic location is not a location that may prevent the wireless communication component 160 from communicating with the satellite device 170 or the base station 180, the controller 145 may provide instructions to the wireless communication component 160 to establish a wireless connection. The controller 145 may perform similar actions with respect to information regarding expected weather conditions at a geographic location such that wireless communication cannot be established by the wireless communication component 160. As described above, the controller 145 may conserve computational resources that would have been used in a failed attempt to establish a wireless connection (e.g., due to the geographic location, expected weather conditions, among other examples) by preventing the wireless communication component 160 from attempting to establish a wireless connection.

[0060] Assume that controller 145 provides instructions to wireless communication component 160 to establish a wireless connection. Based on the instructions, controller 145 may establish a wireless connection (e.g., with satellite device 170 or base station 180) in a manner similar to that described above and provide an indication that the wireless connection has been established. Assume that controller 145 determines (e.g., based on an indication from wireless communication component 160) that a wireless connection is possible. Based on determining that a wireless connection is possible, controller 145 may cause wireless communication component 160 to transmit sensor data (from stereo camera 155) over the wireless connection to one or more computing devices 175 for processing.

[0061] The sensor data may include images acquired by the stereo camera 155 and / or a disparity map generated based on the images. In some cases, the machine 105 may include a LIDAR device. In such cases, the sensor data may include a 3D point cloud of the environment based on data acquired by the LIDAR device. The one or more computing devices 175 may receive the sensor data and process the sensor data to generate processed data.

[0062] The one or more computing devices 175, when processing the sensor data, may perform one or more operations (e.g., filtering operations) on the sensor data. The one or more computing devices 175 may, for example, among other examples, process the sensor data to remove or reduce an amount of noise (e.g., outlier data) from the sensor data and / or remove items that may be distorted in the sensor data or distort the rendering of the sensor data. The one or more computing devices 175 may, for example, among other examples, perform filtering on the sensor data (e.g., perform filtering on a disparity map) to remove or reduce the amount of noise (e.g., outlier data) and / or remove items.

[0063] With respect to performing filtering, the one or more computing devices 175 may process the sensor data using spatial noise reduction filtering, temporal / time filtering, and / or digital signal processing techniques, among other examples. As a result of processing the sensor data in this manner, the one or more computing devices 175 may increase the resolution of the sensor data (e.g., increase the measure of smoothness of the sensor data).

[0064] In addition to or as an alternative to removing or reducing noise and / or removing items, the one or more computing devices 175 may process the sensor data to remove image occlusions from the sensor data. The one or more computing devices 175 may, for example, use one or more computer vision techniques to identify and remove one or more items that may have blocked the line of sight of the stereo camera 155. The one or more computing devices 175 may, for example, analyze the sensor data using one or more object detection techniques (e.g., Single Shot Detector (SSD) techniques, and / or You Only Look Once (YOLO) techniques, etc.) to identify one or more items. In addition to or as an alternative to removing image occlusions, the one or more computing devices 175 may interpolate data in one or more portions of the sensor data. As an example, the one or more computing devices 175 may perform image processing on the sensor data (e.g., using one or more image processing techniques) to interpolate the sensor data.

[0065] By performing one or more operations above, the one or more computing devices 175 may increase a measure of quality of the sensor data and may increase a probability of identifying useful information that may facilitate operation of the machine 105. The one or more computing devices 175 may generate filtered data based on performing one or more operations above. The one or more computing devices 175 may process the filtered data (e.g., using semantic segmentation techniques) to determine information about one or more holes on the ground, information about one or more piles of material on the ground, information about the material, and / or information about the material within the machine work tool 140, among other examples.

[0066] The information regarding the one or more holes may include, among other examples, a location of the one or more holes, a depth of the one or more holes, and / or a geometry (or shape) of the one or more holes. The information regarding the one or more peaks may include, among other examples, a location of the one or more peaks, a height of the one or more peaks, a geometry (or shape) of the one or more peaks, and / or a volume of the one or more peaks. The information regarding the material may include, among other examples, a type of material, such as sand, clay, and / or hard rock. The information regarding the material within the machine work tool 140 may include, among other examples, a volume of the material, a geometry (or shape) of the material, and / or a fill level of the machine work tool 140.

[0067] Information regarding the one or more holes, the one or more mountains, and / or the material may be included in the surface data. The surface data may be used by the machine 105 to facilitate navigation of the machine 105 within the environment. The surface data may be used by the machine 105, for example, to identify one or more navigable portions of the environment and to identify one or more non-navigable portions of the environment. As an example, the one or more navigable portions may include one or more portions of the ground that do not include one or more holes and / or do not include one or more mountains, among other examples of items that may impede navigation operations of the machine 105. The one or more non-navigable portions may include one or more portions of the ground that include one or more holes and / or one or more mountains, among other examples of items that may impede navigation operations of the machine 105. In this regard, the surface data may be used for autonomous navigation operations of the machine 105.

[0068] The surface data may be used by the machine 105 to facilitate material movement operations (e.g., excavation operations) of the machine 105. The surface data may be used by the machine 105, for example, to identify one or more locations in the ground (other than the location of one or more holes and / or one or more mountains) for material movement operations. In this regard, the surface data may be used for autonomous material movement operations of the machine 105.

[0069] Information regarding the material and / or information regarding the material in the machine work tool 140 may be included in the volumetric data. The volumetric data may be used (e.g., by the machine 105 and / or a back office system, among other examples) to determine a measure of the productivity of an operator of the machine 105, among other examples. The volumetric data may include, for example, information identifying a volume of material moved by the machine 105 over a period of time and / or information indicative of a measure of completion of a material movement task, among other examples.

[0070] In some implementations, one or more computing devices 175 may process the filtered data using one or more machine learning models. The one or more computing devices 175 may, for example, provide the filtered data as input to one or more machine learning models, which may provide information about one or more holes, information about one or more peaks, information about the material, and / or information about the material in the machine working tool 140 as output. For example, a first machine learning model may provide information about one or more holes (as output), a second machine learning model may provide information about one or more peaks (as output), and a third machine learning model may provide information about the material (as output).

[0071] The one or more computing devices 175 may aggregate the various outputs into processed data, including source data and volumetric data. The wireless communication component 160 may receive the processed data from the one or more computing devices 175 via a wireless connection and provide the processed data to the controller 145 to facilitate machine operations (e.g., navigation operations and / or material movement operations, among other examples). In some cases, the processed data may be provided to a site manager's device and / or a back office system, among other examples.

[0072] The one or more machine learning models may be trained using past data (e.g., past images, past disparity maps, and / or past point clouds) that identify different holes (e.g., different sizes, shapes, and / or depths), different peaks (e.g., different sizes, shapes, and / or heights), different types of material, different volumes of material, different machine work tools, and / or different volumes of material within the machine work tools, among other examples. In some cases, PointNet may be an example of one or more machine learning models used with respect to point clouds. The one or more machine learning models may be trained in a manner similar to that described above.

[0073] The controller 145 may receive additional data for processing in conjunction with the sensor data from one or more sensor devices 165. The additional data may include implement pose data, machine pose data, linkage pose data, hydraulic displacement data, and / or pressure data (e.g., for hydraulic cylinders), among other examples. The controller 145 may cause the wireless communication component 160 to transmit the additional data via a wireless connection to one or more computing devices 175 for processing along with the sensor data to generate processed data.

[0074] For example, when determining the volume of material in the machine work tool 140, the one or more computing devices 175 may determine the coordinates of the machine work tool 140 in the disparity map based on the tool pose data. The coordinates of the machine work tool 140 may enable the one or more computing devices 175 (and / or one or more machine learning models) to identify a portion of the sensor data (e.g., a portion of the disparity map) that includes the machine work tool 140 when determining the volume of material in the machine work tool 140. By identifying the portion of the sensor data, the one or more computing devices 175 (and / or one or more machine learning models) may conserve computational resources that would have been used to process the entire sensor data for purposes of determining the volume of material in the machine work tool 140. In some examples, when determining the volume of material in the machine work tool 140, the one or more computing devices 175 may determine the geometry of the machine work tool 140. In some cases, the one or more computing devices 175 may receive information regarding the geometry of the machine work tool 140 from the machine 105 (e.g., from the controller 145 via the wireless communication component 160). Alternatively, the one or more computing devices 175 may receive information identifying the machine 105 (e.g., from the controller 145 via the wireless communication component 160) and may use the information identifying the machine 105 to obtain information regarding the geometry of the machine work tool 140 (e.g., from a data structure). The one or more computing devices 175 may determine the geometry of an inner surface of the machine work tool 140 based on the information regarding the geometry of the machine work tool 140. The one or more computing devices 175 may determine the portion of the inner surface that is in contact with the material within the machine work tool 140 based on analyzing the sensor data. The one or more computing devices 175 may, for example, determine how high up the machine work tool 140 is the material on all sides of the machine work tool 140. Additionally, the one or more computing devices 175 may determine the contour shape of the material above the top surface of the machine work tool 140.One or more computing devices 175 may combine all the surfaces and calculate one enclosed single volume.

[0075] In some implementations, the controller 145 may cause the wireless communication component 160 to transmit data (e.g., sensor data, additional data, among other examples) over a wireless connection based on determining whether the mechanical work tool 140 is embedded in the ground or has been removed from the ground. The controller 145 may determine whether the mechanical work tool 140 is embedded in the ground or has been removed from the ground based on, for example, the implement pose data and / or the linkage pose data. If the mechanical work tool 140 is removed from the ground, the controller 145 may cause the wireless communication component 160 to transmit the data.

[0076] By determining whether mechanical work tool 140 is embedded in the ground or removed from the ground, controller 145 may prevent wireless communication component 160 from transmitting data that would have caused one or more computing devices 175 (and / or one or more machine learning models) to attempt to identify mechanical work tool 140 if mechanical work tool 140 is embedded in the ground during an excavation operation. As such, controller 145 may conserve computational resources that would have been used to transmit data that may not be useful to one or more computing devices 175 and / or to attempt to identify mechanical work tool 140 if mechanical work tool 140 was embedded in the ground.

[0077] In some implementations, the controller 145 may cause the wireless communication component 160 to transmit the processing information over a wireless connection to one or more computing devices 175 to process the sensor data based on the processing information. The processing information may include filtering information that identifies how to filter the sensor data (from the stereo camera 155) to obtain filtered data.

[0078] The filtering information may include, for example, information identifying an order for performing one or more filtering operations described above, and / or information identifying different computing devices (of the one or more computing devices 175) for performing different filtering operations, among other examples. In addition to or as an alternative to including filtering information, the processing information may include information identifying one or more machine learning models described above, information indicating that a first machine learning model is used to determine information regarding one or more holes, information indicating that a second machine learning model is used to determine information regarding one or more mountains, etc.

[0079] In some implementations, after establishing the wireless connection, controller 145 (and / or wireless communication component 160) may determine the signal strength of satellite device 170 and the signal strength of base station 180. Based on determining the signal strength of satellite device 170 and the signal strength of base station 180, controller 145 may periodically cause wireless communication component 160 to establish a wireless connection with satellite device 170, establish a wireless connection with base station 180, terminate a wireless connection, or not attempt to establish a wireless connection. When controller 145 causes wireless communication component 160 to terminate a wireless connection or not attempt to establish a wireless connection, controller 145 may cause sensor data (from stereo camera 155) to be processed locally on machine 105, as described below.

[0080] In some implementations, after establishing a wireless connection, the controller 145 may determine whether the machine 105 is in an idle or active state, as described above, and / or may determine whether the machine work tool 140 is in a locked or unlocked position, and / or may determine whether the machine 105 is operating in a manual or autonomous mode, and / or may determine the geographic location of the machine 105. The controller 145 may perform such actions periodically and / or based on a trigger event, as described above.

[0081] If the controller 145 determines that the machine 105 is idle, the machine work tool 140 is in a locked position, the machine 105 is operating in a manual mode, and / or the geographic location prevents the wireless communication component 160 from communicating with the satellite device 170 or the base station 180, the controller 145 may cause the wireless communication component 160 to terminate the wireless connection or not attempt to establish a wireless connection. If the controller 145 causes the wireless communication component 160 to terminate the wireless connection or not attempt to establish a wireless connection, the controller 145 may cause the sensor data (from the stereo camera 155) to be processed locally on the machine 105, as described below.

[0082] In some implementations, when wireless connectivity is not possible, the controller 145 stores the sensor data received from the stereo camera 155. For example, assume that the controller 145 determines that wireless connectivity is not possible as described above, and the controller 145 receives sensor data after determining that wireless connectivity is not possible. Based on determining that wireless connectivity is not possible, the controller 145 may cause the sensor data to be stored (or buffered) locally for a threshold time. In some examples, the controller 145 may cause the sensor data to be buffered locally using a circular buffer (or ring buffer), a single buffer, a double buffer, among other examples.

[0083] After the threshold time has expired, controller 145 may determine whether a wireless connection is possible (as described above). Assume that controller 145 determines that a wireless connection is possible after the threshold time has elapsed. Based on determining that a wireless connection is possible after the threshold time has elapsed, controller 145 may cause the sensor data to be transmitted to one or more computing devices 175 (as described above). In some cases, if controller 145 determines that a wireless connection is not possible after the threshold time has elapsed, controller 145 may cause the sensor data to be processed locally, as described below.

[0084] In some implementations, if a wireless connection is not available, the controller 145 may cause the machine 105 to pause an operation being performed or initiated by the machine 105. For example, assume that the controller 145 determines (as described above) that a wireless connection is not available and that processed data is not being received from the computing device 175. The controller 145 may determine whether the machine 105 is performing or initiating an operation that uses the processed data.

[0085] For example, assume that the controller 145 determines (e.g., based on information from a memory of the machine 105) that the machine 105 is performing or beginning an autonomous operation (e.g., an autonomous navigation operation, an autonomous material movement operation, among other examples). Further assume that the processed data (generated by the one or more computing devices 175) includes surface data and / or volumetric data, and that the surface data may be used by the machine 105 for the autonomous navigation operation and the volumetric data may be used by the machine 105 for the autonomous material movement operation. The controller 145 may cause the machine 105 to pause operation based on determining that a wireless connection is not possible before the processed data is received and based on determining that the machine 105 is performing or beginning an operation that uses the processed data.

[0086] The operation may be paused until the wireless communication component 160 re-establishes the wireless connection and receives process data from the computing device 175 via the wireless connection. In some cases, the controller 145 may cause the machine 105 to pause the operation if the controller 145 determines that the wireless connection is not possible for a threshold time (e.g., after transmitting sensor data). In some cases, the controller 145 may provide a notification indicating that the wireless connection is not possible after the threshold time has elapsed. The notification may be provided via the operator cabin 120 (e.g., via an integral display) and / or provided to the operator's device and / or provided to the back office, among other examples. In some cases, the controller 145 may allow the machine 105 to continue performing the operation if the controller 145 determines that the operation is not an operation that uses the processed data.

[0087] In some implementations, the controller 145 may cause the sensor data (from the stereo camera 155) to be processed locally for one or more of the reasons described above. The controller 145 may process the sensor data to generate machine-processed data. The controller 145 may process the sensor data in a manner similar to that described above with respect to one or more computing devices 175 processing the sensor data. The controller 145 may, for example, perform one or more operations (described above) to remove noise from the sensor data, reduce noise in the sensor data, remove distorted items from the sensor data, and remove image occlusions from the sensor data, among other examples.

[0088] The controller 145 may generate the machine filtered data based on performing one or more operations. The one or more operations performed by the controller 145 may be less computationally complex than the one or more operations performed by the one or more computing devices 175. As a result, the machine filtered data may differ from the data filtered by the one or more computing devices 175 (e.g., the resolution of the machine filtered data may be less than the resolution of the filtered data from the computing devices 175).

[0089] The controller 145 may process the machine-filtered data using one or more machine learning models (similar to the one or more machine learning models described above) in a manner similar to that described above. As a result of processing the machine-filtered data, the controller 145 may generate machine-processed data including machine surface data (similar to the surface data described above) and / or machine volumetric data (similar to the volumetric data described above). In some cases, due to computational constraints of the machine 105, one or more machine learning models (used by the controller 145) may provide outputs that are less accurate than the outputs provided by one or more machine learning models used by the one or more computing devices 175. In this regard, the machine-processed data may be less accurate than the processed data provided by the one or more computing devices 175.

[0090] The controller 145 may determine a volume of the region of interest (e.g., the volume of a pile of material or the volume of material within the mechanical work tool 140, among other examples). In some circumstances, the controller 145 may determine the volume of material within the mechanical work tool 140 based on the geometry of an inner surface of the mechanical work tool 140 in a manner similar to that described above. In some circumstances, the controller 145 may analyze sensor data (e.g., images) to identify the region of interest. The controller 145 may analyze the images using, for example, one or more object detection techniques (e.g., Single Shot Detector (SSD) techniques and / or You Only Look Once (YOLO) techniques, etc.) to identify the region of interest.

[0091] The controller 145 may generate a graphical representation (e.g., a three-dimensional (3D) graphical representation or a 2.5 dimensional graphical representation) based on the image. The graphical representation may represent an area surrounding the machine 105 (or a portion of an area surrounding the machine 105), including the area of ​​interest. In this regard, the graphical representation may enable the controller 145 to determine characteristics of the area surrounding the machine 105, including characteristics of the area of ​​interest (e.g., the volume of the area of ​​interest). The graphical representation may be generated based on a disparity map of the image.

[0092] Controller 145 may determine a position and / or orientation of one or more parts of machine 105 based on the tool pose data and / or machine pose data. Controller 145 may determine coordinates of the region of interest relative to machine 105 based on the position and / or orientation of the one or more parts. Controller 145 may, for example, determine the coordinates of the region of interest relative to a particular part of machine 105. Controller 145 may, for example, consider a particular part of machine 105 to be a center point of a graphical representation (e.g., coordinates "0,0,0" of the 3D graphical representation), and the coordinates of the region of interest may correspond to the coordinates of the graphical representation relative to the center point.

[0093] Controller 145 may identify a portion of the 3D graphical representation (hereinafter referred to as the "3D portion") that corresponds to the region of interest based on the coordinates. Controller 145 may identify the 3D portion to conserve computational resources that would be used to process the entire 3D graphical representation to determine the volume of the region of interest. Controller 145 may determine the volume of the 3D portion using one or more computational models, one or more computational algorithms, and / or other machine algorithms that may be used to determine the volume of the 3D graphical representation.

[0094] In some examples, the controller 145 may determine (e.g., periodically and / or based on a trigger event) whether the wireless communication component 160 can establish a wireless connection in a manner similar to that described above. For example, after determining that a wireless connection is not possible, the controller 145 may determine (e.g., periodically and / or based on a trigger event) whether the wireless communication component 160 can re-establish a wireless connection. The controller 145 may cause the wireless communication component 160 to establish a wireless connection based on determining that the wireless communication component 160 can re-establish a wireless connection. The wireless communication component 160 may establish a wireless connection as described above.

[0095] In some circumstances, algorithms (used by one or more computing devices 175 to perform the actions described above) may be updated on one or more computing devices 175 with no or minimal downtime to the machine 105. Algorithms may be updated, for example, with new features, tools, and / or information to gather new information from existing sensors of the machine 105. In this regard, additional value (or functionality) may be added to the machine 105 without retrofitting the machine 105 with new hardware to change how it processes data or with no or minimal downtime to the machine 105.

[0096] With regard to algorithm updates, as new algorithms (e.g., for noise / artifact removal, feature extraction, machine learning model training, among other examples) are developed and validated on non-production machines, the new algorithms may be introduced to production machines in various manners. In one example, the new algorithms may be replaced (on one or more computing devices 175) with the same identifier(s) as the corresponding old algorithm(s) so that existing machines transmitting data (to one or more computing devices 175 for processing) can immediately recognize the improvements upon request. The process is a continuous integration style update that occurs only on the server side (e.g., only on the one or more computing devices 175 side) and affects all machines that use one or more computing devices 175 for remote processing.

[0097] As another example, new algorithms can be substituted (on one or more computing devices 175) under new identifier(s) and existing machines can have their on-board software flashed with updates that transmit requests to the new identifier(s). The process is a continuous integration style update that occurs only on the server side (e.g., only on one or more computing devices 175) and a software distribution update that occurs on the client side (e.g., on the machine 105). No hardware changes are required, but new software must be pushed to the machines. This process can be used to manage which machines are upgraded with minimal downtime.

[0098] As yet another example, new algorithms can be provided (on one or more computing devices 175) under new identifier(s) and existing machines can be configured to use the new identifier(s) with their existing on-board software, for example through a manual change made through a display. This process is a continuous integration style update done on the server side and a software configuration change done on the client side. No hardware changes are required to the machines 105. This process can be used to manage which machines are upgraded with little to no downtime.

[0099] The number and arrangement of devices shown in Figure 2 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged devices compared to those shown in Figure 2. Furthermore, two or more of the devices shown in Figure 2 may be implemented within a single device, or a single device shown in Figure 2 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of system 200 may perform one or more functions described as being performed by another set of devices of system 200.

[0100] 3 is a flow chart of an example process 300 for remote processing of sensor data. One or more process blocks of FIG. 3 may be performed by a controller (e.g., controller 145). One or more process blocks of FIG. 3 may be performed by another device or devices separate from or including the device, such as a stereo camera (e.g., stereo camera 155), a wireless communication component (e.g., wireless communication component 160), a satellite device (e.g., satellite device 170), one or more computing devices (e.g., one or more computing devices 175), and / or a base station (e.g., base station 180).

[0101] 3, process 300 may include receiving sensor data of the environment, including the ground on which the work machine is located, from one or more sensor devices on the work machine (block 310). For example, the controller may receive sensor data of the environment, including the ground on which the work machine is located, from one or more sensor devices on the work machine, as described above.

[0102] The one or more sensor devices are one or more first sensor devices, the sensor data is first sensor data, and the method further includes receiving second sensor data from one or more second sensor devices regarding at least one of a position or orientation of the one or more parts of the work machine.

[0103] 3, the process 300 may include determining whether a wireless connection is possible (Block 320). For example, the controller may determine whether a wireless connection is possible, as described above.

[0104] Process 300 includes determining whether the work machine is in an idle state or an active state or determining whether the implement of the work machine is in a locked position or an unlocked position, and determining whether a wireless connection is possible further includes determining whether a wireless connection is possible when the work machine is in an active state or when the implement is in an unlocked position.

[0105] As further shown in FIG. 3, process 300 may include selectively transmitting the sensor data to one or more computing devices for processing or processing the sensor data locally by the work machine based on whether a wireless connection is available (block 330).

[0106] The controller may, for example, selectively transmit the sensor data to one or more remote computing devices for processing or process the sensor data locally by the work machine based on whether a wireless connection is available; if a wireless connection is available, the sensor data is transmitted to one or more remote computing devices and has the one or more remote computing devices process the sensor data to provide first processed data, the first processed data including at least one of first surface data regarding one or more portions of the ground surface or first volumetric data regarding a volume of material moved or to be moved by the work machine; and if a wireless connection is not active, the sensor data is processed locally by the work machine to generate second processed data, as described above, the second processed data including at least one of second surface data regarding one or more portions of the ground surface or second volumetric data regarding a volume of material moved or to be moved by the work machine.

[0107] Transmitting the sensor data may include transmitting one or more of: one or more images of the environment, one or more disparity maps of the environment, and / or one or more point clouds of the environment.

[0108] Transmitting the sensor data includes transmitting the sensor data via a satellite device configured for high speed communication or a cellular network configured for high speed communication to one or more computing devices for processing.

[0109] Transmitting the sensor data includes transmitting the sensor data to have one or more remote computing devices perform filtering operations on the sensor data, the filtering operations including two or more of removing noise from the sensor data, removing image occlusions from the sensor data, or removing information regarding one or more obstacles in the environment from the sensor data.

[0110] Transmitting the sensor data includes transmitting the sensor data and processing information to have one or more remote computing devices process the sensor data based on the processing information to generate processed data, the processing information including at least one of information identifying a manner of filtering the sensor data to obtain filtered sensor data, or information identifying one or more machine learning models to process the sensor data or the filtered sensor data to generate the processed data.

[0111] 3, the process 300 may include providing the first processed data or the second processed data to facilitate operation of the work machine (block 340). The controller may provide the first processed data or the second processed data to facilitate operation of the work machine, for example, as described above.

[0112] The first volume data or the second volume data includes at least one of information identifying a first volume of material in a bucket of an implement of the work machine or information identifying a second volume of material located in one or more portions of the ground, and providing the first processed data or the second processed data includes providing at least one of information identifying the first volume or information identifying the second volume.

[0113] The first surface data or the second surface data includes at least one of information identifying one or more holes located in one or more portions of the ground or information identifying one or more piles of material located in one or more portions of the ground, and providing the first processed data or the second processed data includes providing at least one of information identifying the one or more holes or information identifying the one or more piles to facilitate at least one of movement of a work machine or operation of an implement of the work machine over the ground.

[0114] The one or more sensor devices are one or more first sensor devices, and the sensor data is first sensor data, and the method further includes: selectively transmitting or processing the sensor data includes selectively transmitting or processing the first sensor data and the second sensor data to obtain first processed data or second processed data, respectively.

[0115] Process 300 may include determining that the wireless communication component is unable to receive the processed data after transmitting the sensor data, and preventing the work machine from performing an action based on the processed data if the wireless communication component is unable to receive the processed data after transmitting the sensor data.

[0116] Process 300 may include causing the machine to autonomously navigate a terrain based on surface data regarding one or more portions of the terrain, or causing the machine to autonomously operate an implement to move material based on volumetric data regarding a volume of material moved or to be moved by the implement.

[0117] Although Figure 3 illustrates example blocks of process 300, process 300 may include additional blocks, fewer blocks, different blocks, or blocks in a different arrangement than the blocks depicted in Figure 3. Additionally or alternatively, two or more of the blocks of process 300 may be performed in parallel or combined. [Industrial Applicability]

[0118] The present disclosure relates to machines that selectively transmit sensor data (e.g., from a stereo camera) to one or more remote computing devices for processing or process the sensor data locally based on whether a wireless connection is available. The sensor data may be transmitted to one or more remote computing devices for processing to generate processed data, for example, if a wireless connection is available. Alternatively, if a wireless connection is not available, the sensor data may be processed locally on the machine.

[0119] Processing images from a stereo camera to generate useful information for a machine is a computationally complex and time-consuming task that can benefit from the use of high-performance computing resources. Processing images using a machine's typical computing resources presents several problems. For example, images processed by a machine's typical computing resources may generate information that is less accurate and / or of lower quality than information generated by a high-performance computer resource. Processing images using a work machine's typical computing resources consumes an excessive amount of computing resources. Such computing resources may be required for various operations of the work machine. Thus, processing images using the typical computing resources may adversely affect the work machine's capabilities for various operations. Moreover, work machines may be used in rocky and harsh conditions that may cause the typical computing resources to fail prematurely. In addition, the machine's typical computing resources may generate information in an inefficient manner.

[0120] The present disclosure solves the above problems. For example, by transmitting sensor data to one or more remote computing devices for processing, the processed data of the present disclosure has higher accuracy and quality than data generated using typical computational resources of a machine. In this regard, the processed data may improve the accuracy of operations performed by the machine, and / or improve the precision of operations performed by the machine, and / or improve the speed of operations performed by the machine, and / or improve the efficiency of operations performed by the machine, among other examples. In addition, by transmitting sensor data to one or more remote computing devices for processing, the processed data is generated in an efficient manner. Furthermore, by communicating with one or more computing devices using a satellite device configured for high-speed communication or a cellular network configured for high-speed communication, the machine may receive the processed data in a timely manner.

[0121] Moreover, the one or more computing devices are no longer subject to the rocky and harsh conditions commonly found in many job sites and therefore are no longer prone to premature failure. Additionally, by transmitting sensor data to one or more remote computing devices for processing, the machine conserves computational resources that would have been consumed by processing the images locally. Additionally, by transmitting sensor data to one or more remote computing devices for processing, the machine reduces power consumption, thus conserving excess amounts of power that would have been consumed by typical computational resources when processing the images. Additionally, the use of one or more remote computing devices for processing provides flexibility with respect to equipment upgrades for the machine.

[0122] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations may be made in light of the above disclosure or acquired from practice of the implementations. Furthermore, any implementations described herein may be combined unless a reason why one or more implementations cannot be combined is explicitly provided in the foregoing disclosure. Although particular combinations of features are recited in the claims and / or disclosed herein, these combinations are not intended to limit the disclosure of the various implementations. Each dependent claim listed below may be directly dependent on only one claim, but the disclosure of the various implementations includes each dependent claim in combination with all other claims in the claim group.

[0123] As used herein, "a," "an," and "set" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, "the" is intended to include one or more items referenced in conjunction with "the" and may be used interchangeably with "one or more." Additionally, the phrase "based on" is intended to mean "based at least in part on" unless otherwise specified. Additionally, as used herein, the term "or" is intended to be inclusive when used in a series and may be used interchangeably with "and / or" unless otherwise specified (e.g., when used in conjunction with "either" or "only one of"). Additionally, as used herein, the term "remove" encompasses the partial reduction of a property or characteristic as well as the elimination of that property or characteristic.

Claims

1. A method performed by a controller (145) of a work machine (105), comprising: receiving sensor data from one or more sensor devices (155) of the work machine (105) of an environment including a ground surface on which the work machine (105) is located; determining whether a wireless connection is available; selectively transmitting the sensor data to one or more remote computing devices (175) for processing or processing the sensor data locally by the work machine (105) based on determining whether the wireless connection is available; if the wireless connection is available, the sensor data is transmitted to the one or more remote computing devices (175) and the one or more remote computing devices (175) process the sensor data to provide first processed data, the first processed data including at least one of first surface data relating to one or more portions of the ground surface, or first volumetric data relating to a volume of material moved or to be moved by the work machine (105); if the wireless connection is not possible, the sensor data is processed locally by the work machine (105) to generate second processed data, the second processed data including at least one of second surface data relating to one or more portions of the ground surface, or second volumetric data relating to a volume of material moved or to be moved by the work machine (105); The first volume data or the second volume data is information identifying a first volume of material within an implement of the work machine (105); information identifying a second volume of material located in one or more portions of the ground surface; and providing the first processed data or the second processed data includes providing at least one of the information identifying the first volume or the information identifying the second volume. And, providing the first processed data or the second processed data to facilitate operation of the work machine (105); A method comprising:

2. the one or more sensor devices (155) (150) are one or more first sensor devices (155), the sensor data is first sensor data; The method comprises: receiving second sensor data from one or more second sensor devices (150) relating to at least one of a position or an orientation of one or more portions (130, 135, and 140) of the work machine; Selectively transmitting or processing the sensor data includes selectively transmitting or processing the first sensor data and the second sensor data to obtain the first processed data or the second processed data, respectively. The method of claim 1.

3. Transmitting the sensor data includes: transmitting the sensor data to the one or more remote computing devices (175) for processing; a satellite device (170) configured for high speed communications; or A cellular network (180) configured for high speed communication To send via The method of claim 1 or 2, comprising:

4. The second volume data is information identifying a third volume of material loaded in front of the device; The first surface data or the second surface data is information identifying one or more holes located in one or more portions of said ground; or information identifying one or more piles of material located on one or more portions of the ground; and Providing the first processed data or the second processed data includes: providing at least one of information identifying the one or more holes or information identifying the one or more peaks; movement of the work machine over the ground; or Operation of the implement of the work machine To facilitate at least one of the following: The method of claim 2 , comprising:

5. determining whether the work machine (105) has been idle or active for a threshold time; or Determining whether an implement (130, 135, 140) of the work machine (105) is in a locked or unlocked position. Further comprising: Determining whether the wireless connection is possible includes: Whether the wireless connection is possible, when the work machine (105) is in the active state; or When the instrument (130, 135, 140) is in the unlocked position If at least one of the following is true, The method of claim 1 , comprising:

6. 1. A system comprising: one or more sensor devices (155) configured to acquire sensor data of an environment including a ground surface on which the work machine (105) is located; A controller (145); Equipped with The controller (145) determining whether a wireless connection is available; transmitting, if the wireless connection is available, the sensor data to one or more remote computing devices (175) for processing using a wireless communication component; the sensor data being transmitted to the one or more remote computing devices (175) to cause the one or more remote computing devices (175) to filter the sensor data and generate processed data; And, receiving the processed data from the one or more remote computing devices (175); the processed data includes at least one of surface data relating to one or more portions of the ground surface, or volumetric data relating to a volume of material moved or to be moved by the work machine (105); The surface data is information identifying one or more holes located in one or more portions of said ground; or information identifying one or more piles of material located on one or more portions of the ground; and providing the processed data includes providing at least one of information identifying the one or more holes or information identifying the one or more peaks to facilitate operation of the work machine (105). And, A system configured to:

7. the controller (145) is configured to provide the volumetric data to facilitate material movement operations to provide the processed data; The volume data is information identifying a first volume of material within an implement of the work machine (105); information identifying a second volume of material located in one or more portions of the ground surface; or Information identifying the volume of material moved by said work machine (105) over a period of time. at least one of The system of claim 6.

8. The controller (145) may include, in order to transmit the sensor data, configured to transmit the sensor data and processing information to the one or more remote computing devices (175) and cause the one or more remote computing devices (175) to process the sensor data based on the processing information to generate the processed data; The processing information is information identifying a manner in which the sensor data is filtered to obtain filtered sensor data; or Information identifying one or more machine learning models that process the sensor data or the filtered sensor data to generate the processed data. at least one of 8. A system according to claim 6 or 7.

9. The controller (145) determining that the wireless connection is not possible; and based on determining that the wireless connection is not possible, causing the sensor data to be stored for a threshold time; determining whether the wireless connection is possible after the threshold time has elapsed; and [0023] The controller (145) may further comprise a controller for transmitting the sensor data to the one or more computing devices (175). and based on determining that the wireless connection is available after the threshold time has elapsed, causing the sensor data to be transmitted to the one or more remote computing devices (175). The system of claim 6.

10. The controller (145) determining that the wireless communication component is unable to receive the processed data after transmitting the sensor data; preventing the work machine (105) from performing an action based on the processed data if the wireless communication component is unable to receive the processed data after transmitting the sensor data; and configured to: The system of claim 6.

Images