Selective remote processing of data for autonomous drilling operations

Abstract

Selectively perform remote processing of data for autonomous drilling operations. The machine may include a work implement, one or more sensor devices, and a controller. The controller may be configured to receive data from the one or more sensor devices regarding a surface on which the machine performs excavation operations, transmit the data to one or more remote computing devices, cause the one or more remote computing devices to generate excavation information based on the data, and receive the excavation information from the one or more remote computing devices. The excavation information may include information identifying a set of excavation locations within the surface region and information identifying corresponding dump sites. The controller may be configured to navigate the machine to one of the excavation locations and cause the work implement to begin excavation operations at the excavation location based on the excavation information.

Description

[Technical Field]

[0001] The present disclosure relates generally to autonomously performing excavation operations, for example, determining whether remote processing of data (from multiple sensor devices) is performed to autonomously perform excavation operations. [Background technology]

[0002] An excavator may perform excavation operations at a site. A typical computing device in the excavator may enable adjustments to the excavator's work equipment during autonomous operations (e.g., during the autonomous execution of an excavation operation). However, typical computing devices are subject to computational constraints that prevent the typical computing device from performing computationally complex and time-consuming tasks. For example, the computational constraints prevent the typical computing device from processing sensor data from multiple sensor devices (e.g., in the excavator) to determine an optimal sequence of excavation and dump locations for an excavation operation (e.g., during the autonomous execution of an excavation operation). Furthermore, excavators may be used in rough terrain and under harsh conditions, which may cause typical computing devices to fail prematurely.

[0003] U.S. Patent No. 10,066,367 (Patent '367) discloses a system configured to be mounted on a vehicle for adjusting the position of a work implement while the vehicle is performing autonomous operations. Patent '367 discloses that the vehicle can monitor the height, tilt angle, and / or load of the work implement while operating and adjust one or more parameters related to the work implement to achieve a desired finish profile. While Patent '367 discloses adjusting the position of the work implement during autonomous operations, Patent '367 does not address computational constraints (e.g., associated with autonomous operations), does not address the sequence for determining excavation and dump locations for excavation operations, and does not address other limitations of typical computing devices discussed above.

[0004] The present disclosure solves one or more of the problems set forth above and / or other problems in the art. Summary of the Invention

[0005] a first data entry device for entry of a first excavation operation into the one or more remote computing devices; a second data entry device for entry of a first excavation operation into the one or more remote computing devices; a second data entry device for entry of a first excavation operation into the one or more remote computing devices; a second data entry device for entry of a first excavation operation into the one or more remote computing devices;

[0006] 1. A system including a controller configured to receive data from one or more sensor devices regarding an earth surface on which a machine is to perform an excavation operation, transmit the data to one or more remote computing devices, cause the one or more remote computing devices to generate excavation information for the excavation operation based on the data, receive the excavation information from the one or more remote computing devices based on the excavation information including information identifying a set of excavation locations within an earth surface area and information identifying an orientation of the machine at each excavation location of a subset of the excavation locations, navigate the machine to excavation locations within the subset of excavation locations based on the excavation information, turn responsive to the orientation at the excavation locations, and cause a work implement of the machine to perform the excavation operation at the excavation locations.

[0007] A machine including a work implement, one or more sensor devices, and a controller, the controller being configured to receive data from the one or more sensor devices regarding an earth surface on which the machine is to perform excavation operations, transmit the data to one or more remote computing devices, cause the one or more remote computing devices to generate excavation information based on the data, receive excavation information from the one or more remote computing devices including information identifying a set of excavation locations within an earth surface area and information identifying corresponding dump locations, and, based on the excavation information, navigate the machine to excavation locations and cause the work implement to begin excavation operations at the excavation locations. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an embodiment described herein. [Figure 2] FIG. 1 is a schematic diagram of an example of a system described herein. [Figure 3] 1 is a flowchart of an exemplary process associated with remote processing of data to generate drilling information. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present disclosure relates to a machine that causes one or more remote computing devices to generate excavation information based on data regarding the earth surface on which the machine performs excavation operations. The data may be obtained from one or more sensor devices located on different parts of the machine (e.g., located on the boom, stick, and / or implement). As an example, the one or more sensor devices may include, for example, one or more stereo ("stereo") cameras, and the data may include one or more images of the earth surface. The data may be transmitted to the one or more remote computing devices via a satellite device configured for high-speed communication or via a cellular network configured for high-speed communication (e.g., a 5G network). The one or more remote computing devices may include one or more cloud computing devices (e.g., one or more high-performance remote computing devices).

[0010] The one or more remote computing devices may generate excavation information based on the data. The excavation information may include, for example, information identifying a series of excavation locations (and paths between successive excavation locations), information identifying corresponding dump locations, recommended machine attitude data identifying a machine orientation for each excavation location, recommended implement attitude data identifying a position and / or orientation of the machine implement for each excavation location, recommended link attitude data identifying a position and / or orientation of the machine boom and / or stick for each excavation location, information identifying an amount of material to be excavated (or removed) for each excavation location, etc.

[0011] Once the excavation information is generated, the one or more remote computing devices may determine (or identify) multiple sets of excavation locations (and paths between the excavation locations) and identify an optimal set of excavation locations by considering different factors. The various factors may include minimum excavation cycle time, maximum fuel consumption, minimum work implement wear, etc. An excavation cycle may include a combination of machine actions, such as the machine digging material, swinging, and dumping material. The various factors are configurable by a user (e.g., a machine operator). For example, among other examples, the different factors may be configured based on the type of machine, the amount of machine usage, the geographic area in which the machine is located, weather conditions in the geographic area, and the time and date the excavation operation is being performed. The optimal set of excavation locations may correspond to the set of excavation locations included in the excavation information. The excavation information may be transmitted to the machine (e.g., via a satellite device or cellular network) to facilitate the excavation operation. It should also be understood that the terms "excavation" and "excavation operation" are used broadly and, as discussed above, include dumping material previously collected by the work machine.

[0012] If wireless communication is interrupted or if a wireless connection is established and the data transfer rate of the wireless connection does not meet the transfer rate threshold, the machine may suspend data transfer and buffer the data. When wireless communication is re-established or the data transmission rate of the wireless connection meets the transmission rate threshold, the machine may resume transmitting data. If wireless communication is interrupted or the data transmission rate does not meet the transmission rate threshold, the machine may suspend operations that depend on the processed data (e.g., suspending drilling operations, suspending navigating operations, etc.). In some examples, if data cannot be transmitted to one or more remote computing devices, the data may be processed locally on the machine. Additionally, the machine may provide a notification (e.g., in the machine's cab) indicating that the data will not be transmitted to one or more remote computing devices and / or that the data is being processed locally on the machine.

[0013] The term "machine" refers to a machine that performs a task related to an industry, such as, for example, mining, construction, agriculture, transportation, or another industry. Additionally, one or more work implements may be connected to the machine. By way of example, the machine may include a construction vehicle, a work vehicle, or a similar vehicle related to the above industries.

[0014] 1 is a schematic diagram of an embodiment 100 described herein. As shown in FIG. 1, the exemplary embodiment 100 includes a machine 105. The machine 105 is embodied as an earthmoving machine, such as an excavator. Alternatively, the machine 105 may be another type of machine, such as a bulldozer.

[0015] As shown in FIG. 1 , machine 105 includes ground engaging members 110, machine body 115, cab 120, and rolling elements 125. Ground engaging members 110 may include tracks (shown in FIG. 1 ), wheels, rollers, and / or the like for propelling machine 105. Ground engaging members 110 are mounted to a subframe that carries the rolling elements (not shown) and are driven by one or more engines and transmission systems (not shown). Machine body 115 is mounted to a rotatable frame that is coupled to the rolling elements supported on the subframe, such that cab 120 and machine body 115 are rotatable relative to the subframe and ground engaging members 110. Cab 120 is supported by machine body 115, which is mounted to the rotatable frame. Operator controls 124, such as an integrated display (not shown) and an integrated joystick, may include one or more input components.

[0016] In the case of an autonomous machine, operator controls 124 may not be designed for use by an operator located in cab 120, but may instead be designed to operate independently of an operator in work machine 105. In this case, for example, operator controls 124 may include one or more input components that provide an input signal for use by another component without operator input. Rotational elements 125 may include one or more components that enable rotation (or rotation) of the rotating frame (and machine body 115). For example, rotational elements 125 may enable rotation (or rotation) of the rotating frame (and machine body 115) relative to ground engaging members 110.

[0017] As shown in FIG. 1 , machine 105 includes boom 130, stick 135, and mechanical work tool 140. Boom 130 is pivotally attached at its proximal end to machine body 115 and hinged to machine body 115 by one or more fluid-actuated cylinders (e.g., hydraulic or pneumatic cylinders), electric motors, and / or other electromechanical components. Stick 135 is pivotally attached to a distal end of boom 130 and articulatably coupled to boom 130 by one or more fluid-actuated cylinders, motors, and / or other electromechanical components. Boom 130 and / or stick 135 may be referred to as a link. Mechanical work tool 140 is attached to the distal end of stick 135 and is articulatable relative to stick 135 by one or more fluid-actuated cylinders, motors, and / or other electromechanical components. Mechanical work tool 140 may be a bucket (shown in FIG. 1 ) or another type of tool or implement that can be attached to stick 135. Mechanical work tool 140 may also be referred to as a work implement.

[0018] 1 , machine 105 includes a controller 145 (e.g., an electronic control module (ECM), a computer vision controller, an autonomous controller, etc.), one or more inertial measurement units (IMUs) 150 (individually referred to herein as “IMU 150” and collectively referred to herein as “IMU 150”), one or more stereo cameras 155, a wireless communication component 160, and one or more sensor devices 165. Controller 145 may control and / or monitor operation of machine 105. For example, controller 145 may control and / or monitor operation of machine 105 based on signals from operator control device 124, signals from IMU 150, signals from stereo cameras 155, signals from wireless communication component 160, and / or signals from one or more sensor devices 165.

[0019] As shown in FIG. 1 , IMUs 150 are mounted at different locations on components or portions of machine 105, such as machine body 115, boom 130, stick 135, and machine work tool 140. IMU 150 includes one or more devices that may receive, generate, store, process, and / or provide signals indicative of the position and orientation of the component of machine 105 to which IMU 150 is mounted. For example, IMU 150 may include one or more accelerometers and / or one or more gyros. The one or more accelerometers and / or one or more gyros generate and provide signals that can be used to determine the position and orientation of IMU 150 relative to a reference frame, and accordingly, the position and orientation of the assembly. While the examples discussed herein relate to IMU 150, the present disclosure is well suited to using one or more other types of sensor devices that may be used to determine the position and orientation of a component of machine 105.

[0020] One or more first IMUs 150 may be disposed on the boom 130 and / or stick 135. The one or more first IMUs 150 may generate signals that may be used to determine the position and / or orientation of the boom 130 and / or stick 135, which may be used to generate link attitude data indicative of the position and / or orientation of the boom 130 and / or stick 135. One or more second IMUs 150 may be disposed on the machine work tool 140. The one or more second IMUs 150 may generate signals that may be used to determine the position and / or orientation of the machine work tool 140, which may be used to generate work implement attitude data indicative of the position and / or orientation of the machine work tool 140. One or more third IMUs 150 may be disposed on the machine body 115. The one or more third IMUs 150 may generate signals that can be used to determine the position and / or orientation of the mechanical body 115, which may be used to generate machine attitude data indicative of the position and / or orientation of the mechanical body 115.

[0021] IMU 150 may provide link attitude data, implement attitude data, and / or machine attitude data to controller 145 periodically (e.g., every 30 seconds, every minute, every two minutes, etc.). Additionally or alternatively, IMU 150 may provide link attitude data, implement attitude data, and / or machine attitude data to controller 145 based on a trigger event. A trigger event, as used herein, may include examples such as a request from controller 145, detection of movement of machine body 115, detection of movement of machine work tool 140, detection of movement of boom 130 and / or stick 135, a request from an operator, a request from a background system, machine 105 starting a new task, after machine 105 completes a digging cycle, after machine work tool 140 completes digging, and / or machine 105 being relocated to a different geographic location during a swing before dumping material, etc.

[0022] Stereo camera 155 may include one or more devices that may acquire and provide environmental data, including the earth surface on which machine 105 performs excavation operations. This data may include image data (e.g., three-dimensional (3D) image data) of the environment. Stereo camera 155 may provide data to controller 145 periodically (e.g., every second, every other second, and other examples) and / or based on a trigger event. While the examples described herein relate to stereo camera 155, the present disclosure is well suited to use with one or more other types of devices, such as non-stereo cameras, light detection and ranging (LIDAR) devices, and / or radio detection and ranging (RADAR) devices, as well as other examples of devices (e.g., sensor devices) configured to provide data about the environment. Any reference below to stereo camera 155 should be understood to include these other devices unless specifically stated otherwise.

[0023] 1 shows stereo cameras 155 provided on boom 130, stick 135, and machine work tool 140. In practice, machine 105 may include one or more additional stereo cameras 155, fewer stereo cameras 155, and / or differently positioned stereo cameras 155 (e.g., stereo cameras 155 positioned on other parts of machine 105).

[0024] Wireless communication component 160 includes one or more devices that enable machine 105 to communicate with other devices, as described herein. Wireless communication component 160 may include examples such as a transceiver, a separate transmitter and receiver, an antenna, a Bluetooth transceiver, or another type of wireless local area network transceiver. Wireless communication component 160 may transmit data (e.g., from one or more stereo cameras 155) to one or more remote computing devices and receive drilling information from one or more remote computing devices, as described herein. Controller 145 can use the drilling information to cause machine 105 to perform drilling operations at the earth's surface.

[0025] Wireless communication component 160 may be configured to transmit data in the form of a continuous data stream from one or more stereo cameras 155, or alternatively, transmit data in the form of transmission bursts. In some examples, wireless communication component 160 may transmit data and receive drilling information via a satellite device, via a base station, and / or via another device that enables wireless communication component 160 to transmit data to and receive data from one or more remote computing devices, as described below.

[0026] The wireless communication component 160 may communicate with one or more remote computing devices over a network including one or more wired and / or wireless networks, such as a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, a code division multiple access (CDMA) network, a 3G network, a 4G network, or other type of cellular network), the Internet, a cloud computing network, a wireless local area network (LAN), an intranet, an optical fiber-based network, a wide area network (WAN), a public surface mobile network (PLMN), a metropolitan network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, and / or a combination of these or other types of networks.

[0027] Wireless communication component 160 may enable machine 105 to communicate with one or more other machines independent of machine 105 (e.g., via Bluetooth or another type of wireless local area network). For example, wireless communication component 160 may receive external machine data from one or more sensor devices of one or more other machines, as described herein. The external machine data may include data regarding the Earth's surface. The external machine data may identify one or more views of the Earth's surface that are different from one or more views identified by data from one or more stereo cameras 155.

[0028] Additionally or alternatively, to receive the external data, wireless communication component 160 may transmit instructions (e.g., from controller 145) to one or more other machines to cause the one or more other machines to transmit the external machine data to one or more remote computing devices, as described herein. In some cases, the data (e.g., from one or more stereo cameras 155) and the external machine data may form a comprehensive view of the earth's surface. As described herein, the one or more remote computing devices may process the data (e.g., from one or more stereo cameras 155) and the external machine data to generate drilling information.

[0029] Sensor devices 165 include one or more devices that may receive, generate, store, process, and / or provide signals related to one or more components of machine 105. For example, sensor devices 165 may include examples such as, for example, hydraulic displacement sensors, pressure sensors (e.g., for hydraulic cylinders), soil moisture sensors, machine speed sensors, engine speed sensors, and / or global positioning system (GPS) devices.

[0030] 1 , exemplary embodiment 100 is associated with satellite device 170, includes one or more remote computing devices 175-1 through 175-N (N≧1) (hereinafter collectively referred to as remote computing devices 175 and individually as remote computing device 175), and is also associated with base station 180. Satellite device 170 may include one or more devices configured for high-speed communications (e.g., geostationary satellites configured for high-speed communications). For example, satellite device 170 may be configured to provide high-speed Internet access (e.g., to machine 105 via wireless communications component 160).

[0031] Remote computing device 175 may include one or more devices configured to generate drilling information based on data and / or external machine data, as described herein. Remote computing device 175 may be implemented as a single computing device (e.g., a single server) or multiple computing devices (e.g., multiple servers), e.g., multiple computing devices in one or more data centers. In some examples, remote computing device 175 may be included in a cloud computing environment. Additionally or alternatively, remote computing device 175 may be included in a background system.

[0032] The base station 180 may connect to a cellular network (e.g., as described above). For example, the base station 180 may connect to a 5G network. The base station 180 may include a base transceiver station, a radio base station, a Node B, an Evolution Node B (eNB), a Next Generation Node B (gNB), a base station subsystem, a cellular site, a cellular tower (e.g., a cellular tower, a mobile phone tower, etc.), an access point, a transmit / receive point (TRP), a radio access node, a macrocell base station, a microcell base station, a skin cell base station, and / or a femtocell base station, or a similar type of device.

[0033] As noted above, Figure 1 is provided as an example. Other examples may differ from the example described with reference to Figure 1.

[0034] FIG. 2 is a schematic diagram of an example system 200 described herein. As shown in FIG. 2, system 200 includes controller 145, IMU 150, stereo camera 155, wireless communication component 160, one or more sensor devices 165, and one or more remote computing devices 175. System 200 may be associated with a machine 210, regardless of ownership, including satellite device 170, base station 180, and sensor device 220. Machine 210 may be similar to machine 105 or different from machine 105 (e.g., if machine 105 is an excavator, machine 205 may be a different excavator or a different type of machine, such as an automated dump truck). Sensor device 220 may include, for example, a stereo camera, a LIDAR device, a radio detection and ranging (radar) device, etc.

[0035] Wireless communication component 160 may be configured to establish a wireless connection with one or more remote computing devices 175 via satellite device 170 or base station 180 (e.g., establish a wireless connection with satellite device 170 or base station 180 and then establish a wireless connection with one or more remote computing devices). Wireless communication component 160 may be configured to attempt to establish a wireless connection periodically (e.g., every 30 seconds, every minute, etc., for example) and / or based on a trigger event (e.g., based on a request from controller 145, based on receiving an indication that machine 105 is operating in an autonomous mode, based on wireless communication component 160 being activated, etc., for example).

[0036] The wireless communication component 160 may transmit the data (captured by the one or more stereo cameras 155) over a wireless connection to one or more remote computing devices 175 for processing. The one or more remote computing devices 175 may process the data to generate processed data, and the wireless communication component 160 may receive the processed data from the one or more remote computing devices 175 over a wireless connection.

[0037] Wireless communication component 160 may be configured to prioritize establishing a wireless connection with satellite device 170 (e.g., prioritize establishing a wireless connection with base station 180). For example, wireless communication component 160 may be configured to establish a wireless connection with satellite device 170 if the signal strength of base station 180 does not exceed a signal strength threshold difference for satellite device 170. As another example, wireless communication component 160 may be configured to establish a wireless connection with satellite device 170 if the signal strength of satellite device 170 meets a criterion, regardless of the signal strength of base station 180.

[0038] Except for other examples, wireless communication component 160 may be pre-configured with information identifying the threshold difference, may receive information identifying the threshold difference from a device of an operator of machine 105, or may receive information identifying the threshold difference from a back-end system. Alternatively, to preferentially establish a wireless connection with satellite device 170, wireless communication component 160 may be configured to preferentially establish a wireless connection with base station 180 (e.g., over establishing a wireless connection with satellite device 170) in a manner similar to that described above with respect to satellite device 170.

[0039] In some embodiments, wireless communication component 160 may be configured to establish a wireless connection based on the relative signal strength of satellite device 170 and the signal strength of base station 180. For example, wireless communication component 160 may be configured to establish a wireless connection with satellite device 170 when the signal strength of satellite device 170 exceeds the signal strength of base station 180. Alternatively, wireless communication component 160 may be configured to establish a wireless connection with base station 180 if the signal strength of base station 180 exceeds the signal strength of satellite device 170.

[0040] Wireless communication component 160 (and / or controller 145) may periodically (e.g., every 15 seconds, every 30 seconds, and other examples) determine the signal strength of satellite device 170 and the signal strength of base station 180. In this regard, wireless communication component 160 may periodically alternate between establishing a wireless connection with satellite device 170 or establishing a wireless connection with base station 180 based on the periodic determination of the signal strength of satellite device 170 and the signal strength of base station 180.

[0041] As an example of periodically alternatingly establishing wireless connections, assume that wireless communication component 160 has established a wireless connection with satellite device 170. If wireless communication component 160 (and / or controller 145) determines that the signal strength of base station 180 exceeds the signal strength of satellite device 170 (or vice versa), wireless communication component 160 may terminate the wireless connection with satellite device 170 and establish a wireless connection with base station 180.

[0042] Additionally or alternatively, to periodically determine the signal strength, wireless communication component 160 (and / or controller 145) may determine the signal strength of satellite device 170 and the signal strength of base station 180 based on a trigger event (e.g., based on a request from controller 145, based on an indication of a change in operation of machine 105, etc.). Wireless communication component 160 may alternate between establishing a wireless connection with satellite device 170 or establishing a wireless connection with base station 180 based on the trigger event.

[0043] By alternating (e.g., periodically and / or based on a trigger event) between establishing a wireless connection with satellite device 170 and establishing a wireless connection with base station 180, wireless communication component 160 may maintain communication with one or more remote computing devices 175 and attempt to minimize interruptions in the transmission of data and / or reception of drilling information. By preventing interruptions in the transmission of data and / or reception of drilling information, wireless communication component 160 and / or controller 145 may prevent interruptions in the operations performed by machine 105.

[0044] In some examples, if the signal strength of satellite device 170 does not meet the signal strength threshold, wireless communication component 160 may determine that a wireless connection with satellite device 170 may not be established. Similarly, if the signal strength of base station 180 does not meet the signal strength threshold, wireless communication component 160 may determine that a wireless connection with base station 180 may not be established. If both the signal strength of satellite device 170 and the signal strength of base station 180 do not meet the signal strength threshold, controller 145 may determine that the data is to be processed locally on machine 105. Wireless communication component 160 may be pre-configured with information identifying the signal strength threshold, may receive information identifying the signal strength threshold from an operator's equipment, may receive information identifying the signal strength threshold from a back-end system, etc.

[0045] Wireless communication component 160 may be configured to establish a wireless connection based on the output of the machine learning model. For example, wireless communication component 160 (and / or controller 145) may use the machine learning model to predict whether wireless communication component 160 will establish a wireless connection with satellite device 170, whether wireless communication component 160 will establish a wireless connection with base station 180, or whether wireless communication component 160 will establish a wireless connection with satellite device 170 or base station 180.

[0046] For example, controller 145 may provide as input to the machine learning model information identifying the current geographic location and / or current geographic region of machine 105, information identifying one or more obstacles within the current geographic location and / or current geographic region (e.g., one or more images of the one or more obstacles), information identifying the time of day, and / or information identifying predicted weather conditions, etc. As another example, the input may be obtained from an operator's device, one or more stereo cameras 155, and / or one or more sensor devices 165. Based on the input, the machine learning model may provide as output predicted connection information indicating whether wireless communication component 160 will establish a wireless connection with satellite device 170, establish a wireless connection with base station 180, or not establish a wireless connection with satellite device 170 or base station 180.

[0047] The machine learning model may be trained using historical data including historical data for the geographic location and / or geographic region, historical data (e.g., images) of obstacles for the geographic location and / or geographic region, historical time of day, historical data of weather conditions for the geographic location and / or geographic region, historical signal strengths of satellite devices 170 and base stations 180 for the geographic location and / or geographic region (e.g., due to historical obstacles, historical time, historical weather conditions, etc.), historical delays of satellite devices 170 and base stations 180 for the geographic location and / or geographic region, and / or historical data indicating established wireless connections (e.g., satellite devices 170 or base stations 180), etc. The machine learning models may be generated and trained by a trainer device, which can be a discrete hardware or software component (not shown). In other examples, the trainer device may be included in a background system or may be included in one or more remote computing devices 175. The trainer device may provide the machine learning models to the machine 105 for use by the controller 145 and / or the wireless communication component 160. The trainer device may update the machine learning models and provide them to the machine 105 (e.g., on a scheduled, on-demand, triggered, and / or periodic basis). In some cases, the controller 145 may obtain additional training data (e.g., additional historical data similar to the historical data described above) and retrain the machine learning models based on the additional training data. For example, the controller 145 may provide the additional training data to the trainer device to retrain the machine learning models. The machine learning models may be retrained periodically and / or based on a trigger event. In some cases, the machine learning models may be implemented on one or more remote computing devices 175.

[0048] When training a machine learning model, the trainer device may divide the training data into a training set (e.g., a dataset for training the machine learning model), a validation set (e.g., a dataset for evaluating the fit of the machine learning model and / or a dataset for fine-tuning the machine learning model), a test set (e.g., a dataset for evaluating the final fit of the machine learning model), and / or the like. The trainer device may perform preprocessing and / or dimensionality reduction to reduce the training data to a minimal feature set. The trainer device may train the machine learning model on this minimal specialized set and apply classification techniques to the minimal specialized set to reduce processing for training the machine learning model.

[0049] The trainer device may use classification techniques, such as logistic regression classification techniques, random forest classification techniques, gradient-based machine learning (GBM) techniques, and / or similar techniques, to determine classification results (e.g., whether a wireless connection with satellite device 170 or base station 180 is established). In addition to or instead of using classification techniques, the trainer device may use naive Bayes classifier techniques. In this case, the device may perform binary recursive partitioning to partition the training data of a minimum feature set into regions and / or branches, and use this partitioning and / or branching to perform predictions (e.g., whether a wireless connection with satellite device 170 or base station 180 is established). Based on the use of recursive partitioning, the trainer device may reduce the use of computing resources associated with manual, linear ordering and analysis of data items, thereby allowing the trainer device to train a model using thousands, millions, or billions of data items, which can result in a more accurate model than using fewer data items.

[0050] The trainer device may train the machine learning model using a supervised training process that includes receiving input to the machine learning model from subject matter experts (e.g., machine 105 and / or one or more operators associated with one or more machines), which may reduce the amount of time, processing resources, and / or the like to train the machine learning model compared to an unsupervised training process. The trainer device may also use one or more other model training techniques, such as neural network techniques, latent semantic indexing techniques, and / or similar techniques.

[0051] For example, the trainer device may implement artificial neural network processing techniques (e.g., using a two-layer feedforward neural network architecture, a three-layer feedforward neural network architecture, etc.) to perform mode discrimination regarding the mode of establishing a wireless connection with satellite device 170 or base station 180. In this case, the use of artificial neural network processing techniques may improve the accuracy of the machine learning models generated by the trainer device by being more robust to noisy, inaccurate, or incomplete data and by allowing the trainer device to detect patterns and / or trends that a human analyst or system cannot detect using less complex techniques.

[0052] The following examples assume that machine 105 is on the surface of a worksite and that machine 105 is performing excavation operations (e.g., autonomously or semi-autonomously) in an area of ​​the surface. It is further assumed that one or more stereo cameras 155 have acquired data about the surface and provided that data to controller 145 in a manner similar to that described above. Controller 145 may receive data from one or more stereo cameras 155 and determine whether to send the data to one or more remote computing devices 175 for processing to generate excavation information, or to process the data locally on machine 105.

[0053] As part of determining whether a data transmission is sent to one or more remote computing devices 175 for processing or processed locally on machine 105, controller 145 may determine whether a wireless connection is possible. A wireless connection is possible if wireless communication component 160 may establish a wireless connection with satellite device 170 or base station 180, or if wireless communication component 160 has already established a wireless connection and the data transmission rate (of the wireless connection) meets a transmission rate threshold. The transmission rate threshold may be determined by an operator, by a background system, stored in memory of machine 105, etc. If a wireless connection is possible, controller 145 may cause wireless communication component 160 to transmit the data to one or more remote computing devices 175 via the wireless connection. Alternatively, if a wireless connection is not possible, controller 145 may cause the data to be processed locally on machine 105. The wireless connection may not be possible if the wireless communication component 160 is unable to establish a wireless connection with the satellite device 170 or base station 180, or if the wireless communication component 160 has already established a wireless connection and the data transmission rate does not meet the transmission rate threshold.

[0054] In some examples, controller 145 may receive wireless connection information from wireless communication component 160 indicating whether a wireless connection is available, and controller 145 may determine whether a wireless connection is available based on the wireless connection information. In some examples, controller 145 may determine whether wireless communication component 160 needs to establish a wireless connection (e.g., with satellite device 170 or base station 180) and provide instructions to wireless communication component 160 to establish the wireless connection based on a determination of whether wireless communication component 160 establishes a wireless connection with satellite device 170 or base station 180.

[0055] After providing the instructions, controller 145 may receive wireless connection information from wireless communication component 160 and determine whether a wireless connection is possible based on the wireless connection information. Instead of providing instructions based on a determination of whether wireless communication component 160 needs to establish a wireless connection, controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection without determining whether wireless communication component 160 may establish a wireless connection. For example, controller 145 may provide instructions based on received data.

[0056] As part of determining whether wireless communication component 160 should establish a wireless connection, controller 145 may obtain information identifying the signal strength of satellite device 170 or information identifying the signal strength of base station 180 from wireless communication component 160. When the signal strength of satellite device 170 or the signal strength of base station 180 meets a signal strength threshold, controller 145 may provide instructions to wireless communication component 160 to establish a wireless connection. Conversely, if both the signal strength of satellite device 170 and the signal strength of base station 180 do not meet the signal strength threshold, controller 145 may determine that wireless communication component 160 may not establish a wireless connection (e.g., with satellite device 170 or base station 180). Controller 145 may determine that the data (from one or more stereo cameras 155) should be processed locally based on a determination that wireless communication component 160 may not establish a wireless connection.

[0057] Assume that controller 145 provides instructions to wireless communication component 160 to establish a wireless connection. Based on the instructions, controller 145 may establish a wireless connection (e.g., with satellite device 170 or base station 180) in a manner similar to that described above and provide an indication that the wireless connection has been established. Assume that controller 145 determines (e.g., based on instructions from wireless communication component 160) that a wireless connection has been established. Based on the determination that a wireless connection is possible, controller 145 may cause wireless communication component 160 to transmit data (from one or more stereo cameras 155) over the wireless connection to one or more remote computing devices 175 for processing.

[0058] The data may include images acquired by one or more stereo cameras 155 and / or disparity maps generated based on the images. In some cases, machine 105 may include a LIDAR device, in which case the data may include a 3D point cloud of the environment based on data (e.g., 2D semi-data) obtained by the LIDAR device.

[0059] In some cases, controller 145 may receive an indication that wireless communication component 160 has received external machine data from one or more sensor devices 220 of one or more other machines 210. The external machine data may include data related to the Earth's surface. Controller 145 may cause wireless communication component 160 to transmit the external machine data to one or more remote computing devices 175 via a wireless connection. Additionally or alternatively, controller 145 may cause wireless communication component 160 to transmit instructions to one or more other machines 210 to cause the one or more other machines 210 to transmit the external machine data to one or more remote computing devices 175. The data and / or external machine data may be transmitted to one or more remote computing devices 175 periodically (e.g., every 5 seconds, 10 seconds, in examples) and / or based on a trigger event.

[0060] The controller 145 may cause the wireless communication component 160 to transmit expected metric information regarding the excavation operation to one or more remote computing devices 175 via a wireless connection. The expected metric information may include information identifying different types of metrics, such as, for example, an expected amount of time for each excavation cycle, an expected volume of material to be removed during each excavation cycle, an expected amount of time for the excavation operation to be completed in an area, an expected amount of fuel to be consumed during each excavation cycle and / or excavation operation, an expected amount of implement wear during each excavation cycle and / or excavation operation, etc. In some examples, the expected metric information may include information for a first weight identifying an expected amount of time for each excavation period, information for a second weight identifying an expected volume of material to be removed during each excavation period, etc. One or more weights can be different from one or more other weights.

[0061] The weights may be provided by an operator's device, a background system, etc. The weights may be based on, for example, the type of machine 105, the geographic region in which the machine 105 is located, the date and / or time the excavation operation is being performed, weather conditions in the geographic region, the amount of use of the machine 105, etc. For example, with respect to machine type, the cost (e.g., resources, complexity, and / or time) associated with replacing a work implement of a first machine type (e.g., an excavator) may outweigh the cost (e.g., resources, complexity, and / or time) associated with replacing a work implement of a second machine type (e.g., a slip loader). In this regard, minimizing work implement wear on an excavator may be advantageous over a slip steer loader. Thus, the weight of expected work implement wear may outweigh the weight of other metrics for the first machine type, and the weight of expected work implement wear may be less than the weight of other metrics for the second machine type. That is, for an operation that may be performed by at least two types of machines, a machine with a lower operating cost may be used preferentially over a machine with a higher operating cost. The comparison between the different machines may be performed before the job begins, during a simulation, or through field testing. Once the job begins, the selected machine (as a result of the simulation) may be used. In some cases, the costs may be used to optimize the path plan for the selected machine.

[0062] As another example, with respect to geographic regions, the ground surface in a first geographic region may be harder than the ground surface in a second geographic region. In this regard, it may be advantageous to ensure that the rate of implement wear in the first geographic region is minimized in the second geographic region. Thus, the cost (or weighted cost) of the expected amount of implement wear may exceed the cost (or weighted cost) of other metrics for the first geographic region, while the cost (or weighted cost) of the expected amount of implement wear may be less than the cost (or weighted cost) of other metrics for the second geographic region. An optimal path for reducing implement wear may be selected, in this case, increasing the weight of the expected amount of implement wear for the first geographic region and the second geographic region.

[0063] As yet another example, with respect to dates and / or times, a first day and / or a first hour may be associated with an amount of daylight that exceeds an amount of daylight associated with a second day and / or a second hour. In this regard, it may be advantageous to ensure that the excavation operation is completed on the next day and / or the second hour in less time than on the first day and / or the first hour. This may allow the weight of the expected amount of time to complete the excavation operation to exceed the weight of other metrics during the next day and / or the second hour, and the weight of the expected amount of time to complete the excavation operation may be less than the weight of other metrics during the first day and / or the first hour. The controller 145 may obtain the expected metric information from, for example, the machine 105's memory, a background system, a device of the machine 105 operator, etc.

[0064] The controller 145 may cause the wireless communication component 160 to transmit region information regarding the Earth's surface region via a wireless connection to one or more remote computing devices 175. The region information may include information identifying the size of the region (e.g., examples of region depth, region diameter, region length, region width, etc.), the location of the region within the Earth's surface, the geometric shape of the region, etc. Alternatively, in one example, the one or more remote computing devices 175 may receive region information from a background system, a device of the operator of the machine 105, and one or more other machines 210.

[0065] One or more remote computing devices 175 may receive the data, external data, expected metric information, and / or area information. The one or more remote computing devices 175 may process the data, external data, expected metric information, and / or area information to generate drilling information. When processing the data and / or external data (hereinafter referred to as “data”), the one or more remote computing devices 175 may perform one or more operations (e.g., filtering operations) on the data. For example, among other examples, the one or more remote computing devices 175 may process the data to remove or reduce the amount of noise (e.g., anomalous data) from the data and / or remove items in the data that may be distorting and cause the data to exhibit distortion. For example, the one or more remote computing devices 175 may perform filtering of the data (e.g., filtering of a disparity map) to remove or reduce the amount of noise (e.g., anomalous data) and / or remove examples of such items.

[0066] With respect to performing filtering, among other examples, the one or more remote computing devices 175 may process the data using spatial noise reduction filtering, temporal / time filtering, and / or digital signal processing techniques. As a result of processing the data in this manner, the one or more remote computing devices 175 may improve the resolution of the data (e.g., improve the smoothness metric of the data).

[0067] Additionally or alternatively, to remove or reduce noise and / or remove items, one or more remote computing devices 175 may process the data to remove image occlusions from the data. For example, one or more remote computing devices 175 may use one or more computer vision techniques to identify and remove one or more items that may be obstructing the line of sight of one or more stereo cameras 155. For example, one or more remote computing devices 175 may analyze the data using one or more object detection techniques (e.g., single-shot detector (SSD) technique, YOLO (You One-Only) technique, and / or similar techniques) to identify the one or more items. Additionally or alternatively to removing image occlusions, one or more remote computing devices 175 may insert data into one or more portions of the data. As an example, one or more remote computing devices 175 may perform image processing on the data (e.g., using one or more image processing techniques) to interpolate the data.

[0068] By performing one or more of the operations described above, the one or more remote computing devices 175 may enhance their measurement of data quality and increase the likelihood of identifying useful information that facilitates operation of the machine 105. Based on performing one or more of the operations described above, the one or more remote computing devices 175 may generate filtered data. The one or more remote computing devices 175 may process the filtered data (e.g., using semantic segmentation techniques) to determine examples such as information about one or more pits or trenches on the earth's surface, information about one or more stacks of material on the earth's surface, information about the material, and / or information about the material within the machine work tool 140.

[0069] Among other examples, the information regarding the one or more pits or trenches may include the location of the one or more pits or trenches, the depth of the one or more pits or trenches, and / or the geometry (or shape) of the one or more pits or trenches. The information regarding the one or more piles may include, for example, the location of the one or more piles, the height of the one or more piles, the geometry (or shape) of the one or more piles, and / or the volume of the one or more piles. The information regarding the material may include the type of material, such as sand, clay, and / or hard rock. Among other examples, the information regarding the material within the machine work tool 140 may include the volume of the material, the geometry (or shape) of the material, and / or the fill level of the machine work tool 140. The information regarding the one or more pits or trenches, the information regarding the one or more piles, and / or the information regarding the material may be included in the surface data.

[0070] The surface data may identify the current topography of the earth's surface. The one or more remote computing devices 175 may compare the current topography of the earth's surface with the expected topography of the earth's surface after the excavation operation to determine examples of one or more first portions of the earth's surface from which material will be removed, one or more second portions of the earth's surface from which material will be added, etc. This area information may identify the expected topography.

[0071] Based on a comparison of the current topography of the earth's surface with the expected topography of the earth's surface, the one or more remote computing devices 175 may identify multiple sequences of drilling locations (and paths between successive drilling locations) and corresponding dump locations. In some cases, the one or more remote computing devices 175 may evaluate multiple sequences of drilling locations and corresponding dump locations with respect to expected metric information. For example, a first sequence (of the multiple sequences) meets an expected amount of time per drilling cycle, a second sequence (of the multiple sequences) meets an expected amount of material to be removed per drilling cycle, a third sequence (of the multiple sequences) meets an expected amount of time it will take to complete drilling operations in the area, a fourth sequence (of the multiple sequences) meets an expected amount of fuel to be consumed per drilling cycle and / or during drilling operations, and a fifth sequence (of the multiple sequences) meets an expected amount of time per drilling cycle and an expected amount of fuel to be consumed per drilling cycle and / or during drilling operations.

[0072] The one or more remote computing devices 175 may evaluate multiple sequences of excavation locations and corresponding dump locations and identify an optimal sequence of excavation locations and corresponding dump locations, where “optimal” means providing one or more improved sequences in terms of time, number of work machine trips, and / or fuel involved in completing the task, and / or amount of material moved. The optimal sequence may be optimal because it is associated with the lowest total weighted cost of the total weighted costs of the multiple sequences. The one or more remote computing devices 175 may use a machine learning model to identify multiple sequences of excavation locations and corresponding dump locations and identify an optimal sequence of excavation locations and corresponding dump locations. For example, the one or more remote computing devices 175 may provide data, area information, expected metric information, etc. as input to the machine learning model. The machine learning model may identify (or predict) a sequence of excavation locations and corresponding dump locations (e.g., an optimal sequence) based on the input. The predicted sequence may be a sequence in which parameters (of the machine learning model) may be selected to minimize a loss function used during training of the machine learning model. The loss function may encode many of the same weighted costs that may be considered in the numerical optimization. As its output, the machine learning model may provide an optimal sequence of drilling locations and corresponding dumping locations.

[0073] The machine learning model may be trained using historical data of the earth's surface (e.g., historical images of the earth's surface, historical disparity maps of the earth's surface and / or historical point clouds of the earth's surface), historical sizes of the area (e.g., historical depths, historical diameters, etc.), historical predictive metric information regarding historical drilling operations, etc. The machine learning model may be trained in a manner similar to that described above. In addition to or as an alternative to using a machine learning model, the one or more remote computing devices 175 may use, for example, one or more numerical optimization models, one or more graphical search algorithms, one or more heuristic algorithms, one or more overlay path planning algorithms, etc.

[0074] The one or more remote computing devices 175 may generate the excavation information based on the optimal sequence. In some examples, the excavation information may include information identifying a sequence of paths between excavation locations (e.g., geographic locations) and consecutive excavation locations, information identifying corresponding dump locations (e.g., geographic locations), information identifying a method of dumping material at each dump location, recommended machine attitude data identifying an orientation of the machine 105 at each excavation location, recommended tool attitude data identifying a position and / or orientation of the machine work tool 140 at each excavation location, recommended link attitude data identifying a position and / or orientation of the boom 130 and / or stick 135 at each excavation location, information identifying an amount of material to be excavated (or removed) at each excavation location, surface data, and other examples.

[0075] Wireless communication component 160 may receive excavation information from one or more remote computing devices 175 via a wireless connection and provide the excavation information to controller 145 to facilitate the excavation operation. For example, controller 145 may navigate machine 105 to a first one of the excavation locations based on the excavation information. Controller 145 may adjust the orientation of machine 105 (e.g., the orientation of machine body 115) based on the recommended machine attitude data when machine 105 is at the first excavation location. For example, controller 145 may match the orientation of machine 105 to the orientation of machine 105 at the first excavation location. For example, machine 105 may move to position A and then rotate ground engaging members 110 (e.g., tracks) in the opposite direction so that machine 105 faces O and begins excavating in O' direction. Alternatively, once machine 105 reaches position A, machine 105 may move to position A and use a planned path that places machine 105 in O' direction. After reaching position A, machine 105 may begin excavating in O' direction. Similarly, the controller 145 may adjust the orientation and / or position of the links based on the recommended link attitude data, and / or may adjust the orientation and / or position of the machine work tool 140 based on the recommended work implement attitude data.

[0076] After coordinating, controller 145 may start and end the drilling operation at the first drilling location and advance machine 105 toward the second of the drilling locations along a path between the first and second drilling locations. After starting the drilling operation, controller 145 may cause wireless communication component 160 to transmit additional data regarding the earth's surface (e.g., acquired by one or more stereo cameras 155) to one or more remote computing devices 175. Additionally or alternatively, controller 145 may cause wireless communication component 160 to transmit machine attitude data, implement attitude data, link attitude data, and / or current measurement information to one or more remote computing devices 175.

[0077] The current metering information may include volumetric information of material removed by the machine 105 during the excavation operation, information identifying the amount of time elapsed between one or more excavation cycles that have occurred, measurements of ground surface hardness during the excavation operation (e.g., based on data from a hydraulic displacement sensor), measurements of ground surface humidity during the excavation operation (e.g., based on data from a soil moisture sensor), and other examples.

[0078] The controller 145 may determine the volume of material based on one or more images included in the additional data. For example, the controller 145 may use the one or more images to identify an area of ​​interest (e.g., examples of material within the mechanical work tool 140, a stack of material at a dump site corresponding to the first excavation location, etc.). For example, the controller 145 may analyze the one or more images using one or more object detection techniques (e.g., Single Lens Detector (SSD) technique, YOLO (You Load) technique, halftone dot screen, and / or similar techniques) to identify the area of ​​interest.

[0079] The controller 145 may generate a graphical representation (e.g., examples of a two-dimensional semi-graphical representation, a three-dimensional graphical representation, etc.) of the region of interest based on the one or more images. The controller 145 may determine the volume of the graphical representation using one or more computational models, one or more computational algorithms, and / or other machine algorithms that may be used to determine the volume of the graphical representation. The controller 145 may determine the volume of the removed material as the graphically represented volume. Alternatively, the controller 145 may determine the volume of the removed material as a volume derived from the interior surface of the machine work tool 140 based on information identifying the volume derived from the interior surface of the machine work tool 140. The controller 145 may obtain the information identifying the volume derived from the interior surface from examples of the memory of the machine 105, a background system, etc.

[0080] The controller 145 may cause the additional data (from one or more stereo cameras 155), machine attitude data, work implement attitude data, link attitude data, and / or current metric information to be transmitted periodically (e.g., every 30 seconds, every minute, every minute, and other examples) and / or based on a trigger event (e.g., after each excavation cycle). The controller 145 may cause the additional data (from one or more stereo cameras 155), machine attitude data, work implement attitude data, link attitude data, and / or current metric information to be transmitted to the wireless communication component 160 to cause the one or more remote computing devices 175 to determine whether to update the excavation information. For example, in determining whether the excavation information should be updated, the one or more remote computing devices 175 may determine whether a material fill level in the machine work tool 140 meets a threshold fill level. For example, the additional data may include one or more images of the mechanical work tool 140 (acquired during one or more excavation periods at the first excavation location), and the one or more remote computing devices 175 may analyze the one or more images to determine the fill level of the mechanical work tool 140.

[0081] In some cases, one or more remote computing devices 175 may analyze one or more images using one or more computational models, one or more computational algorithms, and / or other machine algorithms that may be used to determine a volume (e.g., a volume of material within machine work tool 140). If one or more remote computing devices 175 determine that the fill level of material within machining work tool 140 does not meet a threshold fill level or if the volume does not meet a threshold volume, one or more remote computing devices 175 may determine that the excavation information should be updated. In other words, if one or more remote computing devices 175 determine that machine work tool 140 has not excavated a sufficient amount of material, one or more remote computing devices 175 may determine that the excavation information is updated.

[0082] If the one or more remote computing devices 175 determine that the fill level of material does not meet the threshold fill level, the one or more remote computing devices 175 may update the recommended tool pose data, the recommended link pose data, and / or the recommended machine pose data. The one or more remote computing devices 175 may perform an update that causes the machine 105 to increase the volume of material to move during the excavation period (e.g., at the first excavation location and / or the second excavation location). In some cases, based on the update, the controller 145 may send instructions to one or more other machines 210 to autonomously navigate to the current location of the machine 105 and move an additional amount of material to increase the amount of material moved.

[0083] The one or more remote computing devices 175 may make similar adjustments based on measurements of ground surface hardness and / or measurements of ground surface moisture. Assume that the one or more remote computing devices 175 analyze the additional data and determine that the excavation operation has encountered a fault along the path between the first excavation location and the second excavation location. The one or more remote computing devices 175 may update the sequence of excavation and / or dump locations to add and / or remove excavation and / or dump locations.

[0084] In some cases, one or more remote computing devices 175 may determine current (and / or forecasted) weather at a current work location of the machine 105 and update the excavation information based on the current (and / or forecasted) weather. For example, if one or more remote computing devices 175 determine that the current (and / or forecasted) weather at the current work location is unfavorable for excavation operations, the one or more remote computing devices 175 may identify a different work location. The different work location may be the next work location in the machine 105's line of work locations or may be a work location not included in the line of work locations. The one or more remote computing devices 175 may include information identifying the different work location in the updated excavation information to cause the machine 105 to abort excavation operations at the current work location and navigate to the different work location.

[0085] The one or more remote computing devices 175 may generate updated excavation information based on one or more of the adjustments described above. The updated excavation information may include, by way of example, an updated sequence of excavation locations, updated information identifying corresponding dump locations, updated information identifying how material should be dumped at each dump location, updated recommended machine attitude data identifying the orientation of the machine 105 at each excavation location, updated recommended tool attitude data identifying the position and / or orientation of the machine work tool 140 at each excavation location, updated recommended link attitude data identifying the position and / or orientation of the boom 130 and / or stick 135 at each excavation location, updated surface data, and other examples. The one or more remote computing devices 175 may transmit the updated excavation information, and the wireless communication component 160 may receive the updated excavation information. The controller 145 may instruct the machine 105 to adjust the excavation operation based on the updated excavation information, as described above.

[0086] In some cases, wireless communication component 160 may transmit data as a continuous data stream such that one or more remote computing devices 175 continuously update and continuously provide updated drilling information. Wireless communication component 160 may continuously receive updated drilling information from one or more remote computing devices 175. Controller 145 may use the updated drilling information to dynamically adjust the operation of machine 105 in real time or near real time. For example, controller 145 may navigate machine 105 from a current drilling location to an updated drilling location, adjust the position and / or orientation of machine 105, adjust the position and / or orientation of boom 130, adjust the position and / or orientation of stick 135, and / or adjust the position and / or orientation of machine work tool 140.

[0087] In some cases, machine 105 may use the surface data and / or updated surface data to facilitate navigation of machine 105 through an environment (e.g., to facilitate autonomous or semi-autonomous navigation). For example, machine 105 may use the surface data and / or updated surface data to identify one or more navigable portions of an environment and identify one or more non-navigable portions of the environment. As one example, but excluding other examples, the one or more navigable portions may include one or more portions of the earth's surface that do not include one or more pits and / or one or more stakes. The one or more non-navigable portions may include one or more portions of the earth's surface that include one or more pits and / or one or more stakes, as well as other examples of items that may impede navigational operations of machine 105. In this regard, surface data may be used for autonomous (or semi-autonomous) navigation operations of machine 105. In other examples, controller 145 may use the surface data and / or updated surface data to navigate machine 105 from one excavation site to another excavation site, or from a current location associated with a work site to a destination location associated with another work site.

[0088] In some embodiments, if a wireless connection is not feasible, the controller 145 may store data (e.g., data received from one or more stereo cameras 155, one or more sensor devices 165, one or more other machines 210, etc.). For example, the controller 145 may determine that a wireless connection is not feasible as described above, and data may be received after the controller 145 determines that a wireless connection is not feasible. Based on the determination that a wireless connection is not feasible, the controller 145 may locally store (or buffer) the data for a threshold amount of time. In some examples, the controller 145 may locally buffer the data using, for example, a circular buffer (or ring buffer), a single buffer, a dual buffer, etc.

[0089] After a threshold amount of time has elapsed, controller 145 may determine whether a wireless connection is possible (as described above). Assume that controller 145 determines that a wireless connection is possible after the threshold amount of time. Controller 145 may, based on the determination, determine that a wireless connection is possible after the threshold amount of time, such that buffered data is transmitted to one or more remote computing devices 175 (as described above). In some cases, if controller 145 determines that a wireless connection is not possible after the threshold amount of time, controller 145 may process the data locally, as described below.

[0090] In some embodiments, controller 145 may cause machine 105 to suspend drilling operations if a wireless connection is not viable (e.g., not viable within a threshold time after transmitting data). For example, controller 145 may determine (as described above) that a wireless connection is not viable and that drilling information is not being received from one or more remote computing devices 175. Based on determining that drilling information is not being received, controller 145 may cause machine 105 to suspend drilling operations. Drilling operations may be suspended until wireless communication component 160 re-establishes a wireless connection and receives drilling information via the wireless connection. Instead of suspending drilling operations, controller 145 may cause machine 105 to slow down the speed at which it performs drilling operations.

[0091] In some embodiments, the controller 145 may locally process data (e.g., from one or more stereo cameras 155), external data (e.g., from one or more other machines 210), expected metric information, and / or area information for one or more of the reasons described above. As an example, the controller 145 may cause the wireless communication component 160 to perform data processing locally by causing the wireless communication component 160 to terminate a wireless connection or by preventing the wireless communication component 160 from attempting to establish a wireless connection (e.g., if a wireless connection is not feasible). The controller 145 may generate a notification indicating that the data, expected metric information, and / or area information is being processed locally. With exceptions as otherwise indicated, the notification may be provided to the operator's equipment, a back office, or the operator's equipment of one or more other machines 210.

[0092] The controller 145 may process the data, external data, expected metric information, and / or area information to generate machine excavation information, as well as one or more remote computing devices 175 that process the data. In some cases, due to computational limitations of the machine 105, the machine excavation information may be less accurate than the excavation information generated by the one or more remote computing devices 175. In some cases, controller 145 may compare the area information with information about one or more other areas that have been excavated. Information about each of the one or more other areas may be stored in memory of machine 105 in association with the corresponding excavation information.

[0093] In some examples, the controller 145 may determine (e.g., periodically and / or based on a trigger event) whether the wireless communication component 160 may establish a wireless connection in a manner similar to that described above. For example, after determining that the wireless connection is not viable, the controller 145 may determine (e.g., periodically and / or based on a trigger event) whether the wireless communication component 160 may re-establish the wireless connection. The controller 145 may cause the wireless communication component 160 to establish the wireless connection based on a determination that the wireless communication component 160 can re-establish the wireless connection. The wireless communication component 160 may establish the wireless connection as described above.

[0094] In some cases, algorithms (used by one or more remote computing devices 175 to perform the operations described above) may be updated on one or more remote computing devices 175 while the machine 105 has no or minimal downtime. For example, algorithms may be updated with new features, tools, and / or information to gather new information from existing sensors of the machine 105. In this regard, additional value (or functionality) may be added to the machine 105 without retrofitting the machine 105 with new hardware to process data differently or with no or minimal downtime for the machine 105.

[0095] Regarding updating algorithms, once a new algorithm (e.g., examples of noise and artifact removal, feature extraction, machine learning model training, etc.) is developed and validated on a non-production machine, the new algorithm may be deployed to a production machine in a different manner. In one example, the new algorithm may be substituted (on one or more remote computing devices 175) under the same identifier as the old corresponding algorithm, allowing existing machines that send data (for processing to one or more remote computing devices 175) to immediately perceive the improvement upon request. This process is a continuous, integrated style update only on the server side (e.g., only on one or more remote computing devices 175) that affects all machines that perform remote processing using one or more remote computing devices 175.

[0096] As another example, a new algorithm can be substituted (on one or more remote computing devices 175) under a new identifier, and existing machines may refresh their on-board software with an update that sends a request to the new identifier. This process is a continuous integration-style update on the server side only (e.g., on one or more remote computing devices 175) and a software distribution update on the client side (e.g., on the machine 105). No hardware changes are required, but new software must be pushed to the machine. This process allows for managed computer upgrades with minimal downtime.

[0097] As another example, a new algorithm can be provided (on one or more remote computing devices 175) under a new identifier, and existing machines may be configured to use the new identifier by manually changing their existing on-board software, e.g., via a display. This process is a continuous integration-style update on the server side and a software configuration change on the client side. No hardware changes are required on the machines 105. This process allows for control of which machines are upgraded with little to no downtime.

[0098] The number and arrangement of devices shown in Figure 2 are provided as an example. In practice, there may be additional, fewer, different, or differently arranged devices compared to the devices shown in Figure 2. Furthermore, two or more devices shown in Figure 2 may be implemented within a single device, or a single device shown in Figure 2 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (e.g., one or more devices) of system 200 may perform one or more functions described as being performed by other sets of devices in system 200.

[0099] 3 is a flowchart of an example process 300 related to remote processing of data to generate drilling information. One or more processing blocks of FIG. 3 may be performed by a controller (e.g., controller 145). One or more processing blocks of FIG. 3 may be performed by another device or set of devices separate from or including the device, such as one or more stereo cameras (e.g., one or more stereo cameras 155), a wireless communication component (e.g., wireless communication component 160), a satellite device (e.g., satellite device 170), one or more remote computing devices (e.g., one or more remote computing devices 175), and / or a base station (e.g., base station 180).

[0100] 3, process 300 may include receiving first data from one or more sensor devices regarding a surface on which the machine is to perform an excavation operation (block 310). For example, as described above, the controller may receive first data from one or more sensor devices regarding a surface on which the machine is to perform an excavation operation.

[0101] The one or more sensor devices include at least one of one or more stereo cameras or one or more light detection and ranging (LIDAR) devices, and the one or more remote computing devices include one or more cloud computing devices.

[0102] Receiving the first data includes receiving data from one or more first sensor devices of the machine, and optionally receiving data from one or more second sensor devices of one or more machines separate from the machine.

[0103] 3, process 300 may include determining that the first data may be transmitted to one or more remote computing devices to cause the one or more remote computing devices to generate the drilling information (block 320). For example, as described above, the controller may determine that the first data may be transmitted to one or more remote computing devices to cause the one or more remote computing devices to generate the drilling information.

[0104] 3, process 300 may include transmitting the first data to one or more remote computing devices based on a determination that the first data is available for transmission (block 330). For example, as described above, the controller may transmit the first data to one or more remote computing devices based on a determination that the first data may be transmitted.

[0105] 3, process 300 includes receiving excavation information from one or more remote computing devices, the excavation information including information identifying a series of excavation locations and information identifying corresponding dump locations (block 340). For example, the controller may receive excavation information from one or more remote computing devices, the excavation information including information identifying a series of excavation locations and information identifying corresponding dump locations, as described above. The excavation information includes information identifying a series of excavation locations and information identifying corresponding dump locations.

[0106] The excavation information further includes recommended implement attitude data for identifying a position and orientation of the machine's implement at each dump site in the dump site subset, and the method further includes adjusting the position and orientation of the implement based on the recommended implement attitude data during dump operations at each dump site in the dump site subset.

[0107] 3, process 300 may include causing a tool of the machine to begin an excavation operation on the earth surface based on the excavation information (block 350). For example, as described above, the controller may cause a tool of the machine to begin an excavation operation on the earth surface based on the excavation information.

[0108] The excavation information further includes recommended implement attitude data identifying a position and orientation of the machine's implement at each excavation location of the excavation location subset, and the method further includes adjusting the position and orientation of the implement based on the recommended implement attitude data prior to excavation operations at each excavation location of the excavation location subset.

[0109] The excavation information further includes recommended machine attitude data identifying an orientation of the machine at each excavation location of the subset of excavation locations, and further includes adjusting the orientation of the machine based on the recommended machine attitude data prior to excavation operations at each excavation location of the subset of excavation locations.

[0110] 3, process 300 may include, after initiating the excavation operation, transmitting second data regarding the earth surface to one or more remote computing devices (block 360). For example, as described above, after initiating the excavation operation, the controller may transmit second data regarding the earth surface to one or more remote computing devices.

[0111] 3, process 300 may include receiving updated excavation information generated based on the second data from one or more remote computing devices (block 370). For example, as described above, the controller may receive updated excavation information generated based on the second data from one or more remote computing devices.

[0112] 3, process 300 may include causing the machine to adjust excavation operations based on the updated excavation information (block 380). For example, as described above, the controller may cause the machine to adjust excavation operations based on the updated excavation information.

[0113] Process 300 includes, after performing the excavation operation, receiving third data of the environment including the earth surface from the one or more sensor devices, transmitting the third data to one or more remote computing devices, and causing the one or more remote computing devices to generate navigation information from a current location of the machine to a destination location, the current location being associated with a first work site and the destination location being associated with a second work site, receiving the navigation information from the one or more remote computing devices, and causing the machine to navigate from the current location to the destination location.

[0114] Although Figure 3 illustrates example blocks of process 300, in some embodiments, process 300 may include additional blocks, fewer blocks, different blocks, or blocks arranged differently compared to the blocks illustrated in Figure 3. Additionally or alternatively, two or more blocks of process 300 may be performed in parallel. [Industrial Applicability]

[0115] The present disclosure relates to a machine that determines whether to have one or more remote computing devices process data (from one or more sensor devices of the machine) to generate excavation information for an excavation operation autonomously performed by the machine. The data may include data about the earth surface on which the machine performs the excavation operation. The one or more sensor devices may be located on different parts of the machine (e.g., with a boom, stick, and / or work implement). For example, the data may include one or more images of the earth surface, and the one or more sensor devices may include, for example, one or more stereo cameras. The data may be transmitted to the one or more remote computing devices via a satellite device configured for high-speed communication or via a cellular network configured for high-speed communication (e.g., a 5G network).

[0116] As part of planning a drilling operation, identifying drilling information is a computationally complex and time-consuming task that requires powerful computing resources. Determining drilling information using the typically limited computing resources of a machine presents several challenges. For example, the typical computing resources of a machine may generate drilling information with reduced accuracy and / or quality relative to drilling information generated by powerful computer resources.

[0117] Furthermore, using a typical computing device to excavate information consumes excessive computing resources and computation time on the machine. These computing resources may need to be used for other operations performed by the machine. Excessive computation time may cause the machine to idle for long periods of time until the computation is complete, resulting in reduced machine productivity. To save time (e.g., to save computation time), the computation may be terminated early, in which case a suboptimal path may be selected. Alternatively, no path may be selected, potentially resulting in an error. Selecting a suboptimal path (or not selecting a path) may result in poor machine performance or failure. Therefore, using typical computing resources to determine excavation information may prevent the machine from performing other operations. Furthermore, excavators may be used in rough terrain and harsh conditions, which may cause typical computing devices to fail prematurely, preventing the determination of planning information. The present disclosure solves these problems.

[0118] The drilling information of the present disclosure provides for improving the accuracy, timeliness, and quality of data generated using the machine's typically limited computing resources, for example, by transmitting the data to one or more remote computing devices for processing. In this regard, the drilling information may enable the machine to perform drilling operations more accurately, to perform drilling operations more quickly, to perform drilling operations more efficiently, to reduce the time required to identify an optimal path, etc. Additionally, by communicating with one or more remote computing devices using a satellite device configured for high-speed communication or a cellular network configured for high-speed communication, the machine may receive the drilling information such that the machine may make adjustments in real time or near real time during drilling operations.

[0119] Additionally, the one or more remote computing devices are not subject to the rough terrain and adverse conditions of the field and are therefore not subject to premature failure. Furthermore, by transferring data to the one or more remote computing devices for processing, the machine conserves limited computing resources typically consumed when processing images.

[0120] The foregoing disclosure provides illustration and description, but is not intended to completely limit or restrict the embodiments to the precise form disclosed. Modifications and variations can be made based on the above disclosure or acquired from practice. Furthermore, any of the embodiments described herein can be combined unless the above disclosure clearly provides a reason why one or more embodiments cannot be combined. Although particular combinations of features may be recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of various embodiments. While each dependent claim listed below may depend directly on only one claim, the disclosure of various embodiments includes each dependent claim in combination with all other claims in the claim set.

[0121] As used herein, the terms "a," "one," and "a set" are intended to include one or more items and may be used interchangeably with "one or more." Additionally, as used herein, the article "the" is intended to include one or more items recited in conjunction with the article "the," and may be used interchangeably with "one or more." Additionally, the phrase "based on" is intended to mean "based at least in part on," unless otherwise noted. Additionally, as used herein, the term "or" is intended to be inclusive when used in a series and may be used interchangeably with "and / or" unless otherwise noted (e.g., when used in combination with "either" or "only one of"). Additionally, for ease of description herein, spatially relative terms, such as "below," "below," "above," "above," and the like, may be used to describe the relationship of one element or feature to another element or feature, as illustrated. Spatially relative terms are intended to encompass different orientations of devices, apparatuses, and / or elements during use or operation in addition to the orientation depicted in the figures. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may be interpreted accordingly.

Claims

1. A method performed by a controller (145) of a machine (105), comprising: receiving first data from one or more sensor devices (155, 220) relating to a surface on which the machine (105) is performing an excavation operation; determining that the first data can be transmitted to one or more remote computing devices (175) to cause the one or more remote computing devices (175) to generate drilling information; transmitting the first data to the one or more remote computing devices (175) based on determining that the first data is available for transmission; receiving excavation information from the one or more remote computing devices (175), the excavation information including information identifying a set of excavation locations and information identifying corresponding dump locations; initiating an excavation operation on the ground surface with a work implement (140) of the machine (105) based on the excavation information; transmitting second data relating to the earth surface to the one or more remote computing devices (175) after initiating the excavation operation; receiving updated drilling information from the one or more remote computing devices (175) generated based on the second data; causing the machine (105) to adjust excavation operations based on the updated excavation information; receiving third data of an environment including the earth surface from the one or more sensor devices (155) after performing the excavation operation; transmitting third data to the one or more remote computing devices (175) to cause the one or more remote computing devices (175) to generate navigation information from a current location of the machine (105) to a destination location, the current location being associated with a first work site and the destination location being associated with a second work site; receiving said navigation information from said one or more remote computing devices (175); navigating the machine (105) from the current location to a destination location; A method comprising:

2. The step of receiving the first data includes: receiving data from the one or more first sensor devices (155) of the machine (105); and receiving the data from one or more second sensor devices (220) of the one or more machines (210) separate from the machine (105).

3. the excavation information further includes work implement attitude data indicating the position and orientation of the work implement (140) of the machine (105) at each excavation location; The method comprises: The method of claim 1 , further comprising adjusting the position and orientation of a work implement (140) based on the work implement attitude data prior to excavation operations at each excavation location.

4. The excavation information further includes machine attitude data indicating the orientation of the machine (105) at each excavation location; The method comprises: The method of claim 1, further comprising adjusting the orientation of the machine (105) based on the machine attitude data prior to drilling operations at each drilling location.

5. the excavation information further includes, for each dump site, implement attitude data indicating a position and orientation that the implement (140) of the machine (105) should assume; The method comprises: The method of claim 1, further comprising adjusting the position and orientation of the work implement (140) based on the work implement attitude data during each dump operation at each dump site.

6. 1. A system comprising: a controller (145), the controller (145) receiving data from one or more sensor devices (155, 220) relating to the earth surface on which the machine (105) is performing excavation operations; transmitting the data to one or more remote computing devices (175), causing the one or more remote computing devices (175) to generate drilling information for a drilling operation based on the data; information identifying a series of excavation locations within the surface region and information identifying the orientation of the machine (105) at each excavation location. Based on the drilling information, receiving the drilling information from the one or more remote computing devices (175); Based on the drilling information, Navigating the machine (105) to a drilling site and turning it according to its orientation at the drilling site; A system configured to cause a work implement (140) of the machine (105) to perform excavation operations at the excavation site.

7. the excavation site is a first excavation site; The controller (145) further comprises: volumetric information regarding a volume of material removed during excavation operations at the excavation site; transmitting volumetric information to one or more remote computing devices (175), the volumetric information being transmitted to cause the one or more remote computing devices (175) to generate updated excavation information; From said one or more remote computing devices (175), receiving the updated excavation information including work implement attitude data indicative of a position and orientation of a work implement (140) at a second excavation location in the series of excavation locations; The system of claim 6, configured to adjust a position and orientation of the work implement (140) based on the work implement attitude data prior to excavation operations at the second excavation location.

8. The controller (145) transmitting additional data regarding the earth surface to one or more remote computing devices (175) after initiating said drilling operation; receiving updated drilling information from the one or more remote computing devices (175) generated based on the additional data; The system of claim 6 , further configured to cause the machine (105) to adjust excavation operations based on updated excavation information.

9. The controller (145) After initiating the excavation operation, additional data relating to the ground surface is transmitting additional data to the one or more remote computing devices (175), which is transmitted to generate updated drilling information; determining that updated drilling information has not been received from the one or more remote computing devices (175) within a threshold time period after transmitting the additional data; based on a determination that the updated drilling information has not been received from the one or more remote computing devices (175); causing the machine (105) to cease excavation operations; or The system of claim 6 , configured to cause the machine (105) to reduce a speed at which it performs the excavation operation.

Images