Method and system for determining machine condition - Patents.com

Abstract

A method and system for determining machine condition. The machine (100) includes a rotation sensor (124) configured to sense rotation of the machine's upper frame (102) relative to the machine's lower frame (104). The machine (100) also includes a three-dimensional position sensor (126) spaced apart from an axis of rotation (106) of the upper frame (102) relative to the lower frame (104). The machine (100) may also include a number of additional sensors, including sensors for detecting track motion (128), image sensors (130), ranging sensors, IMUs, linear displacement sensors, and / or similar sensors. A computing system receives various inputs from the sensors and fuses the data to determine status information for the machine (100).

Description

[Technical field]

[0001] The present disclosure relates generally to machine state determination, and more specifically, to methods and systems for determining a machine's orientation and / or position by fusing data generated by multiple on-machine sensors. [Background technology]

[0002] To facilitate some civil engineering operations activities (e.g., mining, construction, dredging, excavation, etc.), including remote control and / or autonomous activities, information regarding the location and / or orientation of the machine may be required. For example, some computer-aided excavation operations require locating the machine, such as in the environment, and / or determining the orientation of the machine prior to and / or during excavation. Traditionally, such locating and / or determining the orientation may be based on global positioning sensors, including GNSS (Global Navigation Satellite System) sensors. For example, some traditional applications use two GNSS sensors coupled to known spaced locations on a portion of the body. The portion to which the sensors are coupled moves relative to one or more other portions of the body. The global position from the two GNSS sensors (and their relationship to the machine body) provides the position and orientation, or heading, of the machine. However, it is expensive to equip a machine, especially a fleet of machines, with two GNSS sensors. Therefore, it may be desirable to provide a system that uses a single GNSS sensor and utilizes data from other sensors to determine the precise location and / or orientation of the vehicle.

[0003] System designs focus on attempting to determine the relative positions of machine implements and / or parts. For example, Neyer et al., patent publication GB2571004A ("Publication 004"), describes a system and method for controlling a mobile machine using various sensor modalities. For example, Publication 004 may describe equipping a mobile work machine with displacement sensors, angle sensors, inertial sensors, rotational rate sensors, acceleration sensors, and one or more cameras to determine the position of a manipulator arm.

[0004] While the system described in the '004 publication may include sensors for determining various conditions of the work machine, the '004 publication does not describe many of the techniques for determining the state of the machine, such as the orientation of the machine, that are detailed therein.

[0005] The present disclosure is directed to one or more improvements over the prior art. Summary of the Invention

[0006] An example machine includes a lower frame configured to move along a surface, an upper frame rotatable relative to the lower frame, a first sensor configured to measure an angular orientation of the upper frame relative to the lower frame, a Global Navigation Satellite System (GNSS) sensor coupled to the upper frame and configured to sense a global position, one or more processors, and a memory having executable instructions stored thereon, which when executed by the one or more processors cause the machine to perform the following actions: accept a first input from the first sensor indicative of a first angular orientation of the upper frame relative to the lower frame at a first time, accept a second input from the first sensor indicative of a second angular orientation of the upper frame relative to the lower frame at a second time, receive from the GNSS sensor a first global position associated with the first time and a second global position associated with the second time, and determine an orientation of the machine based at least in part on the first input, the second input, the first global position, and the second global position.

[0007] An exemplary system includes a machine having a lower frame and an upper frame movable relative to the lower frame, a three dimensional position sensor disposed on the upper frame at a location spaced from an axis of rotation of the upper frame relative to the lower frame, an additional sensor coupled to the machine, one or more processors, and a memory having executable instructions stored therein that, when executed by the one or more processors, cause the machine to perform the following actions: receive first sensor data associated with a first machine position and second sensor data associated with a second machine position different from the first machine position from the additional sensor, determine an estimated movement of the machine between the first and second machine positions based at least in part on the first and second sensor data, receive a plurality of global positions associated with the machine from the three dimensional position sensor, and determine an orientation of the machine at the second machine position based at least in part on the plurality of global positions associated with the machine and the estimated movement of the machine.

[0008] One example of a method for determining a machine orientation includes receiving sensor data from a rotation sensor indicating a rotation of an upper frame of the machine relative to a lower frame of the machine; determining an estimated arc of the three-dimensional position sensor based at least in part on the sensor data and a position of the three-dimensional position sensor; receiving a plurality of global positions from the three-dimensional position sensor; and determining an orientation of the machine based at least in part on the plurality of global positions and the estimated arc. [Brief description of the drawings]

[0009] [Figure 1] 1 depicts an exemplary environment for determining a machine's position and / or orientation, according to an embodiment of the present disclosure. [Diagram 2] 1 is a schematic diagram of an example of determining a machine's position and / or orientation using sensor data from a single three-dimensional position sensor and a rotation sensor, according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 illustrates the example of FIG. 2 according to an embodiment of the present disclosure. [Figure 4]FIG. 13 is a schematic diagram of another example of determining a position and / or orientation of a machine, according to an embodiment of the present disclosure. [Diagram 5] 13A-13C are schematic diagrams of further examples of determining a position and / or orientation of a machine, according to an embodiment of the present disclosure. [Figure 6] 1 is an example of a computing environment for determining a position and / or orientation of a machine according to an embodiment of the present disclosure. [Figure 7] 1 is a flow diagram of an exemplary method for determining a position and / or orientation of a machine and controlling the machine based on the position and / or orientation, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] This disclosure relates generally to methods, systems, and techniques for determining machine condition. The particular components described herein may be components on a machine, such as a ground engaging machine, an earth moving machine, etc., although the techniques described herein may be applicable to any number of other machines. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like features.

[0011] FIG. 1 illustrates an example of a machine 100 implementing state determination techniques according to aspects of the disclosure. The machine 100 is illustrated as an excavator and generally includes multiple components and parts that are movable relative to one another, such as to move the machine 100, perform tasks with the machine 100, etc. More specifically, the machine 100 illustrated in FIG. 1 includes an upper or upper frame 102 movably coupled to an undercarriage, and a lower or lower frame 104. In an illustrative embodiment of the disclosure, the upper frame 102 is configured to rotate relative to the lower frame 104, for example, about an axis 106 that extends through a center of rotation of the upper frame 102. As illustrated, the axis 106 is generally aligned in the vertical or z-direction. As described in further detail herein, aspects of the disclosure include quantifying the relative rotation and using the quantification, along with other data, to determine a state of the machine 100, such as orientation and / or position.

[0012] As shown in FIG. 1, the lower frame 104 is supported on or otherwise coupled to a first track 108a and a second track 108b, collectively referred to herein as "tracks 108". The tracks 108 are independently movable, for example, to propel the machine 100 along a ground surface, which may be substantially located in an xy plane in FIG. 1. 1. These two tracks may be independently controllable to facilitate movement of the lower frame 104 (and thus the machine 100) relative to the ground surface, such as, for example, rotation, translation, etc. In some examples, the tracks 108 may be hydraulically actuated. Movement of the machine 100 along the ground surface (e.g., in an xy plane) via the tracks 108 changes the position and orientation of the machine 100. Although the tracks 108 are illustrated as including two tracks, in other embodiments, the machine 100 may include more or fewer tracks. Additionally, the machine 100 is not limited to the use of tracks. For example, but not limited to, the machine 100 may also include wheels instead of or in addition to the tracks 108.

[0013] The upper frame 102 is coupled to and rotatable relative to the lower frame 104, as described above. As shown in FIG. 21 , the cab 110 is supported on the upper frame 102. In some examples, the cab 110 can accommodate an operator of the machine 100, e.g., sized to allow the operator to control the machine 100 to perform one or more tasks. In these cases, the cab 110 may accommodate one or more user controls, such as a joystick, a touch screen display, a steering wheel, levers, user feedback devices, such as displays, gauges, and / or any other features. Additionally, although not shown in FIG. 1 , the cab 110 can also accommodate control-related systems and components of the machine 100, including, but not limited to, a computing system, an electrical system, a mechanical system, a hydraulic system, and the like. Upper frame 102 may also support one or more power sources (not shown), such as an engine, one or more batteries, configured to provide power for operation of machine 100, including, for example, operation of work implements, electronics, and steering, and / or provide torque to drive tracks 108 to propel machine 100 across terrain. Without limitation, the power source may also include a hydraulic system that controls operation of tracks 108 and / or implements and other aspects of machine 100 via the flow of hydraulic fluid.

[0014] As mentioned above, the machine 100 may be embodied as an excavator and may include a boom 112, a stick 114, and a bucket 116. The boom 112 is coupled to the upper frame 102 and configured to be movable relative to the upper frame 102, similar to conventional machines, the stick 114 is coupled to the boom 112 and configured to be movable relative to the boom 112, and the stick 114 is coupled to the stick 114 and configured to be movable relative to the stick 114. In the illustrated example, a pair of first hydraulic actuators 118 are provided to move the boom 112 relative to the upper frame 102. A second hydraulic actuator 120 is provided to move the stick 114 relative to the boom 112, and a third hydraulic actuator 122 is provided to move the bucket 116 relative to the stick. For example, but not by way of limitation, the hydraulic actuators 118, 120, 122 may be controlled to facilitate the completion of a task by the machine 100, such as an excavation task. In other examples, the machine 100 may include other and / or additional implements. For example, the bucket 116 may be replaced with a different implement connected to the stick 114. In other cases, the machine 100 may include implements other than the boom 112 and stick 114. Additionally, other and / or additional actuators may be used to control the implement, e.g., in place of the hydraulic actuators 118, 120, 122.

[0015] The machine 100 also includes a number of sensors, which are shown generally in FIG. 1. Specifically, FIG. 1 illustrates a rotational sensor 124 configured to generate sensor data related to the motion of the upper frame 102 relative to the lower frame 104, e.g., about the axis 106. Without limitation, the rotational sensor 124 may be or include one or more of a rotary potentiometer, a magnetic Hall effect sensor, one or more encoders, an inductive position sensor, a differential transformer, and the like. The rotational sensor 124 may be configured to determine an angle of the upper frame 102 relative to the lower frame 104, e.g., an angle of the upper frame 102 relative to a reference axis or plane. In some examples, the rotational sensor 124 may generate data that allows the rotation of the upper frame 102 relative to the lower frame 104 to be calculated to an accuracy of about 0.1 degrees or better. As described in further detail herein, in techniques according to the present disclosure, the sensor data generated by the rotational sensor 124 is used to determine state information, such as orientation and / or position information, of the machine 100.

[0016] The machine 100 further includes a position sensor 126. The position sensor 126 is configured to determine a position of the position sensor 126, for example, a two-dimensional or three-dimensional global or local position. In some examples, the position sensor 126 may be a Global Navigation Satellite System (GNSS) device, although other position sensors may be considered, including, but not limited to, machine target position sensors, robot total stations, robot tracking stations, and other types of sensor systems. In the example of FIG. 1, the position sensor 126 is coupled to the upper frame 102 of the machine 100, and the position sensor 126 may be a GNSS device or other conventional global positioning device that determines and outputs a global position of the device, typically at regular intervals (e.g., at a frequency of 10 Hz, or 10 outputs per second). The GNSS device may output two-dimensional or three-dimensional coordinates of the GNSS device, for example, a world coordinate system. As is conventionally known, the position of the GNSS device is subject to errors. In this example, the GNSS device, which is the position sensor 126, is located at a location away from the center of rotation of the machine, e.g., away from the axis 106. This causes the GNSS device to move when the machine 100 moves along the ground and when the upper frame 102 rotates relative to the lower frame 104, e.g., does not translate. As described in further detail herein, in the techniques of the present disclosure, the orientation and position information of the machine 100 is determined using position data generated by the GNSS device. Furthermore, unlike some previous systems, the techniques described herein can determine accurate state information utilizing data from only a single position sensor (e.g., a single GNSS device). The machine 100 may include one or more additional position sensors, including but not limited to additional GNSS devices, although such sensors may not be required in accordance with the embodiments described herein.

[0017] The machine 100 may include additional sensors in addition to the rotation sensor 124 and the position sensor 126. For example, FIG. 1 illustrates a track sensor 128 in schematic form. Although the track sensor 128 is illustrated in association with the first track 108a, other track sensors, such as the track sensor 128, may also be associated with the second track 108b. The following description generally relates to the first track 108a, but may also apply to the second track 108b. In general, the track sensor 128 is configured to generate information regarding the motion of the first track 108a. For example, the track sensor 128 may detect the motion (and the displacement associated with that motion) of the first track 108a. In other examples, the track sensor 128 may detect the degree of rotation associated with a sprocket, axle, or other mechanism for driving the first track 108a. In other examples, the track sensor 128 may sense a fluid flow associated with a fluid used to drive the track 108, for example. In a further example, the track sensor 128 may be part of, or may be a part of, a track sensing system that determines the movement of both the first track 108 a and the second track 108 b. As described in further detail herein, information related to the movement of the track 108 may be used to determine the state of the machine 100.

[0018] Other techniques and systems for determining truck motion are also contemplated. For example, truck motion may be inferred from control signals used to move machine 100. For example, such control signals may be generated in response to operator input from an operator in cab 110, from a remote operator, and / or from a computing device acting as an operator (e.g., in an autonomous system). In these examples, track sensor 128 may be configured to generate information related to these signals, and the actual motion of truck 108 and the actual motion of machine 100 may be inferred from the signals. For example, but not by way of limitation, track sensor 128 may detect the physical displacement of a user control, such as a joystick steering wheel, or may identify a control signal, e.g., an electrical signal, that is generated in response to such motion.

[0019] The machine 100 may also include one or more imaging and / or ranging sensors as sensors 130, as shown diagrammatically in FIG. 1. For example, the sensor 130 may be located at a position coupled to the upper frame 102 to generate data regarding the environment of the machine 100. In the illustrated example, the sensor 130 is mounted to the roof of the cab 110. The sensor 130 may be mounted to the roof to provide an unobstructed view, although other locations are contemplated. For example, the sensor 130 may image the environment, including objects within the environment. In some examples, a number of sensors may be provided sufficient to image a predetermined area around the machine 100. In some cases, a number of sensors may be provided sufficient to generate sensor data related to a 360-degree view around the machine. Without limitation, the sensor 130 may include one or more of a camera or other imaging device, a laser radar sensor, a radar sensor, a sonar sensor, a time-of-flight sensor, and / or the like. Although sensor 130 is shown as a single sensor, multiple instances of sensor 130 may be provided, and these instances may be of the same or different modalities. In accordance with the techniques described herein, the sensor data generated by sensor 130 may be used to determine a state of machine 100.

[0020] Machine 100 may also include implement sensors, etc. for generating information related to the implements of the machine. In the example of FIG. 1 , machine 100 includes a first implement sensor 132, a second implement sensor 134, and a third implement sensor 136. More specifically, first implement sensor 132 generates sensor data related to movement of boom 112 relative to upper frame 102, second implement sensor 134 generates sensor data related to movement of stick 114 relative to boom 112, and third implement sensor 136 generates sensor data related to movement of bucket 116 relative to stick 114. In some examples, first implement sensor 132 may be associated with one or both of first hydraulic actuators 118, second implement sensor 134 may be associated with second hydraulic actuator 120, and / or third implement sensor 136 may be associated with third hydraulic actuator 122. Without limitation, one or more of the first implement sensor 132, the second implement sensor 134, and / or the third implement sensor 136 may be configured to determine a linear displacement of each of the first hydraulic actuator 118, the second hydraulic actuator 120, and / or the third hydraulic actuator 122. For example, the linear displacement may be used to determine motion based on known kinematic relationships of the various elements of the implement. In other embodiments, the implement sensors 132, 134, 136 may measure values ​​other than linear displacement. Without limitation, one or more of the implement sensors 132, 134, 136 may be configured to detect relative rotational motion, for example at a pin or other pivot location. In yet another example, the implement sensors 132, 134, 136 may be configured to detect three-dimensional motion of associated components, such as the boom 112, the stick 114, and / or the bucket 116, independent of the relative motion. For example, one or more of the instrument sensors 132, 134, 136 may be an inertial measurement unit (IMU). In other examples, the instrument sensors 132, 134, 136 may include displacement sensors, position sensors, angle sensors, rotation sensors, accelerometers, or any other sensor capable of quantifying, deriving, or inferring motion.Although machine 100 is shown with such sensors, other types and units of sensors are contemplated.

[0021] As described above, the machine 100 is used to perform one or more tasks. Such tasks may include, for example, moving the machine 100 to different positions via the track 108 and / or actuating the bucket 116 (via the boom 112 and / or bucket bar 114) to dig, grade, or the like. The tasks may also include or require rotation of the upper frame 102 (and thus the cab 110, boom 112, stick 114, and bucket 116) relative to the lower frame 104. In some examples, the tasks may be performed in response to instructions from a human operator, for example, seated in the cab 110. The human operator may operate the machine 100 using, without limitation, levers, joysticks, touch screens, or other types of controls. The operator may also use input commands that correspond to various types of functions, actions, or events, such as a "stick in" event, a "stick out" event, a "boom up" event, a "boom down" event, a "bucket curl" event, a "bucket dump" event, or any other event associated with the movement of one or more parts of the machine 100.

[0022] FIG. 1 also illustrates a state determination system 138 associated with the machine 100. The state determination system 138 may be implemented by a computing system or device (including a computing device on the machine 100 or an off-board computing device). The state determination system 138 includes functionality to receive sensor data 140 and determine information regarding the state of the machine 100 (e.g., as state data 142). As used herein, the term "state" may refer to any characteristic or attribute associated with the machine 100. In certain examples, the state determination system 138 may determine the vehicle's position (e.g., in a world coordinate system) and / or orientation. The machine state may also additionally or alternatively include one or more of the heading, attitude, velocity, acceleration, and / or any other feature or characteristic of the machine 100. As described further below, the state determination system 138 includes one or more fusion models 146.

[0023] The sensor data 140 may include data generated by any sensor associated with the machine 100, as described above. Without limitation, the sensor data 140 may include rotation data generated by the rotation sensor 124, for example, as an angular displacement of the upper frame 102 relative to the lower frame 104. The sensor data 140 may include a position generated by the position sensor 126, for example, a global position, for example, a position as a GNSS sensor in a world coordinate system. The sensor data 140 may further include track data generated by the track sensor 128 (and / or any additional track sensors described above). Additionally, the sensor data 140 may include environmental data generated by the sensor 130 and / or the implement sensors 132, 134, 136. The sensor data 140 may also include information from sensors other than those shown in FIG. 1. For example, the sensor data 140 may also include information regarding operator commands issued by an operator of the machine 100 to move the machine 100 and / or portions of the machine 100. In some examples, the sensor data 140 may also include time data, for example as a timestamp, tag, header, etc., to enable a computing system, such as the status determination system 138 or some remote computing system, to acquire the data.

[0024] As shown in FIG. 1, the state determination system 138 may include one or more fusion models 146. As used herein, a "fusion model" refers to a mathematical model, algorithm, training model, and / or other processing components that determine state data from multiple data inputs. For example, the fusion models 146 may be executed to determine the state data 142 from the sensor data 140 generated by various sensors attached to the machine 100. For example, the fusion models 146 may process different inputs, such as the sensor data 140 described above, and use these inputs to estimate the state data 142. In some examples, each of the fusion models 146 may be used to determine different state parameters, such as orientation, velocity, acceleration, etc. In other examples, each of the models 146 may be provided based on the type of sensor data 140 received. In examples, as further detailed herein, the fusion models 146 may include estimation models, probabilistic models, training models, etc. that process the sensor data 140 to estimate the state data 142.

[0025] In some examples, one of the fusion models 146 may be configured to estimate the center of rotation of the machine 100 using position data from the position sensors 126 and rotation data from the rotation sensors 124. For example, as described in further detail below in connection with FIG. 2 and FIG. 3, the fusion model 146 may generate estimates of the center of rotation of the machine 100 from multiple instances of the global position generated by the position sensors 126 and fuse these estimates with measured angular displacements from the rotation sensors 124 to determine a more accurate center of rotation than could be obtained from the position sensors alone. As described above, the rotation sensors 124 determine the center of rotation, e.g., angular displacement about the axis 106, of the upper frame 102 relative to the lower frame 104 with greater accuracy than the positions generated by the position sensors 126. The fusion model 146 may utilize such higher fidelity rotation data to determine a more accurate location of the center of rotation and more accurately determine the position and orientation of the machine 100. Additionally, the techniques described herein may determine state data without requiring a careful calibration step. For example, fusing data from rotation sensor 124 with position data may enable accurate determination of condition data 142 during normal use of machine 100. In other words, in accordance with the techniques described herein, calibration routines may be partially or completely omitted.

[0026] Although FIG. 1 illustrates state determination system 138 including only fusion model 146, state determination system 138 may include additional components. Without limitation, state determination system 138 may include functionality to process inputs. For example, state determination system 138 may include functionality to validate input sensor data 140. For example, state determination system 138 may identify and remove unreliable data, which may include data that does not meet certain latency, fidelity, or other requirements. State determination system 138 may also include functionality to filter data, for example, by time, source, etc. For example, state determination system 138 may identify sensor data associated with periods during which machine 100 moved during the immediately preceding period (e.g., the immediately preceding second, 10 milliseconds, etc.) and pass only these inputs to fusion model 146. State determination system 138 may also process input sensor data 140 to generate data that is passed to fusion model 146. For example, state determination system 138 may receive image data and process the image data to identify features, such as edges, corners, etc. Other functions of state determination system 138 will also be understood from the description. In examples, any processing involved in determining state data 142 from sensor data 140 (and / or other data as further described herein) may be performed by state determination system 138. In other cases, some or all of the functions described herein may be performed by different components.

[0027] As discussed above, the state data 142 may include any information regarding the state of the machine 100. In some examples, the state data 142 may include a two-dimensional or three-dimensional vector indicating the relative position and orientation of the machine 100. An example of a machine pointing vector 144 is shown in FIG. 1. The machine pointing vector 144 may originate from a point on the axis 106, e.g., the center of rotation of the upper frame 102 relative to the lower frame 104, and have a direction corresponding to the front of the upper frame 102. In other examples, the direction of the machine pointing vector 144 may correspond differently to the machine 100 and / or parts of the machine 100. Also, in examples, the machine pointing vector 144 may have other associated data, including, but not limited to, speed, acceleration, reliability metrics, and / or other state information.

[0028] As previously discussed, data relating to machine 100 may be used to determine state data 142, for example, via state determination system 138 implementing fusion model 146. Examples of determining state data 142 in accordance with the techniques described herein are described in more detail below with reference to Figures 2-5.

[0029] More specifically, FIG. 2 is a schematic diagram of a machine 200, which may be the machine 100 performing a task that requires it to rotate relative to a lower frame 204 about a center of rotation 206. In this example, the upper frame 202 corresponds to the upper frame 102 of the machine 100, the lower frame 204 corresponds to the lower frame 104, and the center of rotation 206 may coincide with an axis of rotation, such as axis 106. FIG. 22 is a GNSS sensor 208, which may be an example of the position sensor 126 described above in connection with FIG. 1. In the illustrated example, the lower frame 204 does not move relative to the ground, e.g., there is no truck movement. The machine 100 may remove soil with a bucket from a first location and, for example, rotate at a second location to dump the bucket.

[0030] FIG. 2 specifically illustrates a first machine position in solid lines and a second machine position in dotted lines. In the first machine position, the upper frame 202 has a first rotational position about the center of rotation 206, and in the second machine position, the upper frame 202′ has a different rotational position about the center of rotation. As will be appreciated, the first rotational position is associated with a first time instant, and the second rotational position is associated with a second time instant, which is subsequent to the first time instant. As noted above, in the two machine positions, the lower frame 204 does not move. The upper frame 202 of the machine 200 has rotated counterclockwise (within the plane of FIG. 2) by a rotation angle θ to a second position indicated by the upper frame 202′. As noted above, the rotation angle θ is measured using the rotation sensor 124 (not shown in FIG. 2). The rotation sensor 124 provides an accurate measurement of the angular orientation (or change in orientation) of the upper frame 202 relative to the lower frame 204. For example, the rotation angle θ is the difference between the angular orientation measured the first and second time.

[0031] In the example of FIG. 2, the position (e.g., global position) of machine 100 does not change, but the state of machine 100 does change. Thus, in the example of FIG. 2, in a first position, machine 100 has a first state including a first global position and a first orientation or orientation. This first state is illustrated in FIG. 2 by first machine orientation vector 210. The first state also includes zero (linear and possibly rotational) speed, zero (linear and possibly rotational) acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with first machine orientation vector 210. Without limitation, the magnitude of the first machine orientation vector may correspond to the confidence value. In a second machine position, machine 200 has a second state including a second global position and a second orientation or orientation. This second state is illustrated in FIG. 2 by second machine orientation vector 210′. The second state also includes zero (linear and possibly rotational) speed, zero (linear and possibly rotational) acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with the second machine orientation vector 210'. Without limitation, the magnitude of the second machine orientation vector 210' may correspond to the confidence value.

[0032] In the second machine position, the vehicle's lower frame 204 does not move relative to the first state, e.g., the truck does not cause the machine 200 to move relative to the ground in the world coordinate system. However, the GNSS sensor 208 detects a change in position, e.g., because the GNSS sensor is coupled to the upper frame 202 and the upper frame 202 is moving, e.g., as shown in upper frame 202', due to the rotation of the upper frame 202 relative to the lower frame 204. The GNSS sensor 208 is coupled to the upper frame 202, which rotates relative to the lower frame 204 about the center of rotation 206, at a position spaced from the center of rotation 206, e.g., in the XY plane. Due to this spacing, the GNSS sensor 208 tracks an arc around the center of rotation 206 of the upper frame 202 relative to the lower frame 204, i.e., moving from a first position indicated by the GNSS sensor 208 to a second position indicated by the GNSS sensor 208'. As will be appreciated, as the upper frame 202 rotates from the first position to the second position, the GNSS sensor 208 records multiple global positions including global positions associated with at least the first machine state and the second machine state. Depending on the configuration of the GNSS sensor 208, e.g., how frequently the GNSS sensor 208 determines and / or generates global positions, the GNSS sensor 208 may also determine multiple global positions between the illustrated machine positions.

[0033] In the example of FIG. 2 , rotation of upper frame 202 relative to lower frame 204 is the only motion of machine 200, and the orientation of machine 200 in the second position is the orientation of the machine in the first position rotated by a rotation angle θ, e.g., as indicated by the direction of second machine orientation vector 210′, e.g., as indicated by the direction of first machine orientation vector 210.

[0034] In an ideal case, in the absence of sensor noise or errors due to, for example, machine vibrations, the global position measured by the GNSS sensor 208 may be sufficient to determine the orientation vector 210'. Specifically, since the position of the GNSS sensor 208 on the machine 208 is known, for example, relative to the center of rotation 206 of the machine 200 and / or relative to the position of an instrument on the machine 200, the GNSS sensor 208 will undergo the same rotation (e.g., rotation angle θ) as if the position generated by the upper frame 202 and the GNSS sensor 208 were on an arc at a known distance from the center of rotation 206. The two global positions generated by the GNSS sensor can be used to determine two candidate centers of rotation, for example, at the intersection of circles surrounding the global positions having a radius equal to the distance between the center of rotation 206 and the GNSS sensor 208. The center of rotation 206 can be triangulated from a third global position measured by the GNSS sensor to determine the direction of the machine orientation vector 210'.

[0035] In an ideal case, data from the GNSS sensor 208 is sufficient to determine state information, but real-world applications are different. Specifically, even with rotational motion alone, as in the example of FIG. 1, measurements from the GNSS sensor are often noisy due to GNSS technology, vibrations of the machine 100 during use, and / or other factors. To overcome such noise and real-world inaccuracies, the techniques described herein can utilize rotational data, such as the measured rotation angle θ, to accurately determine various aspects of the machine heading vector 210′, including the direction of the vector.

[0036] FIG. 3 is a schematic diagram illustrating position and orientation determination according to an embodiment of the present disclosure in more detail. Specifically, FIG. 3 illustrates measurements from GNSS sensor 208 of a first point 302(1) and a second point 302(2), the measurements corresponding to upper frame 202 at a first machine position and upper frame 202' at a second machine position. FIG. 3 also illustrates a number of additional points, including a third point 302(3), a fourth point 302(4), and an Nth point 302(N), where N represents any integer greater than four. In general, first point 302(1) through Nth point 302(N), or any subset of points, may be referred to herein as "points 302." While in the example of FIG. 3, five points 302 are illustrated, in other examples, more or fewer points (including points up to first point 302(1) and second point 302(2)) may be used.

[0037] Without limitation, embodiments of the present disclosure use the points 302 and angles determined by the rotation sensor, e.g., rotation angle θ, to determine the position and orientation of the machine. For example, a characteristic of an expected arc 304 that approximates the motion of the GNSS sensor 208 may be determined based on the placement of the GNSS sensor 208 on the upper frame 206 and the measured rotation angle θ. In some embodiments of the present disclosure, an error in the form of an offset of the arc 304 to the measured points 302, e.g., lateral distance, may be determined, and an estimated center of rotation 306 corresponding to the center of rotation 206 may be determined based on these errors. In other examples, the arc 304 may be fitted to the points 302 to obtain the estimated center of rotation 306. For example, techniques for determining the center of rotation may include least square error minimization, averaging, optimization based on model fitting, and the like. In addition, these estimation methods may provide an indication of the uncertainty regarding the final estimate, such as the variance of the error, the median error, and the like. A confidence value associated with the vehicle state estimate, e.g., a confidence value associated with the machine orientation vector, may be based at least in part on this uncertainty.

[0038] Other examples for determining the estimated center of rotation 306 are also possible. For example, in the absence of noise, the perpendicular bisectors of the lines connecting adjacent pairs of points 302 all intersect at the center of rotation 206. However, because the global position points are noisy, the techniques described herein can estimate the center of rotation for pairs of points 302. In this manner, clusters of estimated centers are generated, e.g., a cluster corresponding to each pair of points. The techniques described herein can estimate the estimated center of rotation 306 from the clusters. Such clustering can be used in combination with or instead of the fitting described above.

[0039] In some instances, the first point 302(1) and the second point 302(2) may be more accurate than other instances of point 302, for example because the machine may be stationary while sensing these positions. In the above method, the least squares error from rotation sensor 124 estimated from points 302(1), 302(2) and the center of rotation obtained from the rotation angle θ may be combined as a prior belief by appropriately weighting the error, for example by weighting it higher than other error terms.

[0040] GNSS global position data has some error. Therefore, continuous measurements by GNSS sensors may misrepresent the machine's motion, such as when there is no actual motion. For example, conventional GNSS sensors can find the global position within about 0.1 feet (e.g., about 3 centimeters). In some examples, the techniques described herein can determine the vehicle's orientation and position when data is received for two points that are outside the error range associated with the GNSS sensor. In other words, although a relatively large rotation angle θ is shown in Figures 2 and 3, this is primarily for clarity of explanation. In practice, a rotation angle θ of 5 degrees or less may be sufficient to obtain an accurate center of rotation using the methods described herein. The estimated center of rotation swing angle can then be applied to the previous position and orientation of the machine to determine the machine's orientation vector.

[0041] In the previously described example, other sensor data, such as part of the sensor data 140, may be generated at the vehicle to verify that no lateral movement of the machine is occurring. For example, as described above, information regarding the vehicle's control input may be received by the state determination component. Specifically, if the machine operator inputs a control to move the machine, for example, via a joystick, user interface, steering mechanism, or other control device, that input may be sent to the controller to move the truck 108 in a certain manner. In the example of FIG. 2, the state determination component may determine that no input or command has been given to move the truck, thereby verifying that the next frame is not moving. Other sensor data from any of the sensors described above in connection with FIG. 1 or elsewhere herein may also be used to determine that the lower frame 204 is not moving. For example, an IMU, accelerometer, and / or other sensor coupled to the lower frame may provide information to the state determination component that can verify that the lower frame is not moving (or in other examples, determine the momentum of the lower frame 204).

[0042] The aforementioned techniques are an improvement over conventional systems that require multiple GNSS sensors, such as different known positions on the upper frame 202. Other conventional systems may have a single GNSS sensor (or other type of position sensor), but require the machine to perform an extensive calibration procedure, such as at intervals before and / or during use, to determine the machine's position and orientation. For example, part of a conventional calibration procedure may involve the machine rotating the upper frame relative to the lower frame by a relatively large angle, e.g., 30 degrees or more. Through this calibration procedure, a sufficient number of spaced global positions spanning a sufficiently large angle are used to approximate the center of rotation of the machine within an acceptable error, e.g., by estimating an arc made by points. However, as discussed above, due to the large errors associated with the global positions generated by the GNSS sensors, large angles and a relatively large number of positions are required to provide a machine orientation with acceptable accuracy. On the other hand, aspects of the present disclosure fuse information from a rotation sensor and a (single) GNSS device (and / or data from other sensors) to determine accurate state information of the machine 200, including position and / or orientation, without the need for extensive calibration procedures. In some examples, the techniques described herein can calculate the position and orientation of the machine 200 with greater accuracy without requiring large calibration angles. The techniques described herein can determine machine state information with a rotation angle θ, e.g., an angular rotation of 5 degrees or less. In some examples, the techniques described herein can be performed during normal operation of the machine, e.g., while a task is being performed, thereby avoiding the need for a separate calibration procedure altogether. In other embodiments, the calibration procedure can be significantly simplified, e.g., by having a smaller angular rotation, significantly reducing the time required for calibration.

[0043] If it is known that there is no translational motion of the machine, multiple measurements from the GNSS sensor may be sufficient to determine the vehicle's position and orientation. However, translational motion may change the expected arc 304 of the GNSS sensor 208. Aspects of the present disclosure may also use additional data, such as data from the sensor data 140, to generate a machine orientation vector in response to more complex machine motion. Without limitation, the fusion model 146 may use IMU data, track data, and / or similar data, e.g., sensor data related to the lower frame 204, to determine an estimated path of the GNSS sensor for which the measurements generated by the GNSS sensor 208 are appropriate. As discussed above, relatively small movements are sufficient to perform the techniques described herein, so that the estimated path does not deviate significantly from the arc and / or the estimated arc may be isolated from other movements, e.g., translational movements, during operation of the machine.

[0044] 2 and 3, and any of the variations described in connection therewith, assume rotational motion of the upper frame 102, 202 relative to the lower frame 104, 204, with the rotation sensor 124 obtaining accurate measurements of the rotation and generating status data 142 based on the rotational data and the global position data. However, some types of machine motion may not include such relative rotation. Figures 4 and 5 show examples of these motions that do not include relative rotation.

[0045] Specifically, FIG. 4 is a schematic diagram of a machine 400, which may be machine 100 or machine 200, including an upper frame 402 and a lower frame 404. In this example, the upper frame 402 may correspond to the upper frame 102 of the machine 100, and the lower frame 404 may correspond to the lower frame 104 of the machine 100. Additionally, the upper frame 402 may be rotatable relative to the lower frame 404, e.g., about a center of rotation 406. The center of rotation 406 may coincide with an axis of rotation, such as axis 106. Also shown in FIG. 4 is a GNSS sensor 408, which may be an example of the position sensor 126 described above in connection with FIG. 1. In the illustrated example, the lower frame 404 is moving relative to the ground, but the upper frame 402 is not rotating relative to the lower frame 404, e.g., about the center of rotation 406. The machine 400 may transition from a first location to a second location, e.g., a different part of a construction site.

[0046] In the example of FIG. 4, the upper frame 402 is not rotating relative to the lower frame 404, but the machine state is changing. In the example of FIG. 4, at a first machine position 410, the machine 400 has a first state including a first global position and a first orientation or orientation. This first state is illustrated in FIG. 4 by a first machine orientation vector 412. The first state may also include a non-zero velocity, acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with the first machine orientation vector 412. Without limitation, the magnitude of the first machine orientation vector 412 may correspond to a confidence value. At a second machine position 414, the machine 400 has a second state including a second global position and a second orientation or orientation. The second state is illustrated in FIG. 4 by a machine orientation vector 416. The second state may also include a non-zero velocity, zero (linear and possibly rotational) acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with the second machine orientation vector 416. Without limitation, the magnitude of the second machine orientation vector 416 may correspond to a confidence value.

[0047] As described above, at the second machine position 414, the lower frame 404 of the machine 400 is moving relative to the ground. More specifically, the first track 418 and / or the second track 420, which may correspond to the first track 108a and the second track 108b, are actuated to cause the machine 400 to move relative to the ground in a world coordinate system. As with the previous example, the GNSS sensor 408 generates a global position related to the first and second machine positions 410, 414 (and a position between the first and second machine positions 410, 414), for example, as the GNSS sensor is coupled to the upper frame 402, which is supported by the lower frame 404. However, because the upper frame 402 does not move relative to the lower frame 404, there is no expected arc that can be compared to the measurement points, as shown in the examples of Figures 2 and 3.

[0048] In the example of FIG. 4, other sensor data, such as the types of sensor data 140 described above, may be used to estimate an expected path 422 of the GNSS sensor 408. The global position generated by the GNSS sensor 408 may then be compared to the estimated path 422. In at least some examples, the expected path 422 may be determined based at least in part on track data, such as track data from the track sensor 128 described above. For example, in response to controls input to the machine 400, the motion of the machine 400 may be input by the motion of the first track 418 and / or the second track 420, for example, by an operator, a remote control device, an autonomous control system, or other methods. In some examples, the state determination system 138 may also determine an orientation based at least in part on information regarding these controls of the machine 400. The tracks 418, 420 may be independently controlled to move relative to one another, for example, to impart a desired movement on the ground to the lower frame 404. In examples of the present disclosure, information regarding the respective controls of the tracks 418, 420 may be fused with other data to determine the expected path 422. The information about the control may include, but is not limited to, information about a user input, such as a physical displacement of a joystick or other physical control, an electrical signal, or a change in an electrical signal caused by a user input. In another example, the information about the control may include information about a drive system for moving the truck. For example, the truck may be driven by a hydraulic system, and information about hydraulic fluid flowing through the truck may be used to estimate the actual motion of the truck. In yet another example, the information about the control may be sensor data indicative of the actual motion of the truck. For example, truck sensors may be attached to the truck to determine the displacement of the truck, the rotation of a hub that drives the truck, etc.

[0049] Information related to the motion of the trucks may be used to approximate the translational and / or rotational motion of the machine 100. For example, the state determination system 138 may approximate the first truck path 424 and / or the second truck path 426, for example, using one or more fusion models 146. Since the positions of the GNSS sensors relative to the upper frame 402 are known, the expected path 422 may be determined based on one or more of the first truck path 424 and / or the second truck path 426. Global position data from the GNSS sensors 408 is also available at the time associated with the position along the estimated path 422. In some cases, the state determination system 138 calculates the translation and three-dimensional rotation (yaw, pitch and roll) of the machine relative to its center of rotation based on the machine geometry, the known positions of the sensors, and a mapping of translation amounts input from the left / right trucks to each of the sensing data described above. These estimates based on single sensors may be fused using a sensor fusion algorithm to determine an accurate estimate of the motion of the machine 100, e.g., as a fusion model 146, and then calculate the position and orientation of the machine, e.g., by applying the estimated motion vectors to previous positions and orientations of the machine. For example, in one embodiment, a Kalman filter or an extended Kalman filter may be used to track the center of rotation and orientation of the machine using the sensor measurements as inputs.

[0050] The estimated track paths 424, 426 of the GNSS sensor 408, and the resulting estimated path 422, may be subject to greater error than the estimated arc 304 determined from the rotation sensor 124. For example, the tracks 418, 420 may tend to slide, for example, based on terrain, soil conditions, and / or the like. As a result, machine orientation vectors 412, 416 calculated without the benefit of rotation of the upper frame 402 relative to the lower frame 404 may have less reliability than machine orientation vectors based at least in part on relative rotational motion. As a result, in some embodiments of the present disclosure, when a machine such as the machine 400 is controlled to move from a first position to a second position, as shown in FIG. 4, the machine 400 is controllable to impart a rotation of the upper frame 402 relative to the lower frame 404. Such control may be guided by an operator or may be guided by a controller, for example, in the absence of input from a user. As detailed herein, even small relative rotations, for example, on the order of about 5 degrees or less, are sufficient to implement the techniques described herein.

[0051] FIG. 5 illustrates additional techniques for determining state information, for example, using additional types of sensor data. Specifically, FIG. 5 is a schematic diagram of a machine 500, which may be any of the machines 100, 200, 400, and includes an upper frame 502 and a lower frame 504. In this example, the upper frame 502 may correspond to any of the upper frames 102, 202, 402, and the lower frame 504 may correspond to any of the lower frames 104, 204, 404. Additionally, as with the other examples, the upper frame 502 may be rotatable relative to the lower frame 504, for example, about a center of rotation 506. The center of rotation 506 may coincide with an axis of rotation, such as axis 106. Also illustrated in FIG. 5 is a GNSS sensor 508, which may be the position sensor 126 and / or one of the GNSS sensors 208, 408 described above. In the illustrated example, the lower frame 504 rotates relative to the ground, but the upper frame 502 does not rotate relative to the lower frame 504, e.g., rotates about a center of rotation 406. In this example, the machine may rotate in place, e.g., between a first orientation to excavate material and a second position to dump the excavated material.

[0052] In the example of FIG. 5, there is no rotation of the upper frame 502 relative to the lower frame 504, but the rotation relative to the ground changes the machine state. In the example of FIG. 5, at a first machine position 5, shown by a solid line, the machine 500 has a first state including a first global position and a first orientation or heading. This first state may have an associated first machine orientation vector (not shown in FIG. 5). The first state may also include speed, acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with the first machine position. At a second machine position, shown by a dashed line, the machine 500 has a second state including a second global position (measured by the GNSS sensor 508) and a second orientation or heading. The second state is shown in FIG. 5 by a machine orientation vector 510. The second state may also include speed, acceleration, and / or other characteristics. In some examples, a confidence value may also be associated with the machine orientation vector 510. Without limitation, the magnitude of the second machine orientation vector 510 may correspond to the confidence value.

[0053] The machine 500 also includes an image sensor 512 that may correspond to the sensor 130 shown in FIG. 1. In FIG. 4, the image sensor 512 may be a camera with a field of view 514 that extends generally in front of the machine 500. The field of view 514 in FIG. 5 is merely an example, as the image sensor 512 may have a wider or narrower field of view. Without being limiting, the field of view 514 may be easily obstructed by the tools of the machine 500, and therefore an alternating field of view, such as to one side or the rear of the machine 500, is preferred. In yet another example, a 360-degree field of view camera is used, with a portion of the total field of view being used in accordance with the techniques described herein. Regardless of the extent of the field of view 514, the image sensor 512 may generate image data related to the environment of the machine 500. In the example of FIG. 5, the image sensor 512 generates first image data 516 (shown diagrammatically in FIG. 5) that corresponds to the field of view 514 of the image sensor 512 when the machine 500 is in a first machine position (shown in solid lines). The image sensor 512 also generates second image data 518 (also shown diagrammatically in FIG. 5) that corresponds to the field of view 514 when the image sensor 512 is in a second machine position (shown in dashed lines). The first image data 516 and the second image data 518 overlap in an area 520, as also shown diagrammatically in FIG.

[0054] In the example of FIG. 5, similar to the example of FIG. 4, there is no relative rotation measured by the rotation sensor 124 that can be used to generate an estimated arc that the GNSS sensor 508 will track. However, image data generated by the image sensor 512 may be used to estimate the motion of the GNSS sensor 508. For example, the first image data 516 and the second image data 518 include images of features in the environment surrounding the machine 500. These features include, but are not limited to, buildings, power transmission towers, watchtowers, heavy equipment such as cranes, and the like. In examples, the first image data 516 and the second image data 518 may be analyzed to calculate the angle of rotation of the camera between captures.

[0055] In some examples, the image sensor 512 may capture video sequences or still images at regular time intervals as image data including the first image data 516 and the second image data 518. The state determination system 138 may implement one or more methods for inferring the rotation angle of the camera of the image sensor 512 from the image data. In one example, unique structural points such as a building 524 in the overlap region 520 may be determined and described using, for example, SIFT, SURF, or HoG features, and corresponding points in subsequent images may be determined by matching these features. The images for which matching is required have an overlap that includes at least some unique structure, such as the overlap region 520. The camera motion may be calculated from a set of known point correspondences and the position of the camera relative to a point or axis about which the machine 500 rotates in the illustrated example. Additionally, since the GNSS sensor 508 has a known positional relationship to the image sensor 512, the state determination system 138 may determine an expected arc, for example using the fusion model 146, and the global position determined by the GNSS sensor 508 may be fitted to the expected arc as described above, as shown in the examples of Figures 2 and 3. Alternatively, video sequences captured during rotational motion may be analyzed using structure-from-motion techniques that use feature tracking or optical flow known in computer vision. While the example of Figure 5 uses an image sensor, the techniques just described are equally applicable to ranging sensors, time-of-flight sensors, etc.

[0056] 2 and 3 illustrate techniques for determining state information using rotation data and GNSS data, for example from a single GNSS sensor. FIGS. 4 and 5 illustrate examples where there is no relative rotation, so other techniques are used in combination with a single GNSS sensor to estimate state, including orientation and / or position. As will be appreciated, the present disclosure is not limited to using track data, as in FIG. 4, and / or image data, as in FIG. 5, when there is no rotation data available. In contrast, the various techniques described herein may be used together, for example, using the fusion model 146. As more data types and models are used, the estimation techniques may become more robust. For example, the rotation sensor 124 may produce the most accurate data, but the track data and / or image data, although less accurate, may be used to determine state data 142. In various examples, different types of data may be weighted differently.

[0057] 1-5 and the above disclosure relate to the GNSS sensor being located on the upper frame of the machine, it should be understood that the GNSS sensor may instead be located on the lower frame of the machine and at least some of the other sensors may be located on the upper frame instead. Furthermore, as noted above, the GNSS sensor is only one example of a position sensor, and other types of position sensors may be used in place of the GNSS sensor used in the examples described herein.

[0058] 6 illustrates an example of a system 600 for determining a machine state according to an embodiment described herein. In at least one example, the system 600 may include one or more machines 602 that communicate with one or more remote computing devices 606 via one or more networks 604. For example, the machine 602 may be any of the machines 100, 200, 400, 500. The remote computing devices 606 may include flat panel computing devices, desktop computing devices, laptop computing devices, mobile computing devices, cloud-based computing devices, server-based computing devices, or any other computing devices capable of performing the functions described herein and communicating with the machine 602. The network 604 may include a local area network (LAN), a wide area network (WAN) including, but not limited to, the Internet, or any combination thereof, and may include wired and wireless communication technologies using any suitable protocols and communication technologies.

[0059] The system 600 may include one or more sensors 608, a display 610, one or more user interfaces 612, one or more controllers 614, a processor 616, a memory 618 communicatively coupled to the processor 616, and one or more communication connection components 620. In the illustrated example, the memory 618 of the machine 602 stores a state determination component 622 and a graphical user interface (GUI) generation system 624. Although these systems are shown below and described as separate components, the functionality of the various systems may differ from the conventional systems described above. Furthermore, fewer or more systems and components may be used to perform the various functions described herein. For purposes of illustration, while shown in FIG. 6 as residing in the memory 618, it may be envisioned that the state determination component 622 and / or the GUI generation system 624 may additionally or alternatively be accessible by the machine 602 (e.g., stored on or accessible by a memory separate from the machine 602).

[0060] The sensors 608 may include any sensor described herein. For example, without limitation, the sensors 608 may include the rotation sensor 124, the position sensor 126, the track sensor 128, the sensor 130, and / or other sensors described herein. Without limitation, the sensors 608 may be configured to generate data regarding aspects of the machine 602 and / or the environment of the machine 602.

[0061] User interface 612 may be provided to an operator of machine 602, for example, to enable a user to interact with machine 602. In some examples, an operator of machine 602 may access user interface 612 while the operator is within cab 110. User interface 612 may include a display screen, a touch screen, a joystick, a steering wheel, switches, pedals, and / or any other mechanism or component with which an operator can interface.

[0062] The controller 614 may include components that implement the control signals. Without limitation, the controller 614 may receive signals and determine actions to perform these controls based on operator input received via the user interface 612. The controller 614 may include hydraulic controllers, actuator controllers, electronic controllers, etc. In some cases, data from the user interface 612 and / or the controller 614 may be used to estimate machine movements, as described in further detail herein.

[0063] In at least one example, the state determination component 622 may include functionality to determine state data for the machine 602 based on sensor inputs. For example, the state determination component 622 may be substantially similar to the state determination system 138 described above. In examples, the state determination component 622 may receive sensor data from a rotation sensor and a single GNSS sensor and determine an orientation of the machine 602 based on that information. For example, the state determination component 622 may utilize one or more models, such as the fusion model 140 described above.

[0064] In some examples, the GUI generation system 624 may include functionality to generate one or more interactive interfaces for rendering, for example, via the display 610. In some examples, the GUI generation system 624 may receive information from the state determination component 622 to generate the GUI. In some examples, the GUI may show information about the machine 602 and the state of the information. The state information may include a machine orientation vector. The GUI may also present to the operator a confidence value associated with the state information. As described herein, state information determined using rotational data from a rotational sensor may be more accurate than state data determined using other techniques. This accuracy may be explained to the operator via the GUI.

[0065] The communication connection component(s) 620 enable communication between the machine 602 and the remote computing device(s) 606 and / or other local or remote device(s). For example, the communication connection component 620 can facilitate communication with the remote computing device(s) 606, e.g., over the network 604. The communication connection component(s) 620 may enable Wi-Fi based communication, e.g., via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, other wireless transmissions, or any suitable wired or wireless communication protocol by which individual computing devices may interface with other computing devices.

[0066] In some embodiments, the machine 602 can transmit information, such as instructions to generate a GUI, sensor data, etc., to the remote computing device(s) 606 via the network(s) 604. The remote computing device(s) 606 may receive such information from the machine 602 via the communication connection components 620, 640. In some embodiments, the remote computing device(s) 606 may perform some functions attributed to the machine 602, including, for example, determining a state of the machine 602 or generating a GUI. In at least one example, the remote computing device(s) 606 may include a processor 626 and one or more memories 628 communicatively coupled to the processor(s) 626. In the illustrated example, the memory 628 of the remote computing device(s) 606 may store a state determination component 630, a GUI generation component 632, a state model generation component 634, and / or include a data store 636. In examples, the state determination component 630 may be substantially the same as the state determination component 622 and the GUI generation component 632 may be substantially the same as the GUI generation component 624 .

[0067] The state model generation component 634 may include functionality to generate one or more models including, for example, the fusion model(s) 140 described above. Without limitation, the state model generation component 634 may perform one or more training processes, such as training a machine learning model to generate state information from various sensor inputs. Additionally, the state model generation component 634 may include functionality to model aspects described herein, such as the estimated path, center of gravity, and / or other mechanical components of the GNSS sensor.

[0068] The data store 636 may include models 638, which in some cases may include the fusion model(s) 146. The data store 636 may also store machine-specific information, machine-specific models, and / or other data used to determine state determination aspects, as described herein.

[0069] The remote computing device(s) 606 may also include communication connection component(s) 640 that enable communication between the remote computing device 606 and other local or remote device(s). For example, the communication connection component(s) 640 may facilitate communication with the machine 602 and / or other machines, for example, via the network 604. The communication connection component(s) 640 may enable Wi-Fi based communication, for example, via frequencies defined by the IEEE 802.11 standard, short range radio frequencies such as Bluetooth, other wireless transmissions, or any suitable wired or wireless communication protocol by which individual computing devices may interface with other computing devices.

[0070] The processor(s) 616 of the machine 602 and the processor(s) 626 of the remote computing device(s) 606 may be any suitable processor capable of processing data and executing instructions to perform the operations described herein. By way of example, and not limitation, the processors 616, 626 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts it into other electronic data that may be stored in registers and / or memory. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may be considered processors so long as they are configured to implement coded instructions.

[0071] Memory 618 and memory 628 are examples of non-transitory computer readable media. Memory 618, 628 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the functions resulting from the methods and various systems described herein. In various embodiments, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic random access memory (SDRAM), non-volatile / flash memory type memory, or any other type of memory capable of storing information. The architectures, systems, and single elements described herein may include many other logical, programming, and physical components, and those shown in the accompanying drawings are only examples relevant to the discussion herein.

[0072] Although the various systems and components are illustrated as discrete systems, these illustrations are by way of example only and more or fewer discrete systems may perform the various functions described herein. Further, functionality attributed to machine 602 may be performed on remote computing device(s) 606 and / or functionality attributed to remote computing device(s) 606 may be performed on machine 602. FIG. 7 shows a flow diagram illustrating an exemplary process 700 of the present disclosure that may be related to determining a machine state as described herein. The exemplary process 700 is illustrated as a set of steps in a logical flow diagram that represent actions or operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the steps represent computer-executable instructions stored in a memory. When such instructions are executed, for example, by a processor(s), such instructions may cause the processor(s) and / or various components, including a computing device or a machine associated with a computing device, to perform the described actions or operations. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order of the described operations is not intended to be construed as limiting, and any number of the described operations may be combined in any order and / or in parallel to realize the process. In some embodiments, one or more blocks of the process may be omitted entirely. Additionally, the process 700 may be combined in whole or in part with other methods.

[0073] More specifically, Figure 17 illustrates an exemplary process 700 for determining a state of a machine (e.g., machine 100, 200, 400, 500) and controlling a machine (e.g.,) to perform an excavation task based on the state. Process 700 may be performed by state determination system(s) / component(s) 138, 622, 630 and / or machine(s) 100, 200, 400, 500, 602, although other components may perform some or all of the operations of process 700. Additionally, the operations of process 700 do not necessarily have to be performed in the order shown in Figure 7, but may be performed in a different order consistent with disclosed embodiments.

[0074] At operation 702, process 700 may include receiving first data from a GNSS sensor. For example, when machine 100 is used to perform one or more tasks, state determination system 138 may receive sensor data generated by position sensor 126. In examples, the first data includes a number of global positions generated by the GNSS sensor as machine 100 moves. The GNSS sensor may be coupled to upper frame 102 of machine 100 at a location spaced apart from an axis about which upper frame 102 rotates. Configured in this manner, the GNSS sensor moves in response to most machine movements to generate new global positions.

[0075] At operation 704, process 700 may receive second data from the rotational sensor. For example, when machine 100 is used to perform one or more tasks, state determination system 138 may receive sensor data generated by rotational sensor 124. In examples, the second data includes a rotational displacement of machine upper frame 102 relative to lower frame 104 of machine 100. Rotational sensor 124 may be configured to determine an angular displacement of upper frame 102 relative to lower frame 104 about axis 106. Rotational sensor 124 may determine the rotational displacement with high accuracy, for example, within about 0.1 degrees.

[0076] In operation 706, the process 700 includes determining a swing angle based on the second data. As described above, the rotation sensor 124 measures angular displacement about the rotation axis 106. The swing angle may be an absolute angle, a relative angle, or other angle measurement. In the example of FIG. 2, the swing angle may be a rotation angle θ.

[0077] At operation 700, the process 700 includes estimating a center of rotation based on the swing angle and the first data. As described above in connection with the examples of Figures 2 and 3, aspects of the disclosure include determining state information using GNSS data and rotation data. For example, since the position of the GNSS sensor relative to the center of rotation is known, the swing angle may be used to determine an expected arc of the GNSS sensor. As described above, the global position determined by the GNSS sensor may be compared to the estimated arc to determine an estimated center of gravity. Other example techniques for determining an estimated center of rotation are also described herein.

[0078] At operation 710, process 700 may optionally include receiving additional data. The additional data may be any of the sensor data 140 described above and / or other information related to the machine and / or machine motion. In some cases, the additional data may include, or be used to generate, track data and / or image data described above in connection with Figures 4 and 5, respectively.

[0079] In operation 712, process 700 includes determining machine state information based on the estimated center of rotation, the machine parameters, and optionally additional data. For example, the machine state information may include position, location, orientation, speed, acceleration, and / or other information related to various aspects of the machine 100. Without limitation, the machine state information may be a machine orientation vector indicating the position and orientation of the machine. In examples, once the estimated center of rotation is determined, known spatial relationships between machine parts such as GNSS sensors, center of gravity, tracks, and / or other sensors may be used to determine the orientation of the machine. The additional data may be fused with the sensor data to determine the estimated center of gravity, to verify assumptions regarding vehicle motion (e.g., no lateral motion), and / or for other purposes detailed herein.

[0080] In operation 714, process 700 includes controlling the machine based at least in part on the machine state information. For example, machine 100 may be used to perform mining tasks, including but not limited to mining, grading, etc. In some examples, a 3D model of the terrain to be excavated may be used to assist in these operations. The state of the machine, such as the orientation of the machine, may be required to perform the operations based on the model. In other examples, operation 714 may include generating a GUI for display to an operator, site manager, or other user associated with the machine. [Industrial Applicability]

[0081] The disclosed systems and methods apply to any environment in which machine state data may be needed, such as machine control, machine positioning, etc. In one embodiment, the state determination may be characterized at least in part as a machine orientation vector. For example, the state may be based on rotational sensor data associated with the relative rotation of each part of the machine and a global position generated by a GNSS sensor. The disclosed systems and methods enable more accurate state determination, potentially without the need for complex and cumbersome calibration and / or recalibration routines. For example, the techniques described herein may reduce the time to perform calibrations and recalibrations, including the process of using a machine to perform a task.

[0082] For example, with reference to FIG. 1 , the machine 100 includes an upper frame 102 rotatable about an axis 106 relative to a lower frame 104. A rotation sensor 124 is positioned to measure a relative angular displacement between the upper frame 102 and the lower frame 104. A three-dimensional position sensor 126, e.g., a GNSS sensor, is also coupled to the upper frame 102 at a location spaced apart from the axis 106. Additional sensors may also be disposed on the machine 100, including, but not limited to, a track sensor 128, an image sensor (e.g., as sensor 130), and / or implement sensors 132, 134, 136. A state determination system 138 operating as a computing system associated with the machine 100 may receive sensor data and generated state data 142 from some or all of the sensors, for example, by applying one or more fusion models 146. In examples, state determination system 138 may estimate the motion of machine 100 based on rotation data from rotation sensor 124, track data from track sensor 128, image data from image sensor 130, and / or other sensor data, and compare the global position detected by position sensor 126 to the estimated path. For example, the comparison may identify an estimated center of rotation of the machine, and the known relationship between position sensor 126 and the center of rotation may be used to determine the orientation of machine 100.

[0083] The techniques described herein can improve the efficiency of a work site and / or the efficiency of a machine (such as machine 100). By way of example, and not by way of limitation, the techniques described herein can determine machine state information, including machine orientation, without frequent, complex, and / or disruptive calibration routines, which can result in more efficient use of machine 100, including, but not limited to, reduced fuel consumption and / or wear on parts.

[0084] For example, if an operator must perform a calibration procedure to determine the machine status, the machine cannot be used for production work tasks. Aspects of the present disclosure can determine status information about the machine while the task is being performed, thereby increasing work throughput and reducing wear on machine parts from non-work related use.

[0085] Those skilled in the art will appreciate that a computer program for implementing the disclosed techniques may be stored on and / or read from a computer-readable storage medium. The computer-readable storage medium may store computer-executable instructions that, when executed by a processor, include the processes disclosed herein. Exemplary computer-readable storage media may include magnetic storage devices, such as hard disks, floppy disks, magnetic tapes, or other magnetic storage devices known in the art, optical storage devices, such as CD-ROMs, DVD-ROMs, or other optical storage devices known in the art, and / or electronic storage devices, such as EPROMs, flash drives, or other integrated circuit storage devices known in the art. The computer-readable storage medium may be implemented by one or more components of the machine 100.

[0086] As will be apparent to those skilled in the art, the disclosed system for determining the orientation and / or position of a machine is susceptible to various modifications and variations without departing from the scope of the present disclosure. Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the various embodiments disclosed herein. The true scope of the present disclosure is indicated by the following claims and their equivalents, and it is intended that the specification and examples be considered as merely exemplary.

Claims

1. A machine (100), a lower frame (104) configured to move along a surface; an upper frame (102) rotatable relative to said lower frame (104); a first sensor (124) configured to measure an angular orientation of the upper frame (102) relative to the lower frame (104); a Global Navigation Satellite System (GNSS) sensor (126) coupled to the upper frame (102) and configured to sense a global position; one or more processors; When executed by the one or more processors, receiving a first input from a first sensor indicative of a first angular orientation of the upper frame (102) relative to the lower frame (104) at a first time; receiving a second input from a first sensor (124) indicative of a second angular orientation of the upper frame (102) relative to the lower frame (104) at a second time; receiving from the GNSS sensor (126) a first global position (302(1)) associated with the first time and a second global position (302(2)) associated with the second time; and determining an orientation (144) of the machine (100) based at least in part on the first input, the second input, the first global position, and the second global position.

2. The step of determining the orientation of the machine (100) comprises: determining an estimated path of the GNSS sensor (126) based at least in part on the first angular orientation and the second angular orientation; comparing the first global position (302(1)) and the second global position (302(2)) to an estimated path; and determining an estimated center of rotation of the machine based on the comparison.

3. The machine (100) of claim 1, wherein the difference between the first angular orientation and the second angular orientation is about 5 degrees or less.

4. The machine comprises: and further comprising additional sensors (128, 130) disposed on the machine, the action being: further comprising the action of receiving additional sensor data from the additional sensor; The machine (100) of claim 1, wherein determining an orientation of the machine (100) is further based on the additional sensor data.

5. one or more tracks (108) coupled to the lower frame (104); a controller configured to provide a control input to at least one of the one or more tracks to move the lower frame via the one or more tracks; The additional sensors (128, 130) include a track sensor (128), and the action is receiving track data from the track sensor; determining an estimated movement of the GNSS sensor (126) based at least in part on the track data; and comparing the first global position (302(1)) and the second global position (302(2)) to the estimated motion; The machine (100) of claim 4, wherein the orientation (144) of the machine (100) is based at least in part on an error associated with a comparison of the first global position and the second global position.

6. The additional sensor (128, 130) includes an image sensor (130) or a distance sensor, and the action is: receiving image data from the additional sensor (130), the image data including a first image (516) associated with the first time and a second image (518) associated with the second time; the action of identifying a feature (524) in the first image (516); the action of identifying the feature (524) in the second image (518); determining an estimated motion of the GNSS sensor based at least in part on a positional change of the feature (524) between the first image (516) and the second image (518); and comparing the first global position and the second global position to the estimated motion; The machine (100) of claim 4, wherein the machine orientation is based at least in part on an error associated with a comparison of the first global position and the second global position.

7. A system (100, 200, 300, 400, 600), A machine (100, 200, 300, 400, 602) having a lower frame (104) and an upper frame (102) movable relative to the lower frame (104); a three-dimensional position sensor (126) disposed on the upper frame (102) or the lower frame (104) at a position spaced apart from a rotation axis (106) of the upper frame (102) relative to the lower frame (104); additional sensors (124, 128, 130) coupled to the machine (100); one or more processors; When executed by the one or more processors, receiving first sensor data associated with a first machine position and second sensor data associated with a second machine position different from the first machine position from the additional sensors (124, 128, 130); determining an estimated movement of the machine (100) between the first machine location and the second machine location based at least in part on the first sensor data and the second sensor data; receiving a plurality of global positions associated with the machine from a three-dimensional position sensor (126); and determining an orientation (144) of the machine (100) at the second machine position based at least in part on a plurality of global positions associated with the machine and an estimated movement of the machine.

8. The additional sensor is a rotation sensor (124) configured to determine a rotation of the upper frame (106) relative to the lower frame (108), and the step of determining an orientation (144) of the machine (100) comprises: determining an estimated arc of the three-dimensional position sensor as the estimated motion based at least in part on the angular displacement sensed by the rotation sensor; comparing the plurality of global positions to the estimated arc; and determining an estimated center of rotation of the machine based on said comparison.

9. The additional sensors include a track sensor (128) configured to generate data related to movement of the track (108) relative to the machine (100), and determining an orientation (144) of the machine includes: receiving track data from the track sensor (128); determining an estimated path, the estimated movement, based at least in part on the track data; comparing the plurality of global positions to the estimated path; and determining an estimated center of rotation of the machine based on said comparison.

10. The additional sensor includes an image sensor (130) or a distance sensor, and the step of determining the orientation (144) of the machine includes: receiving first image data (516) and second image data (518) from the additional sensor; determining an estimated movement of the three-dimensional position sensor based at least in part on a difference between the first image data and the second image data; comparing the plurality of global positions to the estimated motion; 8. The system of claim 7, wherein the machine orientation is based at least in part on an error associated with comparing the multiple global positions to the estimated motion.

Images