Control method and interface for controlling a machine

Abstract

Embodiments relating to methods and systems are disclosed for using a control arm operated in a controller space that is linearly mapped to world space to control the movement of a machine having an end effector. In one embodiment, a method for controlling the movement of a machine including an end effector includes obtaining a commanded pose for the end effector in world space, the commanded pose being in controller space, the controller space being linearly mapped to the world space of the machine, and teleoperating the machine based at least in part on the commanded pose to control movement of the end effector.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit under 35 U.S.C. Section 119(e) of U.S. Provisional Application No. 63 / 481,021, filed January 23, 2023, the disclosure of which is incorporated by reference in its entirety.

[0002] (Field) The disclosed embodiments relate to a control method and interface for controlling a machine. [Background technology]

[0003] Controllable arms and excavation machinery are prevalent across many industries, such as construction, foundation work, mining, surgery, shipping, logistics, and others. Operation of such machines is undertaken by skilled operators who can become experts in performing various tasks by utilizing the machine's full capabilities. Despite the high demand for trained operators, a clear shortage of competent operators exists because acquiring proper training requires considerable time and practice. Furthermore, oftentimes, harsh and remote working conditions can make these jobs particularly arduous and unattractive. Harsh working environments can also induce worker fatigue, leading to reduced productivity and work quality during relatively long work sessions. This shortage of skilled operators increases costs and extends timelines associated with construction projects. Conversely, labor shortages due to lack of training also exist in low- and middle-income countries. Summary of the Invention [Means for solving the problem]

[0004] In one embodiment, the system comprises a control arm including a plurality of links with one or more joints disposed between the plurality of links; a plurality of sensors configured to sense a configuration of the one or more joints of the control arm; and at least one processor configured to: obtain a commanded pose for an end effector of the machine in world space, the commanded pose being in controller space, the commanded pose obtained from the plurality of sensors, the controller space being linearly mapped to the world space of the machine; transform the commanded pose from the controller space to a commanded machine configuration in the machine space of the machine; and control movement of the end effector based at least in part on the commanded machine configuration.

[0005] In one embodiment, a method for controlling movement of a machine including an end effector includes obtaining a commanded pose for the end effector in world space, the commanded pose being in controller space, the controller space being linearly mapped to the world space of the machine, and teleoperating the machine based at least in part on the commanded pose to control movement of the end effector.

[0006] In one embodiment, a method for controlling movement of a machine including an end effector includes obtaining a commanded pose for the end effector in world space, the commanded pose being in controller space, the controller space being linearly mapped to the world space of the machine; transforming the commanded pose to a commanded machine configuration in the machine space of the machine; and controlling movement of the end effector based on the commanded machine configuration.

[0007] In one embodiment, a method for controlling operation of a machine including an end effector includes obtaining a commanded pose of the end effector in a controller space, the controller space being linearly mapped to a world space of the machine; transforming the commanded pose in the controller space to a commanded machine configuration in a machine space; and controlling operation of the machine based on the commanded machine configuration to control movement of the end effector.

[0008] In one embodiment, a machine includes an end effector, a control arm including a plurality of links with one or more joints disposed between the plurality of links, a plurality of sensors configured to sense a configuration of the one or more joints of the control arm, and at least one processor. The processor may be configured to: obtain a commanded pose of the end effector in a controller space, the controller space being linearly mapped to a world space of the machine; transform the commanded pose in the controller space to a commanded machine configuration in a machine space of the machine; and control operation of the machine based at least in part on the commanded machine configuration to control movement of the end effector.

[0009] It should be understood that the foregoing concepts, and additional concepts discussed below, may be arranged in any suitable combination, such that the disclosure is not limited in this respect. Furthermore, other advantages and novel features of the present disclosure will become apparent from the following detailed description of various non-limiting embodiments when considered in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every figure.

[0011] [Figure 1]FIG. 1 depicts a controllable machine according to some embodiments.

[0012] [Figure 2] FIG. 2 depicts a control arm, according to some embodiments.

[0013] [Figure 3] FIG. 3 depicts a method of controlling a machine, according to some embodiments.

[0014] [Figure 4] FIG. 4 is a graph of the time to perform the task both before and after training for different controller types.

[0015] [Figure 5] FIG. 5 depicts a schematic embodiment of a system for implementing the systems and methods disclosed herein.

[0016] [Figure 6] FIG. 6 depicts a controllable machine operably coupled to a controller and a control arm, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0017] (Detailed explanation) There are many challenges associated with training operators to operate machines such as excavators and other machines, including teaching untrained operators to understand the often complex and / or nonlinear controls associated with the machine (e.g., multiple separate joysticks associated with various different joints of the machine). Untrained operators must learn and memorize machine-specific models or maps to transform desired world-space movements into the machine's joint space or other types of machine space. The operator learns how to operate the machine through many (e.g., thousands) iterations of controlled movements, which is time- and cost-intensive. For example, training an operator how to properly operate an excavator can require several years (e.g., 3-5 years) before the operator becomes skilled and adept at commanding the excavator. Therefore, training an operator on a typical machine can be viewed as a process of learning a nonlinear transformation map from the machine-joint-space interface to provide the desired world-space movement. In the excavator example, the excavator control system has complex, nonlinear behavior and is typically controlled via a pair of joysticks that command the joint velocities of the excavator's four degrees of freedom (DOF) (i.e., the cab, boom, stick, and bucket). During training, the operator must learn and / or memorize a model or map to translate world space requirements into 4DOF excavation machine joint space commands. Interaction with the real-world environment may also involve additional nonlinear transformations of the machine's joint movements to account for the machine's (e.g., the excavator's bucket) interaction points with the environment. Thus, to operate the machine properly, a trained operator learns a complete nonlinear map that translates desired targets in terms of interaction with the environment into a corresponding set of machine joint movements. This is a nontrivial task and limits the operator's ability to quickly become proficient in operating a machine that includes such controls.

[0018] In light of the above, the inventors have recognized benefits associated with organizing and using controls with a more intuitive interface, which reduces and / or simplifies the mental transitions an operator must learn to operate a machine proficiently. Specifically, the inventors have recognized benefits associated with using a control arm that can have a controller space that corresponds to a downscaled, linearly mapped version of the movements made by the corresponding real-world space of the machine. Such a controller, in some embodiments, may be referred to as a world-space interface controller. This movement of the control arm can be transformed into machine space to implement commands from the operator, as further detailed below. This may reduce or eliminate the operator burden associated with learning and memorizing a mental nonlinear map from traditional machine-space commands (e.g., joint-space velocity commands controlled using a joystick) to world-space commands (e.g., actuation of drilling engine joints).

[0019] As described above, the inventors have recognized benefits associated with controlling the movement of a machine, including an end effector, using commands received from a control arm having a controller space that can be linearly mapped to a world space that encloses the machine. Such a control method can include receiving commands from a control arm that includes multiple links with one or more linear joints (e.g., one or more prismatic joints or any other linearly translatable joints) or rotatable joints (e.g., one or more revolute joints or any other type of rotatable joints) disposed between the multiple links of the control arm, where the configuration of the control arm can correspond to a commanded pose of the end effector. Thus, the method can include obtaining a commanded pose of the end effector in controller space from the control arm and transforming the commanded pose in controller space to a commanded machine configuration in machine space. The machine can then be controlled based on the commanded machine configuration. For example, a machine arm or other support structure, including one or more actuators (e.g., a cart, telescopic joint, revolute joint, or other active actuation system associated with the end effector), can be operated to assume the commanded machine configuration. In the case of a machine that includes an arm, the rotatable joints of the arm can be rotated to a commanded set of angles, which can move the end effector to a commanded pose in world space to interact with the real-world environment in a manner commanded by the operator.

[0020] As described above, in some embodiments, the control arm may include multiple linear or rotatable joints and linkages. In some embodiments, the linear or rotatable joints and linkages of the control arm may have a downscaled arrangement similar to an arm including multiple linear or rotatable joints and linkages of a real-world or simulated machine. For example, the control arm may be downscaled to a size appropriate for use by an operator, but may have the same number of linkages and rotatable joints as the arm of the machine, with the same relationships between the linkages and joints. Of course, interface linkages and linear or rotatable joints having a different layout than the machine they are used to command are also contemplated. For example, a control arm may be used to command the movement of a machine, such as a crane, that does not include an arm and / or control arm, and the arm of the machine may have a different arrangement. Additionally, in some embodiments, a machine being controlled using the world-space interface controller disclosed herein may be a simulated machine that receives commands from a controller and determines corresponding movements of the simulated machine based on the received commands. Therefore, the disclosed systems and methods should not be limited to any particular arrangement of the control arm and / or portions of the machine it is used to control, as the disclosure is not limited in this respect.

[0021] During use, the linear or rotatable joints of the control arm can be manipulated (i.e., moved) by an operator to a commanded configuration. In some embodiments, moving the control arm can include the operator applying a force to some portion of the control arm (e.g., a handle disposed on the control arm) to move one or more linear or rotatable joints and one or more interface linkages to the commanded configuration. Optionally, the linear or rotatable joints can be configured to support at least a portion of the weight of the control arm. For example, springs and / or actuators can be associated with multiple joints to support at least a portion of the weight of the control arm and assist the operator in moving the interface linkages, helping to avoid operator fatigue. The control arm can also include one or more sensors configured to sense one or more parameters that can be used to determine a commanded attitude of the end effector. For example, the one or more sensed parameters can be used to determine the configuration (i.e., the commanded configuration) of the control arm. For example, the control arm may include one or more encoders, potentiometers, displacement sensors, angle sensors, inertial measurement units, accelerometers, gyroscopes, and / or any other suitable sensors configured to measure parameters that can be used to determine the configuration of the linear or rotatable joints and / or adjacent links of the control arm. In some such embodiments, one or more sensors may be configured to sense the angle of one or more rotatable joints. Additionally or alternatively, one or more sensors may be configured to sense the displacement of one or more linear joints.

[0022] In some embodiments, movement of the control arm may be used to command the position of the machine's end effector, and a secondary control may be used to command the orientation or other movement associated with the end effector or other portion of the machine. For example, the control arm may also include one or more secondary controls (e.g., pull levers, switches, buttons, interface screens, telescoping joints, knobs, other suitable controls, and any combination thereof) configured to control the movement of one or more corresponding portions of the machine. For example, the control arm may include a button or pull lever located on a handle grasped by an operator of the control arm configured to control the orientation or other movement of the machine and / or end effector. In one such embodiment, the secondary control may control the orientation of the excavator's end effector bucket relative to the distal end portion of the excavator's arm. Of course, other types of movement and / or secondary controls associated with the control arm's handle may also be used, as the disclosure is not so limited.

[0023] The presently disclosed methods and control systems are not limited to any particular machine or device and, therefore, may be used to control the operation of any suitable machine or device. In some embodiments, this may include controlling the operation of a machine having an arm including multiple linear or rotational joints and corresponding linkages. In such embodiments, the machine space may correspond to a joint space, including the angles associated with each joint. For example, a control arm configuration may be transformed to provide corresponding commanded angles for the joints of the machine's arm. An excavator is just one example of such a machine including a controllable arm (e.g., an excavator boom and stick). However, the disclosed systems and methods may also be applied to other machines including one or more parts that can be commanded to provide a desired movement. For example, excavators, cranes (e.g., overhead cranes, tower cranes, ship-to-shore cranes, etc.), bulldozers, skid-steer loaders, robotic arms remotely controlled by joysticks, other suitable types of construction equipment, and / or any other machine may include one or more parts that can be controlled to output a desired movement of an end effector. It should therefore be understood that the movement of the end effectors associated with these different types of machines may correspond to the operation of several different types of components, including, but not limited to, linear or rotatable joints, rotatable drums for winding / unwinding wire rope, telescoping joints, trolleys, rail-mounted components, and / or any other components that can be actuated to affect the position and / or orientation of the end effectors of the system.

[0024] As used herein, the term "end effector" may refer to a component of a machine configured to interact with the surrounding environment. For example, an end effector of an excavator may be a bucket that interacts with the surrounding environment by digging, pushing, compacting, scraping, crushing, or performing any other suitable action on the surrounding environment as directed by an operator. Other suitable types of end effectors may include, without limitation, buckets, drills, hammers, rakes, thumbs, graspers, couplers, multi-handlers, compactors, augers, forks, crushers, rippers, extenders of a container crane configured to attach to a container and lift the container, connectors attached to a distal end portion of a crane's wire rope, graspers connected to a robotic arm, surgical instruments located on a distal end portion of a robotic arm, and / or any other suitable type of end effector configured to interact with the environment surrounding the machine, and the present disclosure is not limited to such.

[0025] As used herein, the “configuration” of a control arm and / or controllable portion of a machine (e.g., an arm or other suitable portion of a machine) may refer to the relative positioning and orientation of the control arm or associated components within the machine. For example, with respect to an arm including multiple linkages, the configuration may be defined in terms of the angles between the different linkages. Alternatively, the configuration may correspond to the positioning and extension of a crane's wire ropes to position the end effector at a commanded position and / or orientation. Thus, regardless of the specific parameters associated with a given control arm and / or machine, the configuration may describe the position and / or orientation in space of the control arm and / or controlled portion of the machine. Depending on the context, the configuration may be described in controller space, machine space, or world space. For example, the configuration of an end effector (e.g., an excavator bucket) may include the position and / or orientation of the end effector in world space, while the position and / or orientation of the bucket may also be defined as the configuration of the excavator arm in machine space. Similarly, an associated control arm may have a configuration defined in terms of controller space.

[0026] Different reference frames are discussed throughout this disclosure, including controller space, machine space, and world space. World space may describe the reference frame of the real world and / or surrounding environment in which a machine is located and may interact. For example, movement of an end effector in world space corresponds to the observed physical movement of the end effector within the surrounding environment. Similarly, controller space may refer to the reference frame of a control arm, where movement of the control arm is within controller space. In some embodiments, controller space may be linearly mapped to world space, such that movement in controller space results in a corresponding scaled (up or down) movement of the end effector in world space. In contrast, machine space may refer to the reference frame of a moving part of a machine (e.g., an excavator arm or other controllable part of a machine). This may include commanded positions, angles, extension distances, or other suitable parameters that may be used to characterize the configuration of the controllable part of the machine. For example, in the case of an arm that includes multiple rotatable links and corresponding rotatable joints between the links, machine space may correspond to the angle of each joint. In such embodiments, machine space may be referred to as joint space. However, it should be understood that other types of machine spaces are contemplated, such that the present disclosure is not limited to any particular type of controllable machine that includes an end effector.

[0027] As used herein, pose may refer to the position and orientation of a component within a specific frame of reference. For example, the pose of an end effector may refer to both the position and orientation of the end effector within a particular frame of reference. From the perspective of a world space frame of reference, the pose of an end effector may refer to the position and orientation of the end effector in the surrounding environment. However, for purposes of this disclosure, it should be understood that various embodiments disclosed herein that refer to controlling the pose of a particular component also include controlling the position or orientation of the component, such that the disclosure is not limited in this respect.

[0028] As discussed further below, determining the pose of the end effector (e.g., the position and orientation of the end effector of the control arm) may involve using forward kinematics or inverse kinematics. Forward kinematics may use kinematic equations to determine the pose of the end effector using known values ​​for joint parameters (e.g., sensed angles). Inverse kinematics may use kinematic equations to determine the joint parameters needed to achieve the corresponding end effector pose.

[0029] Any of the embodiments relating to the systems described herein, including a machine and / or world space controller, may be implemented in any number of ways. For example, a system may be implemented using direct control, in which any of the disclosed control systems (e.g., the world space controller) are integrated directly into the machine. Alternatively, any of the disclosed systems may be implemented using teleoperation, in which the machine being controlled and the world space controller used to control the machine are located remotely from one another, and sensor and control signals are transmitted between the world space controller and the machine using any suitable communication protocol, including a wired communication protocol, a wireless communication protocol, a wireless network communication protocol, a cellular communication protocol, and / or any other communication method and / or system capable of transmitting information and instructions between the machine and the world space controller. Thus, it should be understood that any of the method and system embodiments disclosed herein may be implemented as either a single integrated system and / or a system in which the machine is located remotely from the control system (i.e., the world space controller) used to remotely operate the machine. For clarity, the following embodiments and examples will refer to the use of an excavator. However, as noted above, the various embodiments disclosed herein may be applied to any suitable type of machine that includes one or more parts whose movement can be controlled to interact with the surrounding environment, as the disclosure is not limited in this respect.

[0030] Turning to the figures, specific non-limiting embodiments will be described in further detail. It should be understood that the various systems, components, features, and methods described with respect to these embodiments can be used either individually and / or in any desired combination, such that the disclosure is not limited to only the specific embodiments described herein.

[0031] In some embodiments, excavator 100, i.e., machine, may include an overall excavator structure including a boom 106, an arm 104, and a bucket 102, as shown in FIG. 1 . Boom 106 may be rotatably coupled to a body 108 of excavator 100 such that it can pivot about a rotatable joint 1 aa connected to body 108. Correspondingly, arm 104 may be rotatably coupled to boom 106 such that it can pivot about a rotatable joint 1 bb between arm 104 and boom 106. Similarly, bucket 102 may be rotatably coupled to the arm such that it can rotate about a rotatable joint 1 cc between arm 104 and bucket 102. Arm 104 and boom 106 may act as individual links that are operated in conjunction with rotatable joints 1 aa, 1 bb, and 1 cc to generate a desired range of movement and orientation of bucket 102 along a commanded path and to move to a commanded machine configuration. Specifically, the controller 110 of the excavator 100 may include one or more processors configured to control one or more actuators 112A, 112B, 112C coupled to each of the separate linkages to control operation of the actuators and hence movement of the linkages relative to each other and to the excavator body 108. For example, the actuator 112A extending between the body 108 and the boom 106 controls the orientation of the boom 106 relative to the body 108, the actuator 112B extending between the boom 106 and the arm 104 controls the orientation of the arm 104 relative to the boom 106, and the actuator 112C extending between the arm 104 and the bucket 102 controls the orientation of the bucket 102 relative to the arm 104. In some embodiments, the actuators 112 may include sensors 101A, 101B, and 101C that may transmit signals to the controller 110 regarding one or more operating parameters of the actuators (e.g., extension length, articulation angle, etc.).

[0032] In the figures, actuators 112A, 112B, 112C are depicted as cylindrical hydraulic actuators. However, as the disclosure is not limited in this respect, it should be understood that the actuators may correspond to any suitable actuators capable of controlling movement of a linkage, including, but not limited to, various linear actuators, rotary actuators, hydraulic actuators, electric actuators, pneumatic actuators, thermal actuators, magnetic actuators, combinations of the foregoing, and / or any other suitable type of actuator.

[0033] The depicted excavator may also include an operator interface 114 that may be operably coupled to the controller 110, whereby the operator interface 114 transmits commands to the controller 110. For example, the operator interface 114 may transmit control arm configurations from the control arms to the controller 110. In some embodiments, the control arms may correspond to control arms similar to those depicted in FIG. 2 , although any other suitable type of control arm may also be used, as the disclosure is not so limited. Additionally, the excavator 100 may optionally include one or more joysticks 116 operably coupled to the controller 110 that, in some embodiments, may be used to control the speed of the bucket 102, the arm 104, the boom 106, and the body 108. The joystick 116 may control the speed by actuating one or more of the actuators 112. Furthermore, the excavator 100 may include one or more pedals or levers 118 operably coupled to the controller 110 that are used to control the speed of the treads and / or wheels to move the entire excavator or other suitable machine, depending on the embodiment. The controller 110 may include one or more processors and associated non-transitory computer-readable memory containing instructions that, when executed, cause the system to perform any of the disclosed methods. Specifically, as described in further detail below, in response to receiving a control arm configuration from an operator interface that may correspond to a commanded pose of the end effector in controller space, the controller 110 may transform the commanded pose in controller space into a machine configuration in machine space and move the end effector (i.e., the bucket 102) to the commanded pose accordingly, prior to actuating the actuators 112.

[0034] Although a bucket is illustrated in the figures, in some embodiments, excavator 100 may include end effectors other than bucket 102 coupled to arm 104. For example, excavator end effectors may include drills, hammers, rakes, thumbs, graspers, couplers, multi-handlers, compactors, augers, forks, breakers, rippers, or any other suitable type of end effector. In addition, end effectors for use with other types of machines may also be used in some embodiments, as described above.

[0035] It should be understood that an excavator is one example of a machine that may utilize the present disclosure, and that any other suitable machine may also utilize the present disclosure, as the disclosure is not so limited. For example, various cranes may implement the systems and methods disclosed herein. One specific example includes a container crane (i.e., a container handling gantry crane or a ship-to-shore crane). In some embodiments, the container crane may include a fixed arm (i.e., a boom) having a cab (i.e., a trolley) and an extender operably coupled to the fixed arm. The cab may be movable in the direction of a longitudinal axis associated with the fixed arm. The extender may also be movable in a direction parallel to the local direction of gravity, such that the extender may move left and right as well as up and down during operation. During operation, the extender may grip a load, such as a container or other cargo, and the extender holding the container may be moved to transport the container. In such embodiments, the extender may be considered an end effector of the crane. In other embodiments, the crane may include other end effectors.

[0036] According to some embodiments, a control arm 200 that can be used as an operator interface is shown in FIG. 2. The control arm 200 can include a handle 218 configured to be grasped by an operator's hand to apply a force to the control arm 200 to manipulate the overall configuration of the control arm 200. The control arm 200 can include multiple rotatable links joined together by corresponding multiple rotatable joints. The handle 218 can include a secondary control 220 as previously described above. The handle 218 can be coupled to a first end portion of a first link 210. The first link 210 can be attached to a second link 212 by a rotatable joint 222A and to a third link 214 by a rotatable joint 222B on a first portion of the first link 210 positioned opposite the second portion of the first link 210 to which the handle 218 is coupled. The second link 212 may be coupled to a fourth link 216 extending between and rotatably coupled to end portions of the second link 212 by a rotatable joint 222C, and the third link 214 may be coupled to a fourth link 216 extending between and rotatably coupled to end portions of the third link 214 by a rotatable joint 222D. In the depicted embodiment of FIG. 2 , the fourth link 216 is coupled to a bottom portion of the second link 212 and a bottom portion of the third link 214, and the first link 210 is coupled to an upper portion of the second link 212 and an upper portion of the third link 214, such that the second link 216 and the first link 210 are coupled to opposite ends of the second link 212 and the third link 214 from each other. In the depicted embodiment, the third and / or fourth linkages 214 and 216 may be rotatably coupled to the support 206, which corresponds to any structure configured to support the proximal portion of the control arm 200. In the depicted embodiment, the rotatable coupler 222D associated with the third and fourth linkages 214 and 216 is connected to the support 206.Thus, forces applied to the handle 218 may cause rotation of the individual rotatable joints 222A-222D relative to the support 204 and each other and corresponding movement of the links 210, 212, 214, 216 such that the handle 218 is moved between different positions in a plane perpendicular to the axes of rotation of the different rotatable joints 222. This may correspond to a change in the position of the handle 218 both vertically and horizontally relative to the local direction of gravity. This movement may correspond to a change in the vertical and horizontal position within the controller space of the control arm and / or the machine space of the excavator.

[0037] The support 206 may be coupled to a rotatable coupling 204, such as a rotatable shaft or other suitable rotatable structure that is capable of supporting the support 206. Thus, rotation of the support 206 and associated rotatable coupling 204 may enable the entire control arm 200 to yaw about the support's axis of rotation, as indicated by arrow 208. The rotational movement of the support 206 may, in some embodiments, correspond to yaw movement in controller space and / or machine space. The rotatable coupling 204 may be coupled to a base 202 on which the entire control arm 200 may be supported. The base may take any suitable shape or form, as the disclosure is not so limited. In some embodiments, the base 202 may be located on some portion of the excavator 100, such as within the body 108.

[0038] While a specific control arm 200 is shown above, it should be understood that other control arm configurations having different numbers, arrangements, and / or shapes of linkages can be used, such that the disclosure is not so limited. For example, a simple arm having a handle and two serially coupled linkages with two rotatable joints can be used, or a more complex arm having any number of linkages and rotatable joints can be used, such that the disclosure is not limited to any specific control arm configuration.

[0039] As discussed above, one or more sensors 226A-226D may be coupled to any suitable portion of the control arm 200 to sense the configuration of the control arm during operation. In some embodiments, one or more sensors 226A-226D may be an encoder configured to determine the angle of the rotatable joint 222 between the different links 210-216. An encoder may also be coupled to the rotatable coupling 208 to determine the yaw angle of the support 206 relative to the base 202. The angle or other parameter measured by the sensors 226A-226D may be used, at least in part, to determine the configuration of the control arm 200 and determine the commanded pose of the end effector. For example, the sensed angle or other parameter may be output to a separate controller of an excavator or other machine to implement command inputs to an operator using the control arm.

[0040] In some embodiments, it may be desirable to support at least a portion of the weight of the operator's arm and / or control arm to help avoid operator fatigue. Therefore, as described above, the rotatable joints 222 may include one or more biasing members 224 corresponding to springs and / or actuators associated with different rotatable joints. The biasing members may be configured to apply torque or other suitable forces to the rotatable joints and support at least a portion of the weight of the control arm and / or operator's arm. Supporting the operator's weight may serve to reduce the force applied by the operator to command a desired pose using the control arm, which may help prevent operator fatigue. The biasing members may be any suitable biasing members, such as torsion springs, compression springs, extension springs connected to separate linkages, any other suitable springs, motors, and / or any combination thereof. Additionally, while biasing members are depicted at each joint, it should be understood that biasing members may be associated with any one or more of the joints and / or other portions of the linkages of the control arm. For example, a spring may extend between adjacent links to apply a desired biasing force. Thus, any suitable configuration may be used in such embodiments to apply a desired biasing force to offload at least a portion of the weight of the control arm.

[0041] A method 300 for controlling the operation of a machine according to some embodiments is depicted in the diagram shown in FIG. 3. For example, the control arm 200 depicted in FIG. 2 or any other suitable embodiment of a control arm may be used in conjunction with method 300 to control the operation of a machine (e.g., machine 100 shown in the depicted embodiment of FIG. 1). As explained above, the control arm may include multiple linkages and rotatable joints and may be manipulated into different configurations in controller space. These configurations formed by moving the control arm may be referred to as commanded configurations or control arm configurations. For example, moving a handle within an allowed configuration may be used to command a corresponding movement of the machine's end effector in world space. This movement of the control arm may, in some embodiments, be linearly related to the corresponding movement in world space, as described above. In some instances, the command from the control arm may be a commanded pose of the end effector, although embodiments in which the control arm configuration corresponds to a commanded position are also envisioned. In either case, an operator may manipulate the handle of the control arm to command the pose of the end effector in controller space using the control arm (see 302).

[0042] To determine the command provided by the operator, the configuration of the control arm may be sensed or otherwise determined at 304 using one or more sensors. As described above, in some embodiments, this may correspond to sensing angles between different linkages and / or different rotatable joints or couplings of the control arm. As discussed above, the commanded pose may be in controller space, which may be linearly mapped to a desired corresponding movement of the end effector in world space. This relationship between movement in controller space and world space may be either linearly upscaled or downscaled depending on the desired application.

[0043] Depending on the relative design of the control arm and portions of the machine manipulated to control the pose of the end effector, a transformation from controller space to machine space may be required to implement the desired commanded pose of the end effector. Accordingly, the commanded pose in controller space may be appropriately transformed to a commanded machine configuration in machine space. Therefore, method 300 may include, at 306, transforming the commanded position in controller space to a commanded machine configuration in machine space. In embodiments where the control arm corresponds to a configuration similar to the arm it is being used to control, the transformation may be a one-to-one transformation, in which case the sensed angle of the control arm's rotatable joint in controller space may be equal to the commanded angle of the machine arm's joint in machine space. However, in other embodiments, more complex transformations (scalar factors, matrix transformations, etc.) may be required to transform commands from a control arm having a relatively different configuration compared to the portion of the machine used to operate to control the movement of the end effector.

[0044] In some embodiments, sensed angles or other parameters in controller space may be transformed to commanded machine angles in machine space using a Jacobian matrix. In embodiments where forward kinematics is used to determine the position of the end effector, the following equations may be used:

number

number

number

[0045] In embodiments where inverse kinematics is used to determine the joint parameters, the following equations are used:

number

number

number

number

[0046] In embodiments with multiple solutions (i.e., redundant configurations for the commanded position of the end effector), a cost function and / or loss function may be used to help determine the solution that will be used to control the machine. As mentioned above, when there are redundant degrees of freedom in machine space, multiple solutions may exist. Any appropriate cost function and / or loss function may be used, including regret, quadratic loss function, 0-1 loss function, linear-quadratic, as the disclosure is not so limited. A rule-based decision system may be used to determine the solution that will be selected based on the results of the cost / loss function. This may be useful in some embodiments because certain configurations and movements of the machine in world space may be less desirable than others. For example, a first configuration may require more time for the machine to move than a second configuration that accomplishes the same task. Some configurations may reduce the amount of progress available for the machine and / or reduce the available power; for example, an arm and boom that are approximately adjacent to each other is less desirable than a configuration with spaced-apart arms and booms but with the end effector in the same location. Certain configurations may also be desirable to avoid due to safety risks to the operator and / or potential damage to the machine itself, and therefore are not selected by the rule-based decision system. Examples of rule-based decisions may include maximizing smoothness, minimizing time or duration of operation, minimizing maximum torque, minimizing power or fuel consumption, any other suitable rule-based decision, and any suitable combination thereof. Thus, it should be understood that the rule-based decision system and / or cost or loss function may be implemented in any suitable manner, such that the disclosure is not limited to any particular implementation of these concepts.

[0047] As noted above, movement of the machine in machine space can result in actual movement of the machine and end effector in world space (i.e., the environment surrounding the machine). Thus, machine movement can be controlled at 308 based on the commanded machine configuration. For example, an excavator can be controlled to move its bucket to a desired position prior to commencing a digging and / or scooping operation. In another example, a crane can be controlled to move an extender holding a container both along the length of the boom and vertically. Additionally, although not shown in the method flow diagrams, in some embodiments, secondary controls can also be used to control one or more movements of the machine and / or end effector. For example, in the embodiment of FIGS. 1 and 2, control arm 200 can be moved to a commanded configuration to control the position of bucket 102 in world space, and secondary control 220 can be operated to control the orientation of the bucket in world space.

[0048] Although not depicted in the above embodiments, in some cases, such as with an excavator, it may be desirable to control the yaw movement of an arm, cab, or other portion of the machine. In such embodiments, it may be desirable to sense the yaw angle of the control arm. In the embodiment of FIG. 2, this may correspond to sensing the yaw angle of the rotatable coupling 204 connected to the support 206, although other arrangements may also be used. To provide the appropriate yaw movement, in some embodiments, the yaw movement of the corresponding portion of the machine (e.g., arm, cab, etc.) may move in a direction corresponding to a rotational offset from a neutral or 0-degree yaw angle (e.g., a straight configuration oriented forward of the operator). Thus, when the operator yaws the control arm left or right, the machine may yaw in the corresponding direction. In some embodiments, the rate of yaw movement may be proportional to the magnitude of the offset of the control arm from the neutral yaw angle. In such embodiments, it may not be necessary to convert the sensed yaw angle to the desired machine configuration. However, embodiments in which a conversion is applied are also envisioned.

[0049] The above methods may be implemented by one or more controllers including at least one processor operably coupled to various controllable portions of a machine as disclosed herein. The methods may be embodied as computer-readable instructions stored on non-transitory computer-readable memory associated with the at least one processor, such that when executed by the at least one processor, a control arm, machine, or other system may perform any of the actions associated with the methods disclosed herein. In addition, it should be understood that the order of the disclosed steps is exemplary, that the disclosed steps may be performed simultaneously in a different order, and / or may include one or more additional intermediate steps not shown, so that the disclosure is not so limited.

[0050] While the present disclosure may be applied to any suitable machine, detailed examples are presented that include a simulator for a hydraulic excavator and different organizational environments. The presented examples also detail a novel world-space interface (WS) in the form of a control arm as disclosed herein. The change in performance of a novice operator while operating an excavator in a simulated environment using a conventional joint-space joystick interface (nonlinear map) and the proposed novel world-space interface (linear map) is also presented.

[0051] The inventors developed a 3D excavator simulator that both evaluates an operator's operational skills and trains them using different interfaces. The simulator simulates multiple different virtual environments corresponding to actual construction sites and the execution of tasks at different levels of difficulty to train and evaluate the skills of inexperienced operators. In the simulator, the simulated excavator mimics the movement dynamics of a real excavator by utilizing a physical system that takes into account both the hydraulic system and the mass and inertia of the excavator's structural components.

[0052] The new WS included a linkage representing an excavator downscaled in physical size, potentially providing a more intuitive control scheme for inexperienced operators. The WS was similar in organization to the control arm shown in Figure 2. The WS included 4 DOF: one vertical rotational joint for controlling cab rotation, a parallel linkage mechanism for the boom and stick, and a secondary control (e.g., a biased lever handle) for the bucket. A spring was coupled to the parallel linkage in the control arm to provide antigravity support (i.e., a force acting in a direction opposite to the local gravity direction) to at least partially support the weight of the linkage, handle, secondary control, and the weight of the human arm. In these experiments, the weight of the arm applied to the WS was assumed to be approximately 3.5 kg. The WS used a magnetic rotary encoder, but other sensors could also be used. The secondary control (e.g., a biased lever handle) was coupled to a linear potentiometer via a brake wire to determine the angle of the secondary control. The sensed angle signal was sent via serial communication through the motor driver to an associated computer. The sensed yaw angle of the WS support was converted to cab input speed using proportional calculations.

[0053] As part of evaluating the WS, we developed a realistic 3D visual simulator for training and evaluating excavator operators performing several common tasks in 19 different environments. The system ran on a high-end computer, processed all 3D graphics within a high-end graphics card, and displayed feedback to the operator using a high-resolution monitor that displayed the simulated surrounding environment to the operator as if the operator were inside the excavator cab. The operator was able to control the virtual 3D excavator using two different physical interfaces: two joysticks or the WS detailed above. As with a real excavator, the operator could drive the virtual excavator (i.e., operate the excavator's treads) using pedals located near the operator's feet or by using two levers.

[0054] The 3D excavator simulator contained several discrete modules. The interface module included the monitor, joystick, levers, and pedals described above. There were three other modules corresponding to three programs that ran simultaneously in real time. One of the three modules was the world space module, which acquired measurements from encoders configured to measure the four degrees of freedom (DOF) of the excavator and calculated velocities that were then sent to the simulator to control the virtual excavator. The four DOFs correspond to the excavator's cab (i.e., body), boom, stick, and end effector (e.g., bucket). A logging module was used to receive data (e.g., speed, angle, bucket tip position, time, etc.) from the simulator and save the data to the computer's hard drive without slowing down the 3D simulation. A vision and logic module was used to handle all graphics and physics calculations for interacting with the 3D simulator and display on four 4K monitors. The final module was divided into three distinct blocks. The first block included management of four virtual cameras (one per monitor), rendering all 3D simulations, presenting the graphical user interface (GUI), and displaying visual effects (VFX) of specific assets (e.g., for excavation and disposal tasks). The second block was used for the movement and interaction logic of the virtual excavator. The movement and interaction logic was programmed to account for dynamics simulating the excavator's hydraulic system, including both simple and realistic behavior, collisions that occur within the environment, management for processes such as digging, loading, and discarding dirt, rocks, or other elements within the environment, and recollection of all desired data for transmission and storage. The third block contained plug-ins that managed the process of reading and writing data to the computer's shared memory area as part of the inter-process communication (IPC) mechanism used to communicate all processes in the system in real time.

[0055] Of the 19 environments, there were 15 training environments that included tasks such as digging and dumping sand, dust, gravel, or mud, clearing debris or stones from roads, moving branches and logs from shallow water, or drilling rocks. These simulated tasks were performed in different environments, including construction sites, highways, roads, and riverbeds, in various locations such as urban areas, mountains, forests, rock piles, mines, or any other suitable environment.

[0056] Training and evaluation sessions were used to collect experimental data from inexperienced operators, including 12 healthy adult males with an average age of 29 years who had never operated a real excavator before. The operators were divided into two groups: one group trained using only the WS interface and the second group trained using only the traditional joystick interface (JS). The operators were trained and evaluated in nine daily sessions. All subjects using both the WS and JS followed the same protocol (i.e., method or process). The operators completed 14 different tasks designed to train the operators in different operational skills, such as digging, dumping, drilling, and rolling. Each task was designed so that the operator could complete the task in approximately 30 minutes. Each operator was trained to operate the excavator in two different environments for 30 minutes per day over a seven-day period.

[0057] Prior to the initial training session and following completion of the training protocol, four environments were used to evaluate operator skills. In these environments, operators were required to complete specially designed tasks as quickly as possible. Comparison of pre- and post-training evaluation data was used to assess the effectiveness of the training. In the first evaluation, repetitive digging and dumping skills were measured. Operators excavated sand in a circle around the excavator and dumped it into eight pipes. Operators dumped sand into pipes of successively decreasing diameter, increasing the operator's required positional accuracy and the difficulty of controlling the end effector positioning. In the second evaluation, simultaneous movement of the cab and other joints was evaluated. Operators created four trenches in front of the excavator, dumped sand into four diagonally arranged trenches of variable width, and then flattened the surfaces of these diagonal trenches. Operators generally expressed the difficulty of operating the excavator using the joystick along the diagonal lines. In the third and fourth evaluation conditions, fine motor coordination was measured. Subjects removed sand along prescribed slopes, including four upward-angled slopes and three downward-angled slopes with different slope angles. Operators also noted that the linear movement associated with following a slope was a difficult movement to perform when using a traditional joystick interface.

[0058] To evaluate the operator's skill, the inventors considered several performance indices. For example, the cycle time per task was considered. The cycle time was defined as the duration from the start of the excavation movement to the completion of the dump movement per repeated trial. In cases of errors during operation, the duration of the error was excluded. Data corresponding to malfunctions, i.e., direct control of the bucket tip over the excavation and dump targets and hitting an obstacle, were also excluded prior to analysis.

[0059] As described above, the experiment involved an operator moving the WS (i.e., control arm) to control a simulated excavator and collecting data during operation. The experimental results indicate that novice or casual operators can more quickly learn the machine's operation and more easily achieve reasonable results. Specifically, Figure 4 shows the experimental results over the cycle time in the first evaluation for an operator using the WS.

[0060] Figure 4 shows the performance of a total of 12 inexperienced operators after seven days of training, performing a simple excavation-to-disposal task on Day 0. Six inexperienced operators used a WS, and six inexperienced operators used a JS. The figure compares the average cycle times of the six inexperienced subjects using a WS with six novices pre-training and after seven days of training. We noted that the novices using a JS initially demonstrated subpar performance but significantly improved after training as they built their internal mental maps. Specifically, subjects using a joystick improved and reduced their cycle times by a statistically significant average of 40% after training. Conversely, the inexperienced operators using a WS performed better than the operators using a JS on Day 0, and their performance was nearly identical to that of a second group of operators trained with a JS for seven days. Furthermore, cycle times using a WS did not change significantly with training (i.e., the 6% reduction was not found to be statistically significant).

[0061] The above results suggest that WS may be a viable alternative for casual (i.e., untrained) operators to achieve reasonable performance from the beginning. In addition, the average cycle time before training with WS is close to the cycle time after training with a joystick. This result may indicate that using WS can potentially help operators provide high operability (i.e., skill) even when they are untrained. This further indicates that WS can potentially be implemented in actual excavators to enable inexperienced operators to start work after basic instructions in actual field situations.

[0062] FIG. 5 illustrates a schematic embodiment of a control arm 200 and machine 100. In the depicted embodiment, control arm 200 includes multiple linkages 210-216, as described above, or other suitable arrangements, to control the allowable movement of the control arm. Multiple sensors 226A-226D may be configured to sense the configuration of the control arm, including, for example, the configuration of linkages 210-216. For example, sensors 226A-226D may sense the angle between one or more sets of linkages. Sensors 226A-226D may also output one or more corresponding signals to a controller 228 of control arm 200, which may include one or more processors that may be configured to implement any of the methods disclosed herein related to determining the configuration of the control arm and corresponding commands for operation of machine 100. Control arm 200 also includes a secondary control 220 that may output control signals to controller 228, as described above in detail, to control one or more secondary movements of machine 100. It should be understood that the depicted control arm 200 may also include any of the other components disclosed herein in connection with any other embodiment of the control arm, such that the disclosure is not so limited.

[0063] It should be understood that the depicted schematic embodiment of machine 100 may also include any suitable components of any of the embodiments of machine 100 disclosed herein. For example, in instances where the machine also includes an optional exemplary control interface, machine 100 may optionally include one or more of operator interface 114, joystick 116, and / or pedals / levers 118 depicted in the figures. The machine may also include a controller 110 including one or more processors, and suitable end effectors, linkages, or other suitable motion systems including one or more actuators 112A-112C coupled to controller 110 to control the movement of the machine. For example, the machine is depicted as including a bucket 102, a boom 104, and an arm 106 rotatably coupled to one another, and actuators 112A-112C configured to control the movement of the depicted structure. The actuators 112A-112C may be operatively coupled to a controller 110, which may control the movement of the actuators 112A-112C, and therefore the movement of various portions of the machine 100. The machine may also include one or more sensors 101A-101C, which may be configured to sense the configuration of one or more linkages, joints, movement stages, or other moving parts of the machine, such as the depicted bucket 102, boom 104, and arm 106. The one or more sensors 101A-101C may output corresponding signals to the controller 110, which may be used to control the operation of the machine 100.

[0064] During operation, controller 228 of control arm 200 may communicate with controller 110 of machine 100. Depending on the embodiment, determining the configuration of control arm 100, transforming between different reference frames, and determining commands for machine 100 may be performed on either controller. Correspondingly, sensed signals in various reference frames, determined configurations, and / or commands may be transmitted between the two depicted controllers 110 and 228 in any of the disclosed reference frames using any suitable wired or wireless communication protocol. Correspondingly, control arm 200 may be located remotely from machine 100 in some embodiments. However, in some embodiments, control arm 200 may be integrated with machine 100. In such embodiments, controller 110 and controller 228 may be combined with each other such that the depicted connections and functions may be implemented by a single controller including one or more processors.

[0065] The above-described embodiments of the techniques described herein can be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computing device or distributed among multiple computing devices. Such processors may be implemented as integrated circuits having one or more processors within the integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor may be implemented within custom circuitry such as an ASIC, or semi-custom circuitry resulting from constructing a programmable logic device. As a still further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one core or a subset of those cores can constitute a processor. However, a processor may be implemented using circuitry in any suitable format.

[0066] Furthermore, it should be understood that a computing device including one or more processors may be embodied in any of several forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computing device may be embedded within a device that is not generally considered a computing device, but with suitable processing capabilities, including a personal digital assistant (PDA), a smartphone, a tablet, or any other suitable portable or fixed electronic device.

[0067] A computing device may also have one or more input / output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include a display screen for visual presentation of output and a speaker or other sound-generating device for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, individual buttons, and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computing device may receive input information through speech recognition or in other audible formats.

[0068] Referring to FIG. 6 , an exemplary system for implementing aspects of the present disclosure includes a general-purpose computing device in the form of a computer 610 or other suitable computing device. For example, the depicted computing device may be used to control one or more operations of a machine (e.g., an excavator) by implementing any of the methods disclosed herein. Components of the computer 610 may include, but are not limited to, a processing unit 620, a system memory 630, and a system bus 621 that couples various system components, including the system memory, to the processing unit 620. The system bus 621 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, also known as a Mezzanine bus.

[0069] Computer 610 typically includes a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computer 610, including both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computer 610. Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0070] The system memory 630 includes computer storage media in the form of volatile and / or nonvolatile memory such as read-only memory (ROM) 631 and random access memory (RAM) 632. A basic input / output system 633 (BIOS), containing the basic routines that help to transfer information between elements within the computer 610, such as during start-up, is typically stored in ROM 631. RAM 632 typically contains data and / or program modules that are immediately accessible to and / or presently being operated on by the processing unit 620. By way of example, and not limitation, FIG. 6 illustrates operating system 634, application programs 635, other program modules 636, and program data 637.

[0071] Computer 610 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, Figure 6 illustrates a hard disk drive 641 that reads from or writes to non-removable, non-volatile magnetic media, a magnetic disk drive 651 that reads from or writes to a removable, non-volatile magnetic disk 552, and an optical disk drive 655 that reads from or writes to a removable, non-volatile optical disk 656, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid-state RAM, solid-state ROM, and the like. The hard disk drive 641 is typically connected to the system bus 621 through a non-removable memory interface such as interface 640, and the magnetic disk drive 651 and optical disk drive 655 are typically connected to the system bus 621 by a removable memory interface such as interface 650.

[0072] The drives and their associated computer storage media, discussed above and illustrated in FIG. 6, provide storage of computer-readable instructions, data structures, program modules, and other data for computer 610. In FIG. 6, for example, hard disk drive 641 is illustrated as storing operating system 644, application programs 645, other program modules 646, and program data 647. Note that these components can be either the same as or different from operating system 644, application programs 645, other program modules 646, and program data 647. Operating system 644, application programs 645, other program modules 646, and program data 647 are given different numbers here to illustrate that, at a minimum, they are different copies. A user can enter commands and information into computer 510 through input devices such as keyboard 662 and pointing device 661, commonly referred to as a mouse, trackball, or touchpad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are connected to the processing unit 620 through a user input interface 660, which is often coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, a game port, or a universal serial bus (USB). A monitor 691 or other type of display device is also connected to the system bus 621 via an interface, such as a video interface 690. In addition to a monitor, computers may also include other peripheral output devices such as speakers 697 and printer 696, which may be connected through an output peripheral interface 695.

[0073] The computer 610 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 680. The remote computer 680 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 610, although only a memory storage device 681 is illustrated in Figure 6. The logical connections depicted in Figure 6 include a local area network (LAN) 671 and a wide area network (WAN) 673, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet.

[0074] When used in a LAN networking environment, the computer 610 is connected to the LAN 671 through a network interface or adapter 670. When used in a WAN networking environment, the computer 610 typically includes a modem 672 or other means for establishing communications over the WAN 673, such as the Internet. The modem 672, which may be internal or external, may be connected to the system bus 621 via the user input interface 660 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 610, or portions thereof, may be stored in a remote memory storage device. By way of example, and not limitation, FIG. 6 illustrates remote application programs 685 as residing on memory device 681. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between computers may be used.

[0075] Such computing devices may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0076] Also, the various methods or processes outlined herein may be coded as software that is executable on one or more processors employing any of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.

[0077] In this regard, the embodiments described herein may be embodied as a computer-readable storage medium (or media) (e.g., computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tape, flash memory, RAM, ROM, EEPROM, circuitry in a field programmable gate array or other semiconductor device, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments discussed above. As is evident from the foregoing examples, a computer-readable storage medium may retain information and provide computer-executable instructions in a non-transitory form for a sufficient period of time. Such computer-readable storage medium or media may be portable such that the program or programs stored thereon may be loaded onto one or more different computing devices or other processors to implement various aspects of the present disclosure as discussed above. As used herein, the term "computer-readable storage medium" encompasses only non-transitory computer-readable media that may be considered an article of manufacture (i.e., an article of manufacture) or a machine. Alternatively, or in addition, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0078] The terms "program" or "software" are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computing device or other processor to implement various aspects of the present disclosure as discussed above. Additionally, it should be understood that in accordance with one aspect of the present embodiments, one or more computer programs that, when executed, perform the methods of the present disclosure need not reside on a single computing device or processor, but may be distributed in a modular manner among several different computers or processors to implement various aspects of the present disclosure.

[0079] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0080] The embodiments described herein may be embodied as methods, for which examples are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be organized in which acts are performed in a different order than that shown, and may include performing some acts simultaneously, even though shown as sequential acts in the illustrative embodiments.

[0081] Additionally, some actions are described as being taken by a "user," "operator," or similar terms. It should be understood that a "user" or "operator" need not be a single individual, and that in some embodiments, actions attributed to a "user" may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0082] While the present teachings have been described in conjunction with various embodiments and examples, it is not intended that the present teachings be limited to such embodiments or examples. Rather, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those skilled in the art. Accordingly, the foregoing description and drawings are by way of example only.

Claims

1. 1. A system comprising: a control arm including a plurality of linkages, with one or more joints disposed between the plurality of linkages; a plurality of sensors configured to sense a configuration of the one or more joints of the control arm; at least one processor, said at least one processor comprising: obtaining a commanded pose for an end effector of a machine in world space, the commanded pose being in controller space, the commanded pose being obtained from the plurality of sensors, and the controller space being linearly mapped to the world space of the machine; transforming the commanded pose from the controller space to a commanded machine configuration in a machine space of the machine; controlling movement of the end effector based at least in part on the commanded machine configuration; and at least one processor configured to perform A system comprising:

2. The system of claim 1 , further comprising the machine, the end effector, and an arm connected to the end effector, the arm configured to control an attitude of the end effector.

3. The system of claim 2 , wherein the machine is located remotely from the control arm.

4. The system of claim 3 , wherein the at least one processor is further configured to remotely operate the machine.

5. The system of claim 2 , wherein the commanded machine configuration is a commanded joint configuration of a plurality of joints of the arm of the machine.

6. 10. The system of any one of the preceding claims, further comprising one or more motors and / or springs associated with the control arm, the one or more motors and / or springs configured to support at least a portion of the weight of the control arm.

7. The system of any one of claims 1-5, wherein obtaining the commanded pose comprises sensing angles of the one or more joints of the control arm using the plurality of sensors.

8. 8. The system of claim 7, wherein the sensed angles are equal to commanded angles of a plurality of joints of the mechanical arm.

9. 8. The system of claim 7, wherein the sensed angles are proportional to commanded angles of a plurality of joints of the mechanical arm.

10. The system of any one of claims 1-6, wherein obtaining the commanded pose comprises sensing displacement of the one or more joints of the control arm using the plurality of sensors.

11. The system of claim 10 , wherein the sensed displacements are equal to commanded displacements of a plurality of joints of the mechanical arm.

12. The system of claim 10 , wherein the sensed displacement is proportional to commanded displacements of a plurality of joints of the mechanical arm.

13. 10. The system of claim 1, wherein transforming the commanded pose in controller space to the commanded machine configuration comprises transforming the commanded pose using a Jacobian matrix.

14. 10. The system of any one of the preceding claims, wherein the at least one processor is configured to: obtain a yaw angle of the control arm using the plurality of sensors; and control yaw movement of the machine based at least in part on the obtained yaw angle.

15. 1. A method for controlling operation of a machine including an end effector, the method comprising: obtaining a commanded pose for the end effector in world space, the commanded pose being in controller space, the controller space being linearly mapped to the world space of the machine; remotely operating the machine based at least in part on the commanded pose to control movement of the end effector; and A method comprising:

16. 16. The method of claim 15, further comprising transforming the commanded pose into a commanded machine configuration in machine space, and teleoperating the machine comprises teleoperating the machine based on the commanded machine configuration to control movement of the end effector.

17. 1. A method for controlling operation of a machine including an end effector, the method comprising: obtaining a commanded pose for the end effector in world space, the commanded pose being in controller space, the controller space being linearly mapped to the world space of the machine; Transforming the commanded pose into a commanded machine configuration in machine space of the machine; controlling movement of the end effector based on the commanded machine configuration; A method comprising:

18. The method of claim 17 , wherein obtaining the commanded pose comprises obtaining the commanded pose from a world-space interface controller.

19. The method of claim 18 , wherein the machine is located remotely from the world-space interface controller.

20. The method of claim 18 , further comprising remotely operating the machine using the world-space interface controller.

21. The method of claim 17 , wherein the machine is a simulated machine.

22. The method of claim 17 , wherein the machine includes an arm associated with the end effector, the arm including a plurality of joints.

23. The method of claim 22 , wherein the commanded machine configuration is a commanded joint configuration of the plurality of joints.

24. The method of any one of claims 17-23, wherein acquiring the commanded pose includes an operator moving a control arm to a commanded control arm configuration.

25. 25. The method of claim 24, wherein moving the control arm includes moving multiple links of the control arm to the commanded control arm configuration.

26. The method of any one of claims 17-23, wherein obtaining the commanded pose comprises sensing the angle of one or more joints of the control arm.

27. 27. The method of claim 26, wherein the sensed angles are equal to commanded angles of a plurality of joints of the mechanical arm.

28. The method of any one of claims 17-23, wherein obtaining the commanded pose comprises sensing displacement of the one or more joints of the control arm.

29. 30. The method of claim 28, wherein the sensed displacements are equal to commanded displacements of a plurality of joints of the mechanical arm.

30. 27. The method of claim 26, wherein the sensed angles are proportional to commanded angles of a plurality of joints of the mechanical arm.

31. The method of any one of claims 17-30, further comprising transforming the commanded pose in the controller space to the commanded machine configuration using a Jacobian matrix.

32. 32. The method of any one of claims 17-31, wherein the method further comprises sensing a yaw angle of the control arm and controlling yaw movement of the machine based at least in part on the sensed yaw angle.

33. 33. A non-transitory computer readable memory comprising instructions that, when executed by at least one processor, perform the method of any one of claims 17-32.

Images