Industrial machine remote operating system and associated apparatus and method

Abstract

The present technology generally relates to an industrial machine remote operating system, apparatus and method. In some embodiments, an industrial machine remote operation system (110) configured in accordance with an embodiment of the present technology includes a control interface (330) including one or more elements associated with remote operation of an industrial machine (100, 200). In at least some embodiments, the control interface (330) may include one or more indicators associated with motion and / or orientation of the machine. Additionally or alternatively, the control interface may include one or more indicators associated with movement and / or orientation of a component of the machine, such as a machine work tool (202).

Description

Technical Field

[0001] The present technology relates to apparatus, systems, and methods for remote operation of industrial machines. Background Art

[0002] Remote operators, such as those operating industrial machines remotely based on a live video feed, may face many difficulties that do not arise when operating these machines in the field. For example, a live operator can directly sense changes in the orientation of the machine, while the remote operator typically relies on the instrument panel to relay this information. As a result, the remote operator may lack or have a reduced sense of the orientation of the machine relative to the machine's surroundings. Additionally, the video feed that some remote operators use to remotely operate the machine may not adequately convey to the remote operator a sense of the machine's depth or relative position, which may reduce the remote operator's perception of the machine's position relative to the surroundings.

[0003] Current teleoperation methods fail to adequately address these issues. For example, U.S. Pat. No. 4,855,822 to Narendra et al. (the "'822 patent") purports to describe "[m]ethods and apparatus ... for driving a vehicle from a remote control station achieving tele-operation of the vehicle." (Id., in the Abstract). However, the methods and apparatus described in the '822 patent are narrowly described in the context of use in a "hostile environment, such as a battlefield" (Id., at [column 4, ll.50-51]). Additionally, the '822 patent describes that "a specific object of the present invention [is] to provide a highly optimized data compression system which enables narrow bandwidth television image data transmission while providing a high resolution video display of instantaneous position information to an operator of a robotic vehicle," (ibid., at [col. 4, ll. 30-35]), and does not teach or suggest a method or apparatus for improving a remote operator's sense of a machine's orientation relative to the machine's surroundings and / or improving a remote operator's perception of a machine's position relative to the surroundings. Summary of the invention

[0004] In some embodiments, the present technology may include a method for improving an operator's awareness of an operating environment when remotely operating an industrial machine. The method may include at least one of the following: (i) displaying a video feed from a camera of the industrial machine via a display screen of a remote operating system; (ii) receiving first telematics data associated with remote operation of the industrial machine at a first time and via the remote operating system; (iii) generating an industrial machine control interface based at least in part on the first telematics data, wherein generating the industrial machine control interface includes displaying the industrial machine control interface on the display screen over the video feed and aligned with at least a portion of the industrial machine; (iv) receiving second telematics data associated with remote operation of the industrial machine at a second time and via the remote operating system; and (v) updating the industrial machine control interface based at least in part on the second telematics data, wherein updating the industrial machine control interface includes maintaining alignment of the industrial machine control interface with the portion of the industrial machine.

[0005] Additionally or alternatively, the present technology may include a system for improving an operator's awareness of an operating environment when remotely operating an industrial machine. The system may include a remote operating system including a display screen; one or more processors; and one or more memory devices. The one or more memory devices may store instructions that, when executed by the one or more processors, cause the one or more processors to: (i) display a video feed from a camera of the industrial machine via the display screen; (ii) receive first telematics data associated with remote operation of the industrial machine at a first time and via the remote operating system; (iii) generate an industrial machine control interface based at least in part on the first telematics data, wherein generating the industrial machine control interface includes displaying the industrial machine control interface on the display screen in alignment with at least a portion of the industrial machine over the video feed; (iv) receive second telematics data associated with remote operation of the industrial machine at a second time and via the remote operating system; and / or (v) update the industrial machine control interface based at least in part on the second telematics data, wherein updating the industrial machine control interface includes maintaining alignment of the industrial machine control interface with the portion of the industrial machine.

[0006] In these and other embodiments, the technology may include one or more non-transitory computer-readable media storing computer-executable instructions. When executed by one or more processors, the instructions may cause the one or more processors to perform operations including at least one of: (i) displaying a video feed from a camera of the industrial machine via a display screen of a remote operating system; (ii) receiving first telematics data associated with remote operation of the industrial machine at a first time and via the remote operating system; (iii) generating an industrial machine control interface based at least in part on the first telematics data, wherein generating the industrial machine control interface includes displaying the industrial machine control interface on the display screen in alignment with at least a portion of the industrial machine over the video feed; (iv) receiving second telematics data associated with remote operation of the industrial machine at a second time and via the remote operating system; and (v) updating the industrial machine control interface based at least in part on the second telematics data, wherein updating the industrial machine control interface includes maintaining alignment of the industrial machine control interface with the portion of the industrial machine.

[0007] Other aspects of the technology will appear below.The features described herein may be used alone or together, or in various combinations of one or more of them. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Schematic diagram of a machine and a remote control station, each configured according to an embodiment of the present technology.

[0009] Figure 2A-2C is a perspective view of a machine configured according to an embodiment of the present technology.

[0010] Figure 3A-7 Various machine control interface elements are shown in accordance with embodiments of the present technology.

[0011] Figure 8 A flowchart illustrating a method for remote operation of a machine according to an embodiment of the present technology.

[0012] Fig. 9 A block diagram showing an overview of a device in which embodiments of the present technology may operate.

[0013] Fig.10 is a block diagram illustrating an overview of an environment in which embodiments of the present technology may operate.

[0014] Fig.11 is a block diagram illustrating elements that may be used in a machine remote operation system according to an embodiment of the present technology. DETAILED DESCRIPTION

[0015] The present technology generally relates to industrial machine remote operation systems and associated devices and methods. In some embodiments, an industrial machine remote operation system configured according to embodiments of the present technology includes a control interface, the control interface including one or more elements associated with remote operation of an industrial machine. In at least some embodiments, the control interface may include one or more indicators associated with the movement or movement of the machine (e.g., in a forward and / or backward direction). Additionally or alternatively, the control interface may include one or more indicators associated with the orientation of the machine (e.g., the roll, pitch, and / or yaw of the machine). In these and other embodiments, the control interface may include one or more indicators associated with the movement or movement of at least one of the machine components. In at least some embodiments, for example, a component may include a working tool of the machine. Thus, the control interface may include one or more indicators associated with the movement of the working tool (e.g., the height of the working tool and / or the position of the working tool relative to the ground).

[0016] Additionally or alternatively, the control interface may include one or more indicators associated with the orientation of the work tool (e.g., pitch, roll, or tilt and / or yaw of the work tool). Any of the indicators may be generated based on data collected from one or more sensors associated with the machine and / or its (one or more) components (e.g., telematics data) and may be displayed to the remote operator together with one or more video feeds from (one or more) cameras mounted to the industrial machine. In at least some embodiments, one or more of the indicators may correspond to one or more components of the machine and be associated with one or more indicators, and each of the indicators may be aligned with the associated component when displayed on the video feed. In some aspects, it is expected that an industrial machine teleoperation system configured according to an embodiment of the present technology will improve the precision, accuracy, and / or safety with which the remote operator can remotely operate the machine. In at least some embodiments, for example, it is expected that the control interface and / or its elements described herein will improve the remote operator's perception of the machine orientation and surrounding environment.

[0017] Those skilled in the art will appreciate that (i) the technology may have advantages over those described herein. Figure 1-11 More embodiments than those shown and described, and (ii) the technology may be used without reference herein Figure 1-11 The embodiments described are practiced without the need for several details.

[0018] A. Embodiments of selected industrial machine remote operation systems and associated devices and methods

[0019] Figure 11 is a schematic diagram of a machine 100 and a remote control station 110 (“remote control system 110”, “remote operator station 110”, “remote operating device 110”, etc.) configured according to embodiments of the present technology. The machine 100 can be operably associated (e.g., paired) with the remote control station so that an operator 50 (e.g., a teleoperator, a human operator, etc.) of the remote control station 110 can remotely operate the machine 100 via the remote control station 110. In the illustrated embodiment, for example, the machine 100 is in a first operating environment 10, and the remote control station 110 and the operator 50 are in a second operating environment 20 that is remote from and / or otherwise different from the first operating environment 10.

[0020] The machine 100 may be a truck, a crawler tractor, an excavator, a wheel loader, a front-end loader, a motor grader, a vehicle, an industrial machine, and / or any other suitable machine. The machine 100 includes one or more components 102, one or more sensors, one or more cameras 105, a transceiver 106, and a controller 108. Each component 102 may operate / work independently and / or together to perform activities associated with the machine 100. The components 102 may include one or more work tools, drive components (e.g., tires, wheels, pedals, tracks, etc.), an engine, a transmission, a powertrain, an actuator (e.g., a hydraulic actuator, a mechanical actuator, a solenoid, etc.), a battery or battery array, a post-processing system, one or more input devices (e.g., a steering component), a hydraulic system, and / or any other suitable components. Each sensor 104 may be connected to (e.g., operatively and / or communicatively coupled to) one or more components 102 and / or machines 100 and / or otherwise arranged to detect data (e.g., telematics data) associated with one or more components and / or machines, for example, to monitor the operation (e.g., position, orientation, movement, motion, etc.) of the components 102 and / or machines 100. Individual sensors in the sensors 104 may include position sensors, temperature sensors, pressure sensors, accelerometers, fuel gauges, strain gauges, machine payload sensors, and / or any other suitable sensors. The sensors 104 may include or may be (e.g., operatively, communicatively, etc.) connected to a machine transceiver 106. The machine transceiver 106 is configured to transmit data and / or video 30 detected by the sensors 104 and / or cameras 105 to the remote control station 110. Machine controller 108 may be in communication with remote control station 110 and operably coupled to one or more components 102 such that machine controller 108 may receive one or more command signals 40 from remote control system 110 (eg, to remotely operate machine 100 ).

[0021] The remote control station 110 may include a transceiver 112, one or more processing devices 114, one or more display devices 116, and one or more input devices 118. The transceiver 112 may communicate with the machine 100 (e.g., the machine transceiver 106 and / or the machine controller 108) so that the transceiver 112 may receive data 30 from the machine transceiver 106 and / or transmit command signals 40 to the machine controller 108. Each of the processing devices 114 may process the data 30 received from the machine 100 and / or otherwise perform functions associated with the operation of the remote control station 110. The display device 116 may display (e.g., to the operator 50) the video data 30 received from the one or more cameras 105. Each of the display devices 116 may include an LCD display, an OLED display, a mini-LED display, a VR headset, a computer display, a mobile phone display, and / or any other suitable display device. The input devices 118 may correspond to the component 102 and may each receive input from the operator 50 when the operator 50 remotely controls the machine 100. Each of the input devices 118 may include a joystick, a keyboard, a mouse, a touch screen, a steering wheel, one or more pedals, one or more levers, one or more switches, one or more buttons, and / or any other suitable input device. Figure 9-11 The remote control station 110 and the environment in which it may operate are described in greater detail.

[0022] Although reference Figure 1 100, but in other embodiments, the present technology may include more machines. In at least some embodiments, for example, the present technology may include at least two, three, four, five, six, seven, eight, nine, ten, or more machines, each of which may be configured to be substantially similar, the same, or different from machine 100 (e.g., including component(s) 102, sensor(s) 104, and / or camera(s) 105).

[0023] Figure 2A-2C is a perspective view of a machine 200 configured according to an embodiment of the present technology. The machine 200 may include a machine 200 that is structurally and / or functionally similar to the above referenced Figure 1 The corresponding features of the described machine 100 are at least substantially similar or identical to some features. Figure 2A-2C In the embodiment, the machine 200 is shown as a front loader, but in other embodiments, the machine 200 can be any other suitable type of machine, including the machine described above with reference to Figure 1 Any machine / industrial machine described. Reference Figure 2A, machine 200 may define a machine or chassis plane M (shown in phantom) in which, for example, machine 200 (e.g., one or more machine tires, such as Figure 2A ) contacts the surrounding ground G. Therefore, it is desirable that the machine plane M remain at least substantially or completely aligned (eg, coplanar) with the ground G during operation.

[0024] Additionally, the machine 200 may include one or more machine axes. Each axis may be used to define a relative position of the machine to a true ground plane G. T The position, rotation, orientation and / or movement of the machine axis (shown in dashed lines) may include the machine roll axis R M , Machine yaw axis Y M and / or machine pitch axis P M . The real ground plane G T Thus, when the ground G is horizontal, the machine plane M and / or the ground plane G may be aligned with the real ground plane G. T The machine plane M and / or the ground plane G may be substantially or essentially parallel / coplanar, and when the ground plane G is inclined (e.g., tilted, uneven, not level, etc.), the machine plane M and / or the ground plane G may be substantially or essentially parallel / coplanar with respect to the true ground plane G. T In other embodiments, the ground plane G may represent a slope or tilted ground, and / or have any other suitable orientation. In operation, the machine 200 and / or at least a portion thereof may (i) be oriented relative to a true ground plane G. T Around the machine rolling axis R M rolling (e.g., left or right), (ii) relative to the true ground plane G T Around the machine pitch axis P M (e.g., up or down) rotation or pitch, and / or (iii) relative to the true ground plane G T Around the machine yaw axis Y M Rotation or yaw (e.g., left or right).

[0025] refer to Figure 2B In the illustrated embodiment, the machine 200 includes a work tool 210 (e.g., a component of the machine 200). The work tool 210 may include one or more work tool axes, each of which may be similar to the one or more work tool axes described above. Figure 2A The machine axes described in detail are at least generally similar or identical. The work tool axes may include a work tool roll or pitch axis R WT , working tool yaw axis Y WT and / or work tool pitch axis P WTEach work tool axis can be used to define the position, rotation, orientation and / or movement of work tool 210 and / or one or more parts thereof relative to ground plane G and / or machine 200. In operation, work tool 210 and / or at least a part thereof can (i) be positioned relative to a true ground plane G and / or T and / or the machine 200 rolling about the work tool axis R WT roll / pitch (e.g., left or right), (ii) relative to the true ground plane G T and / or the machine 200 about the work tool pitch axis P WT (e.g., up or down) rotation or pitch, and / or (iii) relative to the true ground plane G T and / or the machine 200 about the work tool yaw axis Y WT Rotation or yaw (e.g., left or right).

[0026] refer to Figure 2C , one or more cameras 205 may be coupled to machine 200. Each of cameras 205 may have a field of view 207 and may be positioned such that one or more sides of machine 200 and / or at least a portion of the machine's surroundings (e.g., the ground G and / or (one or more) objects located thereon) are within field of view 207, such that each of cameras 205 may be used by a remote operator to remotely operate machine 200. In the illustrated embodiment, for example, the machine 200 includes: (i) a first (e.g., front-facing) camera 205a positioned to image at least a portion of a first (e.g., front) side 201a of the machine 200 and a corresponding first portion of the machine environment, (ii) a second (e.g., rear-facing or back-facing) camera 205b positioned to image at least a portion of a second (e.g., rear or back) side 201b opposite the first side 201a and a corresponding second portion of the machine environment, (iii) a third (e.g., left-facing) camera 205c positioned to image at least a portion of a third (e.g., left) side 203a of the machine 200 between the first side 201a and the second side 201b and a corresponding third portion of the machine environment, and a fourth (e.g., right-facing) camera 205d positioned to image at least a portion of a fourth (e.g., right) side 203b of the machine 200 opposite the third side 203a and a corresponding fourth portion of the machine environment. In other embodiments, more or fewer cameras 205 may be coupled to machine 200 and / or individual ones of cameras 205 may be positioned to image these and any other suitable side(s) of the machine and / or portion(s) of the machine environment.

[0027] Figure 3A-7Various industrial machine control interface elements 330 (“control interface 330 ,” “machine control interface 330 ,” “industrial machine control interface 330 ,” “indicator 330 ,” etc.) are shown in accordance with embodiments of the present technology. Figures 3A-3C For clarity, the machine 200 and control interface 330 are shown in a top view. However, in practice, it should be appreciated that the control interface 330 may be shown on / over a video feed obtained from a camera positioned to image one side (e.g., first side 201a, second side 201b, third side 203a, fourth side 203b, etc.) of the machine 200 (e.g., as described above with respect to Figure 2C For example, when viewed on the display, the control interface 330 may be at least partially aligned with one or more of the sides, components, and / or one or more portions of the machine. Additionally or alternatively, when viewed on the display, the control interface 330 may be positioned / aligned as described below, but as opposed to the machine plane M and / or the ground plane G ( Figure 2A ) onto the machine plane M and / or the ground plane G. As described in detail below, it is desirable to project the elements of the control interface 330 onto the machine plane M and / or the ground plane G ( Figure 2A ) will better indicate to the remote operator the position, orientation and / or movement of the machine and its components relative to the machine's surroundings, which can improve the remote operator's understanding of the machine relative to the environment surrounding the machine (e.g., ground G; Figure 2A-2C ) and can improve the precision, accuracy, and / or safety with which a remote operator can remotely operate a machine. In addition, the control interface 330 described herein can be generated, updated, and / or otherwise determined based on data (e.g., telematics data) associated with the machine and / or machine components and obtained from sensors (e.g., the previously described sensors 104). In addition, one or more of the different features of the control interface 330 of the present technology can be combined together and / or omitted.

[0028] refer to Figure 3A , the control interface 330 may include one or more machine forward path indicators 332, one or more machine reverse path indicators 334, one or more drive component path indicators 336, one or more machine drive component indicators 338, one or more machine side reference indicators 340, and / or one or more machine side offset indicators 342. In the illustrated embodiment, each of the above indicators includes a first (e.g., left) element indicated by the letter “a” (e.g., first / left machine forward path indicator 332). a ) and a second (e.g., right) element indicated by the letter "b" (e.g., second / right machine forward path indicator 332 b ). As mentioned below about Figure 3BIn more detail, the control interface 330 can be dynamic. For example, various elements of the control interface 330 can be rotated, angled, bent, or otherwise updated in real time based on the movement and / or reorientation of the machine 200.

[0029] The machine forward path indicators 332a-b may extend outward / forward from the front 201a of the machine 200 a distance D. F . Distance D F may be any suitable distance and may be determined by an operator of machine 200 and / or automatically based at least in part on the forward travel speed of the machine, as described with respect to Figure 3C 33. Optionally, as shown in the illustrated embodiment, the left machine forward path indicator and / or the right machine forward path indicator 332a-b may extend at least partially between the front portion 201a of the machine 200 and the machine drive component indicator 338. Each forward machine path indicator 332a-b may be aligned with a corresponding outer edge of one of the drive components of the machine 200. In the illustrated embodiment, for example, the left machine forward path indicator 332a is aligned with the left outer edge of the left front tire of the machine 200, and the right machine forward path indicator 332b is aligned with the right outer edge of the right front tire of the machine 200. Thus, each machine forward path indicator 332a-b may be associated with or represent a forward travel path of the machine 200, such that the machine 200 may generally or substantially travel along the machine forward path indicators 332a-b when the operator moves the machine 200 forward.

[0030] The machine reverse path indicators 334a-b may extend outwardly / rearwardly from the rear portion 201b of the machine 200 a distance D R . Distance D R may be any suitable distance and may be determined by an operator of machine 200 and / or automatically based at least in part on the forward travel speed of the machine, as described with respect to Figure 3B Each of the reverse machine path indicators 334a-b may be aligned with a corresponding outer edge of one of the drive components of machine 200. In the illustrated embodiment, for example, the left machine reverse path indicator 334a is aligned with the left outer edge of the left rear tire of machine 200, and the right machine reverse path indicator 334b ​​is aligned with the right outer edge of the right rear tire of machine 200. Thus, each of the machine reverse path indicators 334a-b may be associated with or represent a reverse travel path of machine 200, such that machine 200 may generally or substantially follow the machine reverse path indicators 334a-b when an operator causes machine 200 to move in reverse.

[0031] The drive component path indicators 336a-b may extend in one or both directions in the direction of travel of the machine (e.g., forward and / or rearward). Each drive component path indicator 336a-b may be at least generally parallel to or may be parallel to the corresponding machine forward and / or reverse path indicator 332, 334 and / or may be located a distance D inward from the corresponding machine forward and / or reverse path indicator. T In the illustrated embodiment, for example, the left drive component path indicator 336a is positioned a distance D inward from the left machine forward path indicator 332a and the left machine reverse path indicator 334a. T and the right drive component path indicator 336b is positioned a distance D inwardly from the right machine forward path indicator 332b and the right machine reverse path indicator 334b. T Distance D T The width of the drive component path indicator 336a-b may be at least substantially similar or identical to the width of the associated drive component (e.g., a tire of the machine 200). In the illustrated embodiment, the drive component path indicator 336a-b includes a plurality of line segments, each line segment having a length D L and interval D S Length D L The length of the associated drive component may be at least substantially similar or identical. S The length D may be deducted, for example, from the perimeter / circumference of the associated drive component. L are at least substantially similar or identical, such that the interval D S The distance traveled by the machine during one complete rotation of the associated drive component may correspond to a distance traveled by the machine during one complete rotation of the associated drive component. In other embodiments, one or more drive component path indicators 336a-b may include a line extending continuously from the front 201a and / or rear 201b of the machine 200 (e.g., Figure 3B ), or have any other suitable appearance.

[0032] The drive component leading edge indicators 338a-b can be at least partially aligned with (e.g., parallel to, coplanar with, etc.) the leading edge of the corresponding drive component. In the illustrated embodiment, for example, the left drive component leading edge indicator 338a is coplanar with the leading edge of the front left tire, and the right drive component leading edge indicator 338b is coplanar with the leading edge of the front left tire.

[0033] The machine side reference indicators 340a-b may be at least generally aligned (e.g., co-linear) with corresponding sides of the machine 200. In the illustrated embodiment, for example, the machine left side reference indicator 340a is co-linear with the left side 203a of the machine 200, and the machine right side reference indicator 340b is co-linear with the right side 203b of the machine 200.

[0034] The machine-side offset indicators 342a-b may be or may be at least generally parallel to the corresponding machine-side reference indicators 340a-b and the corresponding side of the machine 200. In the illustrated embodiment, for example, the machine-left offset indicator 342a is parallel to the machine-left reference indicator 340a and the left side 203a of the machine 200, and the machine-right offset indicator 342b is parallel to the machine-right reference indicator 340b and the right side 203b of the machine 200. In addition, each of the machine-side offset indicators 342a-b may be spaced apart from (e.g., offset from) the corresponding machine-side reference indicator 340a-b. In the illustrated embodiment, for example, the machine-left offset indicator 342a is a distance D from the machine-left reference indicator 340a. O1 , and the machine right side offset indicator 342b is a distance D from the machine right side reference indicator 340b O2 . Distance D O1 , D O2 The lengths D of the two machine-side offset indicators 342a-b may be the same or different. One or both are determined by the remote operator of the machine. In the illustrated embodiment, the lengths D of the two machine-side offset indicators 342a-b are M The length D is at least substantially equal to the length of the machine 200 (e.g., between the front side 201a and the rear side 201b) or equal to the length of the machine. M It may be greater or less than the length of machine 200, and / or may be any other suitable length.

[0035] refer to Figure 3B, various elements of the control interface 330 may be dynamic. In the illustrated embodiment, for example, the front portion 201a of the machine 200 has been rotated (e.g., articulated) by a first angle A1 relative to the rear portion 201b of the machine 200 (e.g., to turn the machine 200 to the left), and various elements of the control interface 330 have been correspondingly rotated, angled, bent, or otherwise updated in response to the articulation of the machine. More specifically, in response to the rotation of the machine 200, the machine forward path indicators 332a-b, the machine reverse path indicators 334a-b, the drive component path indicators 336a-b, the machine side reference indicators 340a-b, and the machine side offset indicators 342a-b have each been correspondingly bent or angled (e.g., to represent a bending or bowing motion of the machine 200). For example, in response to the front portion 201a of the machine 200 rotating through the angle A1, the machine forward path indicators 332a-b, the machine reverse path indicators 334a-b, and the drive component path indicators 336a-b may each be curved about the same center of curvature as the machine 200. Additionally or alternatively, because the rotation of the machine 200 changes the orientation of the left side 203a and / or right side 203b of the machine 200, the machine side reference indicators 340a-b and the machine side offset indicators 342a-b may each be angled to maintain their alignment with the machine 200. In the illustrated embodiment, for example, each of the machine side reference indicators 340a-b and the machine side offset indicators 342a-b has been angled to a second angle A2 that is complementary to the first angle A1 and has approximately the same radius of curvature as the machine 200. Although the elements of the machine 200 and the control interface 330 are not necessarily aligned with each other, the machine 200 and the control interface 330 may be aligned with each other. Figure 3B 300 is shown in a rotational state, but it should be understood that the various elements of the control interface 330 can be based on the movement of the machine (for example, when the operator moves the machine 200 in a linear state ( Figure 3A ) and the rotation state ( Figure 3B ) dynamically bends / angles when transitioning between ).

[0036] refer to Figure 3C In some embodiments, the machine forward path indicators 332a-b extend a distance D from the front portion 201a of the machine 200. F The machine forward path indicators 332a-b may be automatically determined based at least in part on the forward travel speed of the machine. For example, when the machine 200 is traveling forward at a first speed S1, the machine forward path indicators 332a-b may extend a first distance D from the front 201a of the machine 200. F1 , and when the machine 200 is traveling forward at a second speed S2 greater than the first speed S1, the machine forward path indicators 332a-b may extend from the front 201a of the machine 200 by a second distance D F2 , the second distance is greater than the first distance D F1Additionally, the machine reverse path indicators 334a-b ( Figure 3A and 3B ) may also increase and / or decrease in length based on the speed at which the machine 200 is traveling (e.g., in reverse).

[0037] Figure 4 1 shows one or more video feeds 30 from a machine 200 shown on a display 116 of a remote control station 110 and including a control interface 330 in accordance with an embodiment of the present technology. The machine 200 and / or one or more aspects thereof may be structurally and / or functionally similar to Figure 1 Machine 100 and / or Figures 2A-3C In the illustrated embodiment, the video feed 30 includes (i) a camera (e.g., Figure 2C (i) a first (eg, front) video feed 31 from a first camera 205a) positioned to image a second side 201b of the machine 200, (ii) a second (eg, front) video feed 31 from another camera (eg, Figure 2C (iii) from another camera (e.g., positioned to image a third side 203a of the machine 200) (e.g., Figure 2C 2D ) is positioned to image the fourth side 203b of the machine, and (iv) a fourth (e.g., right) video feed 34 from another camera (e.g., fourth camera 205d of FIG. 2D ) positioned to image the fourth side 203b of the machine. In other embodiments, the video feeds 30 may include more or fewer video feeds, such as at least one, two, three, five, or any other suitable number of video feeds 30. Additionally or alternatively, the individual video feeds in the video feeds 30 may be from any suitable camera positioned to image any (one or more) sides of the machine 200.

[0038] In the illustrated embodiment, each video feed 30 includes a previously referenced Figures 3A-3COne or more control interface elements 330 are depicted. For example, (i) the first video feed 31 includes left and right machine forward path indicators 332a-b, one or more left and right drive component path indicators 336a-b, and left and right drive component leading edge indicators 338a-b; (ii) the second video feed 32 includes left and right machine reverse path indicators 334a-b and one or more left and right drive component path indicators 336a-b; (iii) the third video feed 33 includes a machine left reference indicator 340a and a machine left offset indicator 342a; and (iv) the fourth video feed 34 includes a machine right reference indicator 340b and a machine right offset indicator 342b. In other embodiments, one or more of these indicators may be omitted from the corresponding video feeds.

[0039] Remote operators (e.g. Figure 1 The operator 50) can view the video 30 on the display 116 of the remote control station while remotely operating the machine 200. Figures 3A-3C As shown on the display 116, one or more control interface elements 330 may be aligned with the machine 200, but are shown to be aligned with the machine plane M and / or the ground plane G ( Figure 2A ). Thus, various elements of control interface 330 may appear to be out of alignment with machine 200 in video feed 30. For example, in first video feed 31, left machine forward path indicator 332a is coplanar with the outer edge of the left front tire of machine 200, but is shown as projected onto machine plane M and / or ground plane G ( Figure 2A ) so that in the plane of display 116, the left machine forward path indicator 332a appears to be inside the outer edge of the left front tire. However, it is desirable to align the various elements of control interface 330 with the machine plane M and / or ground plane G ( Figure 2A ) alignment improves the teleoperator's perception of the machine's position relative to its surroundings. For example, because left machine forward path indicator 332a is shown to the teleoperator as being at ground level, the teleoperator may view left machine forward path indicator 332a to determine the forward travel path of left side 203a of machine 200, e.g., to avoid driving machine 200 into an obstacle and / or to more accurately assess the machine's position relative to an object located on left side 203a of the machine.

[0040] Figure 5A machine pitch indicator and a machine roll indicator configured in accordance with an embodiment of the present technology are shown. The control interface 330 may also include a machine roll indicator 550, a machine pitch indicator 560, and machine reference indicators 551a-b. In the illustrated embodiment, the machine roll indicator 550 and the machine pitch indicator 560 are centered relative to each other and the machine reference indicators 551a-b. The machine reference indicators 551a-b may include a left reference indicator 551a and a right reference indicator 551b. As described in more detail below, the machine roll indicator 550 and the machine pitch indicator 560 may move and / or rotate relative to the machine reference indicators 551a-b, for example, about a point centered between the left reference indicator 551a and the right reference indicator 551b.

[0041] The machine roll indicator 550 may include a current roll indicator 552 and one or more roll value indicators 554. In the illustrated embodiment, the machine roll indicator 550 includes nine roll value indicators 554a-i; in other embodiments, the machine roll indicator 550 may include more or fewer machine roll value indicators 554, such as less than nine, more than nine, or any other suitable number of roll value indicators 554. Each roll value indicator 554 may represent and / or otherwise be associated with a roll value for the machine. Each roll value may be based at least in part on the machine roll axis R M ( Figure 2A ) and / or relative to the machine roll axis, and may include angle, grade percentage, rotation percentage, roll direction (e.g., a first (e.g., left) roll direction R left and the first rolling direction R left The opposite second (eg, rightward) scrolling direction R right ) and / or any other suitable scroll value.

[0042] Each of the roll value indicators 554 may be associated with a roll threshold for the machine. The roll threshold may include a roll value or a roll value range at or beyond which the machine is expected to be unstable and / or have an increased likelihood of tipping or rolling. The roll threshold may be machine specific and / or determined by a remote operator. The roll value indicator 554 corresponding to a roll value within or near the roll threshold(s) may appear different from the remaining (e.g., non-threshold, safe operating range, etc.) roll value indicators 554. In the illustrated embodiment, for example, the roll value indicators 554a and 554i correspond to the right roll value threshold and the left roll value threshold, respectively, and appear different from the non-threshold roll value indicators 554c-g. In addition, in the illustrated embodiment, the scrolling value indicators 554b and 554h represent scrolling values ​​that are near / approaching the scrolling value thresholds 554a, 554i, and therefore, the near-threshold scrolling value indicators 554b, 554h appear different from the threshold indicators 554a, 554i and the non-threshold indicators 554c-554g. In at least some embodiments, the threshold indicators 554a-i can be distinguished by color, for example, the threshold indicators 554a, 554i can be red, the non-threshold indicators 554c-554g can be green, and the near-threshold scrolling value indicators 554b, 554h can be yellow. Additionally or alternatively, each of the threshold, near-threshold and / or non-threshold scrolling value indicators can have the same or different size, transparency, brightness, flashing or blinking effect and / or any other suitable visual characteristics.

[0043] The current scroll indicator 552 can rotate relative to the scroll value indicator 554, for example, to indicate the current scroll value 556 of the machine. The current scroll indicator 552 can rotate in the same direction as the scroll direction of the machine and / or rotate the current scroll value 556. In the illustrated embodiment, for example, the current scroll indicator 552 is in the second scroll direction R right Rotating up corresponds to the current scroll value 556 of the machine rolling to the right or clockwise (from the perspective of the remote operator) corresponding to the current scroll value 556. It is expected that rotating the scroll indicator in the same direction and / or by the same amount as the machine's rolling direction will improve the remote operator's perception of the machine's orientation relative to its surroundings. In some embodiments, the current scroll value 556 can be displayed visually, such as as a number. In the illustrated embodiment, for example, the current scroll value 556 is superimposed on the current scroll indicator 552 and is configured to move / rotate with the current scroll indicator 522.

[0044] In some embodiments, one or more scroll value indicators 554 may be responsive to rotation of the current scroll indicator 552. For example, if the current scroll indicator 552 is at or near one of the near-threshold indicators 554b, 554h and / or the threshold indicators 554a, 554i, the corresponding indicator 554a, 554b, 554h, 554i may change visually, for example, in size, brightness, color, a shimmer or blinking effect, the presence, absence, or change of an outline at least partially or completely surrounding the indicator, and / or any other suitable visual characteristic. Additionally or alternatively, the current scroll indicator 552 and / or the displayed current scroll value 556 may change visually when the scroll value is at or near one of the near-threshold indicators 554b, 554h and / or the threshold indicators 554a, 554i.

[0045] The machine pitch indicator 560 may include one or more pitch value indicators 562, and each pitch value indicator 562 may correspond to a pitch value 564 of the machine. In the illustrated embodiment, the machine pitch indicator 560 includes four pitch value indicators 562a-d, each pitch value indicator corresponding to a respective pitch value 564a-d. In other embodiments, the machine pitch indicator 560 may include more or fewer pitch value indicators 562, such as less than four, more than four, or any other suitable number of value indicators 562. In some embodiments, each pitch value indicator 562 may be spaced apart a distance corresponding to its respective pitch value 564.

[0046] Each pitch value 564 may be based at least in part on the machine pitch axis PM ( Figure 2A ) and / or relative to the machine pitch axis, and may include angle, grade percentage, pitch direction (e.g., first (e.g., positive, upward) pitch P pos direction and a second (eg, negative, downward) pitch P neg In the illustrated embodiment, for example, first pitch value indicator 562a is associated with first pitch value 564a, and second pitch value indicator 562b is associated with second pitch value 564b that is less than first pitch value 564a, such that first pitch value indicator 562a corresponds to an orientation of the machine in which the machine is pitched upward an amount greater than second pitch value indicator 562b and the associated orientation. In some embodiments, one or more pitch values ​​564 may be visually displayed (e.g., displayed as numbers). In the illustrated embodiment, for example, each pitch value 564a-c is superimposed on a corresponding pitch value indicator 562a-c and is configured to move with the corresponding pitch value indicator.

[0047] Each pitch value indicator in the pitch value indicator 562 can be associated with a pitch threshold of the machine. The pitch threshold may include a pitch value or a pitch value range at which or beyond which the machine is expected to be unstable and / or have an increased likelihood of rolling or pitching. The pitch threshold may be machine-specific and / or determined by a remote operator. The pitch value indicator 562 corresponding to a pitch value within or near the pitch threshold (one or more) may appear different from the remaining (e.g., non-threshold, safe operating range, etc.) pitch value indicators 562, which are substantially similar or identical to the threshold, near-threshold, and non-threshold rolling value indicators 554 described in detail above. In at least some embodiments, for example, the pitch value indicator 562 associated with the pitch threshold may be distinguished by color, size, transparency, brightness, flashing or blinking effects, and / or any other suitable visual characteristics. In some embodiments, the visual change of pitch value indicator 562 may be responsive, at least in part, to the proximity of a given pitch value indicator to a machine reference indicator 551a - b .

[0048] In operation, each pitch value indicator 562 may be configured to move relative to the machine reference indicators 551a-b, for example, to indicate the current pitch value 564 of the machine. For example, the pitch value indicator 562 may move based at least in part on the pitch direction, for example, in a first pitch direction P when the machine is pitching positively / upwardly. pos (For example, downward, as shown by arrow P pos and / or in the second pitch direction P when the machine is in downward / negative pitch neg (For example, upward, as shown by arrow P neg In the illustrated embodiment, for example, the pitch value indicator 562 has been moved so that the third pitch value indicator 564c is positioned between the machine reference indicators 551a-b, and therefore, the third pitch value 564c is the current pitch value of the machine. Additionally or alternatively, each pitch value indicator 562 may be moved (e.g., rolled, translated, shifted, etc.) a distance corresponding to the change in the pitch of the machine. For example, the space between the various pitch value indicators may correspond to a 1% change in the pitch of the machine, and therefore, if the pitch of the machine changes from the third pitch value 564c to the second pitch value 564b, one or more pitch value indicators 562b may be shifted to the machine reference indicators 551a-b and shifted a distance corresponding to the change in the pitch of the machine, for example, so that the third pitch value indicator 562b is aligned with the machine reference indicators 551a-b. In these and other embodiments, each pitch value indicator 562 may be configured to be aligned with the machine roll indicator 550, for example, in the roll direction R right , R leftIn the illustrated embodiment, for example, the current scroll indicator 552 is already in the second scroll direction R right The current roll value 556 is rotated about a point centered between the machine reference indicators 551a-b (as described above), and the pitch value indicators 562a-d are also rotated in the second roll direction R right The current roll value 556 is rotated upward. It is expected that rotating the machine pitch indicator 560 in unison with the machine roll indicator 550 will improve the remote operator's perception of the orientation of the machine relative to the machine's surroundings.

[0049] Fig. 6A The embodiment according to the present technology is shown Figure 5 6 and 7. In the illustrated embodiment, the machine pitch indicator 560 and the machine roll indicator 550 are both positioned at or near the center of the first video feed 31 on the front side 201a of the machine 200. In other embodiments, the machine pitch indicator 560 and / or the machine roll indicator 550 may have any other suitable location relative to the first video feed 31 (e.g., top left corner, off-center, etc.), may be positioned in any other suitable video feed on either side of the machine (e.g., the second video feed 32 on the rear side 201b of the machine 200), and / or may have any other suitable configuration.

[0050] The work tool indicator 670 may include a ground reference indicator 672 and a tool height indicator 674, each of which may have a left element and a right element (eg, a left ground reference indicator 672 and a right ground reference indicator 674). a , right ground reference indicator 672 b , Left tool height indicator 674 a , right tool height indicator 674 b ). The ground reference indicator 672 and the tool height indicator 674 may each correspond to the position of the leading edge 607 of the machine working tool 202. In some embodiments, the ground reference indicator 672 may represent the machine plane M and / or the ground plane G ( Figure 2A ), for example, so that the remote operator can refer to ground reference indicator 672 to determine the proximity of work tool 202 to objects in front of machine 200. In other embodiments, ground reference indicator 672 may represent a projection of the current position of leading edge 607 on machine plane M and / or ground plane G ( Figure 2A) on a baseline (e.g., stationary, ground, stowed, etc.) position of leading edge 607. Tool height indicator 674 may be aligned (e.g., tracking, co-linear, etc.) with the current position of leading edge 607 during use of work tool 602. Thus, tool height indicator 674 may move (e.g., vertically) relative to ground reference indicator 672 based at least in part on the vertical movement of leading edge 607, such that a remote operator may reference tool height indicator 674 to determine the current height of leading edge 607 relative to ground reference indicator 672, such as with respect to FIG. Figure 6B More details shown and described.

[0051] Figure 6B Shows Fig. 6A Working tool indicator 670. Fig. 6A and 6B , the work tool indicator 670 may change in response to the movement of the work tool 202. In the illustrated embodiment, for example, the leading edge 607 (shown in phantom) of the work tool 202 has been raised, and the tool height indicator 676 has moved upward to track the movement of the work tool 202. Additionally, in the illustrated embodiment, the left and right tool height indicators 674a-b ( Fig. 6A ) has been transformed into a single or unitary tool height indicator 676 ( Figure 6B In at least some embodiments, the work tool indicator 670 may be positioned between the left tool height indicator and the right tool height indicator 674a-b ( Fig. 6A ) are at least substantially or completely aligned with the corresponding left and right ground reference indicators 672a-b (e.g., when the work tool is in a baseline, stationary, ground, stowed, etc. position ( Fig. 6A )) includes left tool height indicator and right tool height indicator 674a-b ( Fig. 6A ) and may transition to a single tool height indicator 676 ( Figure 6B In some aspects, it is expected that changing the appearance of the work tool height indicator based on the position of the work tool will improve the remote operator's perception of the position / orientation of the work tool relative to the environment of the associated machine, such as by indicating the relative height / alignment of the leading edge of the work tool relative to the ground.

[0052] In some embodiments, a tool height value 678 corresponding to the current height of work tool 202 may be displayed in addition to or in place of tool height indicators 674, 676. In the illustrated embodiment, for example, tool height value 678 is displayed proximate the left end of tool height indicator 676. In other embodiments, tool height value 678 may have another suitable location on display 116.

[0053] refer to Figure 7 The control interface 330 may also include a tool pitch indicator 780 and / or a tool roll or tilt indicator 782 ("tool tilt indicator 782"). The tool pitch indicator 780 may include a tool pitch indicator 780 that is structurally and / or functionally similar to the tool pitch indicator 780. Figure 5 The tool pitch indicator 780 may be substantially similar or identical to at least some aspects of the machine pitch indicator 560 of the present invention. The tool pitch indicator 780 may include one or more pitch value indicators 784, each of which may be aligned with the tool height indicators 774a-b (e.g., longitudinally) and configured to rotate with the tool height indicators 774-b (e.g., in response to the tilt of the work tool). In the illustrated embodiment, the tool pitch indicator 780 is positioned to the right of the right tool height indicator 774b and the tool tilt indicator 782, and is configured to move upward and / or downward in unison with the tool height indicators 774a-b. In other embodiments, the tool pitch indicator 780 may have another suitable position (e.g., upper left corner, off-center, etc.) and / or may have a fixed position.

[0054] The tool tilt indicator 782 may include a Figure 5 The tool tilt indicator 782 may include at least some aspects that are substantially similar or identical to the machine roll indicator 550 of the embodiment of the present invention. The tool tilt indicator 782 may include one or more tilt value indicators 786. In the illustrated embodiment, the marked tilt value indicator 786 is aligned with the right work tool height indicator 774b; in some aspects, the tilt value indicator 786 aligned with the work tool height indicator 774a-b may indicate the current tilt value of the work tool. The tool tilt indicator 782 may be configured to move and / or rotate in unison with the work tool height indicator 774a-b (e.g., relative to the center of the work tool height indicator 774a-b and / or the center of the work tool, as indicated by arrow T) to track or follow the movement (e.g., tilt) of the work tool. In the illustrated embodiment, the tool pitch indicator 780 is aligned with the right tool height indicator 774b. In other embodiments, the tool pitch indicator 780 may be aligned with the left tool height indicator 774a, may have a fixed position on the display 116, or may have another suitable position (e.g., upper left corner, off-center, etc.). One or both of the tool pitch indicator 780 and the tool tilt indicator 782 may be associated with a corresponding operating threshold (e.g., a tool pitch threshold and / or a tool roll / tilt threshold) and may be configured to change (e.g., visually change) when a current value is at or near the associated threshold, consistent with a previous description of the tool pitch indicator 780 and the tool tilt indicator 782. Figure 5 The pitch thresholds of the machines described are generally similar or identical.

[0055] Figure 8 800 is a flowchart illustrating a method 800 for remotely operating a machine according to an embodiment of the present technology. The method 800 is illustrated as a set of blocks, steps, operations, or processes 801-806. Figure 1-8 Several of blocks 801-806 are described. All or a subset of blocks 801-806 may be at least partially controlled by a remote control station (e.g. Figure 1 Remote control station 110), Fig. 9 Device 900, Fig.10 at least a portion of the environment 1000, Figure 1 At least one element of element 1100 of and / or various components of any other suitable device and / or system are performed. In addition or alternatively, all or a subset of blocks 801-806 can be performed at least in part by an operator of the device and / or system (e.g., a remote operator, a human operator, etc.). In addition, any one or more of blocks 801-806 can be performed according to the discussion herein.

[0056] Method 800 begins at block 801 by displaying a video feed from a camera of a machine. The video feed may include the previously referenced Figure 4 and 6A -7, and / or any other suitable video feed. In at least some embodiments, for example, displaying the video feed may include displaying multiple video feeds, each associated with a different side of the machine. The camera may include Figure 1 Camera 105, Figure 2C One or more of the cameras 205a-d and / or any other suitable cameras. The machine may include an industrial machine, Figure 1 Machine 100, Figure 2A-4 and 5-7 of machine 200 or any other suitable machine.

[0057] At block 802, method 800 continues with receiving / collecting first telematics data associated with remote operation of the machine. Receiving / collecting the first telematics data may include receiving / collecting the first telematics data via one or more sensors of the machine. The one or more sensors may include Figure 1 In at least some embodiments, for example, the first telematics data may include data associated with: movement of a work tool of the machine, movement of the work tool relative to one or more work tool axes (e.g., Figure 2B Pitch of working tool P WT Axis, Roll R WT Axis and / or yaw Y WTThe orientation of a machine and / or a portion thereof relative to one or more machine axes (e.g., Figure 2A Pitch of the machine M Axis, rolling R M Axis and / or yaw Y M axis), the orientation of the machine, the speed the machine is traveling, the direction the machine is traveling, and / or any other suitable telematics data associated with remote operation of the machine.

[0058] At block 803, method 800 continues by generating a control interface. The control interface may be associated with remote operation of the machine. The control interface may be structurally and / or functionally similar to the remote operation interface of the machine. Figure 3A-7 The control interface 330 of the machine is generally similar and may include various items of the following: machine forward path indicators 332a-b, machine reverse path indicators 334a-b, drive component path indicators 336a-b, drive component indicators 338a-b, machine side reference indicators 340a-b, machine side offset indicators 342a-b, machine roll indicators 550a-b, machine pitch indicators 560a-b, machine reference indicators 551a-b, work tool indicators 670, ground reference indicators 672a-b, tool height indicators 674a-b, single tool height indicators 676, tool pitch indicator 780, tool roll or tilt indicator 782, and / or any other suitable control interface elements associated with remote operation of the machine. In some embodiments, generating the control interface includes generating the control interface based at least in part on the first telematics data (block 802). In these and other embodiments, generating the control interface may include displaying various elements of the control interface on a display of a video feed (block 801) and / or aligning various elements of the control interface on the displayed video feed with at least a portion of the machine. Figure 4 As shown, the left machine forward path indicator 332a may be coplanar with the outer edge of the left front tire of the machine 200, but is shown as projected onto the machine plane M and / or the ground plane G ( Figure 2A ) so that in the plane of display 116, the left machine forward path indicator 332a appears to be inboard of the outer edge of the left front tire.

[0059] At block 804, method 800 continues with receiving / collecting second telematics data associated with remote operation of the machine. The second telematics data may be substantially similar or identical to the first telematics data (block 802), and may include data associated with the same or different portions as the first telematics data. Thus, receiving / collecting the second telematics data may be substantially similar or identical to receiving / collecting the first telematics data (block 802). In some embodiments, the first telematics data is received / collected at a first time, and the second telematics data is collected at a second time that is different from the first time (e.g., after the first time). In some embodiments, the first telematics data may be received / collected from various components in the machine components at a first time, and the second telematics data may be received / collected from various components in the machine components at a second time, such that the first telematics data and the second telematics data may correspond to the operation of various components in the machine components at least between the first time and the second time.

[0060] At block 805, method 800 continues by determining whether the operating state of the machine has changed. In some embodiments, the operating state of the machine may change when one or more machine components (e.g., one or more components associated with the control interface, block 803) have been moved, rotated, adjusted, etc. by the remote operator. For example, the operating state of the machine may change when the front portion 201a of the machine 200 is articulated relative to the rear portion 201b of the machine, as described above with respect to Figure 3A and 3B As another example, the operating state of the machine may change based on the speed at which the machine 200 is moving, as described above with respect to Figure 3C As another example, the operating state of the machine may change in response to the pitch and / or roll of the machine, as described above with respect to Figure 5 As yet another example, the operating state of the machine may change when the work tool 202 moves, elevates, rotates, pitches, rolls / tilts, etc., as described above with respect to Figure 6A-7as described. In these and other embodiments, the operating state of the machine may change when the second telematics data associated with a given component is different from the first telematics data associated with the given component. Thus, determining whether the operating state of the machine has changed may be based at least in part on the first telematics data (block 802) and / or the second telematics data (block 804). In some embodiments, for example, determining whether the operating state of the machine has changed may include identifying one or more changes in the second telematics data compared to the first telematics data, such as to indicate whether various components of the machine components have been moved, rotated, adjusted, etc. as previously described. In these and other embodiments, determining whether the operating state of the machine has changed may include determining whether the machine and / or one or more of its components are operating at or near an operating threshold. The operating thresholds may include a machine pitch threshold ( Figure 5 ), machine rolling threshold ( Figure 5 ), working tool pitch threshold ( Figure 7 ), tool roll / tilt threshold ( Figure 7 ) and / or any other suitable operating threshold.

[0061] If it is determined at block 805 that the operating state of the machine has not changed (block 805-"No"), the method may return to block 802 and repeat one or more of blocks 802-805. If it is determined at block 805 that the operating state of the machine has changed (block 805-"Yes"), the method may continue to block 806.

[0062] At block 806, method 800 continues by updating the control interface (block 803). In some embodiments, updating the control interface may include updating the control interface based at least in part on the second telematics data (block 804). Updating the control interface may include moving, repositioning, rotating, pivoting, etc., one or more elements of the control interface, such as with respect to the second telematics data. Figure 3A-7 Additionally or alternatively, updating the control interface may include changing the appearance of one or more elements and / or portions thereof of the control interface based on the associated operational thresholds (block 805). Changing the appearance of one or more elements and / or portions thereof may include changing the size, color, brightness, transparency, or any other visual characteristic, such as with respect to the control interface. Figure 5 In these and other embodiments, updating the control interface may include transforming an element of the control interface from a first form to a second form. Fig. 6A and 6B , updating the control interface may include causing the work tool height indicator to be between the left work tool height indicator and the right work tool height indicator 674a-b ( Fig. 6A ) and a single work tool height indicator 676 ( Figure 6B) (e.g., based at least in part on data associated with the height of the work tool). In these and other embodiments, updating the control interface may include any other suitable update to the control interface and / or one or more elements thereof.

[0063] Although the steps of method 800 are discussed and shown in a particular order, Figure 8 The method 800 shown in the embodiment is not limited thereto. A person skilled in the relevant art will recognize that the method 800 shown can be changed and still remain in these and other embodiments of the present technology. For example, in some embodiments, the method 800 can be omitted and / or repeated. Figure 8 One or more steps (eg, block 805 ) of method 800 shown in .

[0064] Although the steps of method 800 are discussed and shown in a particular order, Figure 8 The method 800 shown in the embodiment is not limited thereto. In other embodiments, the method 800 may be performed in a different order. In these and other embodiments, any step of the method 800 may be performed before, during, and / or after any other step of the method 800. In addition, one of ordinary skill in the relevant art will recognize that the method 800 shown may be changed and still remain in these and other embodiments of the present technology. For example, in some embodiments, the steps of the method 800 may be omitted and / or repeated. Fig.10 One or more steps of method 800 shown in .

[0065] The systems, devices and / or methods of the present technology may be embodied as dedicated hardware (e.g., circuit systems), programmable circuit systems appropriately programmed with software and / or firmware, or a combination of dedicated and programmable circuit systems. Thus, embodiments may include a machine-readable medium storing instructions that may be used to cause a computer, microprocessor, processor and / or microcontroller (or other electronic device) to perform a process. The machine-readable medium may include, but is not limited to, an optical disk, a compact disk read-only memory (CD-ROM), a magneto-optical disk, a ROM, a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic or optical card, a flash memory, or other types of media / machine-readable media suitable for storing electronic instructions.

[0066] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Fig. 9A block diagram showing an overview of an apparatus with which embodiments of the present technology may operate. For example, an apparatus may include hardware components of a system or apparatus 900 that generates one or more control interface elements for remote operation of a machine. The apparatus 900 may include one or more input devices 920 that provide input to a CPU (processor) 910, thereby informing it of an action. The action is typically mediated by a hardware controller that interprets signals received from the input device and transmits the information to the CPU 910 using a communication protocol. The input device 920 includes, for example, a mouse, keyboard, touch screen, infrared sensor, trackpad, wearable input device, camera or image-based input device, microphone, or other operator input device.

[0067] The CPU 910 may be a single processing unit or multiple processing units in a device or distributed across multiple devices. The CPU 910 may be coupled to other hardware devices, for example, using a bus such as a PCI bus or a SCSI bus. The CPU 910 may communicate with a hardware controller of a device such as a display 930. The display 930 may be used to display text and graphics. In some examples, the display 930 provides graphical and textual visual feedback to an operator. In some embodiments, the display 930 includes an input device as part of the display, such as when the input device is a touch screen or is equipped with an eye direction monitoring system. In some embodiments, the display is separate from the input device. Examples of display devices are: LCD display screens; LED display screens; projection, holographic or augmented reality displays (e.g., head-up display devices or head-mounted devices); and the like. Other input / output I / O devices 940 may also be coupled to the processor, such as a network card, video card, audio card, USB, FireWire or other external device, sensor, camera, printer, speaker, CD-ROM drive, DVD drive, disk drive, or Blu-ray device.

[0068] In some embodiments, the device 900 also includes a communication device capable of wireless communication with a network node or based on wired communication. The communication device can communicate with another device or server through a network using, for example, the TCP / IP protocol. The device 900 can use the communication device to distribute operations on multiple network devices.

[0069] CPU 910 can access memory 950. Memory includes one or more of various hardware devices for volatile and non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include RAM, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, CD, DVD, magnetic storage device, tape drive, device buffer, etc. Memory is not a propagation signal separated from the underlying hardware; therefore, memory is non-temporary. Memory 950 may include program memory 960 storing programs and software, such as operating system 962, control interface system 964 (which may include instructions for generating control interface elements and / or performing methods for remote operation of the machine) and other application programs 966. Memory 950 may also include data storage 970, which may include database information, etc., which may be provided to program memory 960 or any element of device 900.

[0070] Some embodiments may operate with numerous other general or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, personal computers, server computers, handheld or notebook devices, cell phones, mobile phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above, and the like.

[0071] Fig.10 1 is a block diagram showing an overview of an environment 1000 in which embodiments of the present technology may operate. The environment 1000 may include one or more client computing devices 1005A-D, examples of which may include device 900. The client computing device 1005 may operate in a networked environment using a logical connection to one or more remote computers, such as a server computing device 1010, through a network 1030.

[0072] In some embodiments, server computing device 1010 may be an edge server that receives client requests and coordinates the fulfillment of these requests through other servers such as servers 1020A-C. Server computing devices 1010 and 1020 may include a computing system, such as device 900. Although each server computing device 1010 and 1020 is logically shown as a single server, the server computing devices may each be a distributed computing environment that includes multiple computing devices located at the same or geographically different physical locations. In some embodiments, each server computing device 1020 corresponds to a group of servers.

[0073] The client computing device 1005 and the server computing devices 1010 and 1020 can each act as a server or client to other server / client devices. The server 1010 can be connected to a database 1015. The servers 1020A-C can each be connected to a corresponding database 1025A-C. As discussed above, each server 1020 can correspond to a group of servers, and each of these servers can share a database or can have its own database. The databases 1015 and 1025 can store information in a library (e.g., store). Although the databases 1015 and 1025 are logically shown as a single unit, the databases 1015 and 1025 can each be a distributed computing environment including multiple computing devices, which can be located within their corresponding servers, or can be located at the same or geographically different physical locations.

[0074] The network 1030 may be a local area network (LAN) or a wide area network (WAN), but may also be other wired or wireless networks. The network 1030 may be the Internet or some other public or private network. The client computing device 1005 may be connected to the network 1030 through a network interface, such as through wired or wireless communication. Although the connection between the server 1010 and the server 1020 is shown as a separate connection, these connections may be any type of local area network, wide area network, wired or wireless network, including the network 1030 or a separate public or private network.

[0075] Fig.11 1 is a block diagram showing an element 1100 that can be used in a machine remote operation system according to an embodiment of the present technology. The element 1100 includes hardware 1102, general software 1120, and a dedicated element 1140. As discussed above, a system implementing the disclosed technology can use various hardware, including a processing unit 1104 (e.g., a CPU, a GPU, an APU, etc.), a working memory 1106, a storage memory 1108, and an I / O device 1110. The element 1100 can be implemented in a client computing device (e.g., client computing device 1005) or on a server computing device (e.g., server computing device 1010 or 1020).

[0076] The general software 1120 may include various application programs, including an operating system 1122, local programs 1124, and a basic input and output system (BIOS) 1126. The special components 1130 may be subcomponents of the general software application 1120 (e.g., local programs 1124), which may include a control interface system 964 (see Fig. 9and described above). The dedicated component 1130 may include a telematics data module 1132, a control interface component module 1134, a remote command module 1136, and / or components that can be used to transmit data and control the dedicated components, such as an interface 1138. In some embodiments, the component 1100 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more dedicated components 1130.

[0077] B. Industrial Applicability

[0078] In some embodiments, the machine remote operating system may include a telematics data module 1132, a control interface element module 1134, a remote command module 1136 ( Figure 7 ).

[0079] In operation, the telematics data module 1132 collects and stores telematics data (see Figure 8 802, 804 in FIG. 11 ). The control interface element module 1134 may determine whether the operating state of the machine has changed (see block 805), and generate a control interface (see block 802) and / or update the control interface (see block 806) based on the telematics data collected / stored by the telematics data module 1132. In at least some embodiments, the control interface element module 1134 may generate various control interface elements of the control interface elements 330 described herein. The remote command module 1136 may allow a remote operator to, for example, send a command signal (e.g., Figure 1 The command signal 40) is transmitted to the machine.

[0080] Thus, the disclosed technology provides a machine teleoperation system, and thereby can improve the precision, accuracy and / or safety with which a remote operator can remotely operate a machine. In particular, it is expected that the control interface and / or its elements described herein will improve the remote operator's perception of the machine's orientation and surrounding environment.

[0081] C. in conclusion

[0082] According to the foregoing, it should be understood that, for the purpose of illustration, the specific embodiments of the technology have been described herein, but the well-known structures and functions are not shown or described in detail to avoid unnecessary ambiguity in the description of the technical embodiments. To the extent that any material incorporated herein by reference conflicts with the present disclosure, the present disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, unless the word "or" is explicitly limited to a single item that only refers to a list of two or more items and is unrelated to other items, the use of "or" in this list should be interpreted as including any single item in (a) the list, (b) all items in the list, or (c) any combination of items in the list. As used herein, the phrase "and / or" as in "A and / or B" refers to a single A, a single B, and both A and B. Where the context permits, a singular or plural term may also include a plural or singular term, respectively. In addition, the terms “comprising,” “including,” “having,” and “with” are used herein to indicate the inclusion of at least the listed (one or more) features, without excluding any greater number of the same features and / or additional types of other features.

[0083] In addition, as used herein, the term "substantially" refers to the complete or almost complete scope or degree of action, characteristic, property, state, structure, project or result. For example, an object that is "substantially" enclosed will mean that the object is completely enclosed or almost completely enclosed. In some cases, the precise allowable deviation degree from absolute completeness may depend on the specific context. However, in general, with complete proximity will make it possible to have the same total result as obtaining absolute and overall. The use of "substantially" is equally applicable when used in negative connotations, to refer to the complete or almost complete lack of action, characteristic, property, state, structure, project or result. In addition, the terms "connect" and "connect" are used interchangeably in this article, and refer to both direct and indirect connection or connection. For example, where the context allows, element A "connects" or "connects" to element B can refer to (i) A directly "connects" or directly "connects" to B, and / or (ii) A indirectly "connects" or indirectly "connects" to B.

[0084] The above detailed description of embodiments of the technology is not intended to be exhaustive or to limit the technology to the precise form disclosed above. Although specific embodiments and examples of the technology are described above for illustrative purposes, various equivalent modifications within the scope of the technology are feasible, as will be appreciated by those skilled in the relevant art. For example, although the blocks are presented in a given order, alternative embodiments may execute the blocks in a different order. As another example, the various components of the technology may be further divided into sub-components, and / or the various components and / or functions of the technology may be combined and / or integrated. In addition, although the advantages associated with certain embodiments of the technology have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments necessarily need to exhibit such advantages to fall within the scope of the technology.

[0085] The titles provided herein are for convenience purposes only and do not necessarily affect the scope of the embodiments. In addition, the drawings are not necessarily drawn to scale. For example, the sizes of some elements in the drawings may be enlarged or reduced to help improve the understanding of the embodiments. Moreover, although the disclosed technology is susceptible to various modifications and alternative forms, specific embodiments have been illustrated by examples in the drawings and described in detail below. However, it is not intended to unnecessarily limit the described embodiments. On the contrary, the embodiments are intended to cover all modifications, combinations, equivalents and alternatives that fall within the scope of the present disclosure.

[0086] It should also be noted that other embodiments in addition to the embodiments disclosed herein are within the scope of the present technology. For example, embodiments of the present technology may have different configurations, components, and / or procedures in addition to those shown or described herein. In addition, it will be understood by those of ordinary skill in the art that these and other embodiments may not have several configurations, components, and / or procedures shown or described herein without departing from the present technology. Therefore, the present disclosure and related technology may cover other embodiments not explicitly shown or described herein.

[0087] Reference in this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearance of the phrase "in one embodiment" (or the like) in various places in the specification does not necessarily refer to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. In addition, various features are described that may be exhibited by some embodiments but not by other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not of other embodiments.

[0088] The terms used in this specification generally have their ordinary meanings in the art, in the context of the present disclosure, and in the specific context in which each term is used. It should be understood that the same content can be expressed in a variety of ways. Therefore, alternative language and synonyms can be used for any one or more terms discussed herein, and no matter whether the term is elaborated or discussed in detail in this article, it is not given any special meaning. Synonyms for certain terms are provided. The narration of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification, including the examples of any term discussed herein, is only illustrative, and is not intended to further limit the scope and meaning of the present disclosure or any exemplary term. Similarly, the present disclosure is not limited to the various embodiments given in this specification. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs. In the event of a conflict, this document (including definitions) shall prevail.

Claims

1. A method (800) for improving an operator's awareness of an operating environment when remotely operating an industrial machine (100, 200), the method comprising: displaying (801) a video feed from a camera of the industrial machine via a display screen (116) of a remote operating system; receiving ( 802 ) at a first time and via the remote operation system ( 110 ) first telematics data associated with remote operation of the industrial machine; generating (803) an industrial machine control interface (330) based at least in part on the first telematics data, wherein generating the industrial machine control interface comprises displaying the industrial machine control interface on the display screen over the video feed and in alignment with at least a portion of the industrial machine; receiving ( 804 ) at a second time and via the remote operation system, second telematics data associated with remote operation of the industrial machine; and The industrial machine control interface is updated ( 806 ) based at least in part on the second telematics data, wherein updating the industrial machine control interface includes maintaining alignment of the industrial machine control interface with the portion of the industrial machine.

2. The method according to claim 1, wherein: The industrial machine control interface includes a machine roll indicator associated with a roll angle and a roll direction of the industrial machine, The second telematics data includes changes in the roll angle and the roll direction, and Updating the industrial machine control interface includes rotating the machine scroll indicator in the scroll direction by the scroll angle.

3. The method according to claim 1, wherein: The industrial machine control interface includes a machine pitch indicator associated with a pitch angle and a pitch direction of the industrial machine, The second telematics data includes changes in the pitch angle and the pitch direction, and Updating the industrial machine control interface includes moving the machine pitch indicator in the pitch direction a distance corresponding to the pitch angle.

4. The method according to claim 1, wherein: The industrial machine control interface includes a machine pitch indicator associated with a pitch angle and a pitch direction of the industrial machine, and a machine roll indicator associated with a roll angle and a roll direction of the industrial machine; The second telematics data includes The pitch angle and the pitch direction, and The rolling angle and / or the rolling direction; and Updating the industrial machine control interface includes moving the machine pitch indicator in the pitch direction a distance corresponding to the pitch angle, and The machine roll indicator and the machine pitch indicator are rotated in the roll direction by the roll angle.

5. The method according to claim 1, wherein: The industrial machine control interface includes a machine work tool indicator, the machine work tool indicator being associated with at least one of: (i) height, (ii) pitch angle and pitch direction, and (iii) roll angle and roll direction of a work tool of the industrial machine, the machine work tool indicator comprising— Work tool reference indicator, and Working tool leading edge indicator; The second telematics data includes a change in at least one of: (i) the altitude, (ii) the pitch angle and the pitch direction, and (iii) the roll angle and the roll direction; and Updating the industrial machine control interface includes at least one of: (i) positioning the work tool leading edge indicator relative to the work tool reference indicator at a distance proportional to the change in height, (ii) moving the work tool leading edge indicator in the pitch direction relative to the work tool reference indicator a distance corresponding to the change in pitch angle, and (iii) rotating the work tool leading edge indicator in the roll direction relative to the work tool reference indicator by the change in roll angle.

6. A method according to claim 1, wherein the industrial machine control interface includes (i) a machine orientation indicator, which is associated with a first orientation of the industrial machine relative to a first rotational axis of the industrial machine, and (ii) a component orientation indicator, which is associated with a second orientation of a component of the industrial machine relative to a second rotational axis of the component.

7. The method of claim 1, wherein the industrial machine control interface comprises at least one of: a machine pitch indicator, a machine roll indicator, a machine tool height indicator, a machine path indicator, a machine side reference indicator, a machine side threshold indicator, a machine drive element leading edge indicator, a machine tool leading edge indicator, a machine tool pitch indicator, or a machine tool roll indicator.

8. The method of claim 1, wherein updating the industrial machine control interface comprises changing at least one of an orientation, size, color, brightness, transparency, blinking pattern, outline, shape, and / or motion of at least a portion of the industrial machine control interface.

9. The method according to claim 1, further comprising: setting an operating threshold associated with the industrial machine; as well as analyzing the second telematics data to determine whether the second telematics data indicates that the industrial machine is operating at or near the operating threshold; Wherein updating the industrial machine control interface includes changing at least a portion of the industrial machine control interface if it is determined that the industrial machine is operating at or near the operating threshold.

10. The method of claim 1, wherein updating the industrial machine control interface comprises changing a size of at least a portion of the industrial machine control interface based at least in part on a speed at which the industrial machine is traveling.

Images