Autonomous and semi-autonomous control of aerial robotic system

Abstract

The present disclosure describes systems and methods for performing tasks in an operating environment of aerial equipment with autonomous or semi-autonomous robots. In some embodiments, the robot is disposed at an end of a cantilever of the aerial equipment. The robot may include cameras, actuators, sensors, processors, and manipulators that work together to perform tasks fully autonomously or semi-autonomously. Further, the robot may include tools for performing the task, and computer executable instructions for performing the task may be based on various sensor inputs, the tools, and the task to be performed.

Description

Technical Field

[0001] The present disclosure is generally applicable to aerial robotic systems and methods and operations related to aerial robotic systems. In particular, the present disclosure relates to autonomous and semi-autonomous control of aerial robotic systems. Background Art

[0002] Aerial robotic systems can be used to perform work at aerial work sites, including aerial work sites associated with power transmission lines. Aerial work sites and power transmission lines are accompanied by various hazards. Some hazards can be mitigated by using aerial robotic systems to perform at least part of the work. Additionally, or alternatively, certain tasks can be simplified by using aerial robotic systems.

[0003] Furthermore, human control of an aerial robotic system is not without danger. It may be difficult for a user standing on the ground to see the aerial work environment or to view the aerial work environment through a camera mounted on or near the aerial robotic system. As a result, the user may unknowingly strike components of the power transmission system or may be unable to effectively perform work tasks.

[0004] What is needed are systems and methods for acquiring information about the aerial work environment and for automating certain functions of the work, thereby eliminating the need for a human operator. Summary of the invention

[0005] Aspects, features, and advantages of the presently disclosed subject matter are set forth in part in the following description. Further aspects and advantages may be apparent from the description or by practicing the presently disclosed subject matter.

[0006] In a first embodiment, the present disclosure is directed to a robotic system for performing an automated task in an operating environment of an aerial device. The robotic system includes: at least one processor; a robotic unit including at least one manipulator configured to perform a task in the operating environment; a camera for acquiring an image of the operating environment; a sensor associated with the robotic unit; and one or more non-transitory computer-readable media storing computer-readable instructions that, when executed by at least one processor, perform a method for completing a task. The method includes: acquiring information indicating a location of an object in the operating environment by a camera; acquiring sensor data from a sensor, wherein the sensor data indicates a state of at least one manipulator; and performing a task based on the location of the object and the sensor data.

[0007] In a second embodiment, the present disclosure is directed to a method for performing an automated task by a robotic unit in an operating environment of an aerial device. The method includes: acquiring information indicating a location of an object in the operating environment via a camera; acquiring sensor data via a sensor associated with the robotic unit, wherein the sensor data indicates a state of at least one manipulator of the robotic unit; and manipulating the object via the at least one manipulator to complete the task, wherein the manipulation is based at least in part on the location of the object and the sensor data from the sensor.

[0008] In a third embodiment, the present disclosure is directed to a robotic system for performing an automated task in an operating environment of an aerial device. The robotic system includes: at least one processor; a robotic unit including at least one manipulator configured to perform a task in the operating environment; a camera for acquiring an image of the operating environment; a sensor associated with the robotic unit; and one or more non-transitory computer-readable media storing computer-readable instructions that, when executed by at least one processor, perform a method for completing a task. The method includes: acquiring information indicating a location of an object in the operating environment by a camera; acquiring sensor data from a sensor, wherein the sensor data indicates a state of at least one manipulator; and manipulating the object by at least one manipulator to complete the task, wherein the manipulation is based at least in part on the location of the object and the sensor data from the sensor.

[0009] In a fourth embodiment, the technology described herein relates to the robot system described in Embodiments 1 and 3, wherein the computer executable instructions are further executed to perform the following operations: receiving a first input from a user indicating a task to be performed, and receiving a second input from the user indicating to start execution of the task.

[0010] In a fifth embodiment, the technology described herein relates to the robotic system described in Embodiments 1, 3, and 4, wherein computer executable instructions are further executed to perform the following operations: generate an inertial coordinate system including coordinates of an object, determine tool coordinates of a tool connected to at least one manipulator, and autonomously perform a task based on the position of the object and the tool coordinates in the inertial coordinate system.

[0011] In a sixth embodiment, the technology described herein relates to the robotic system described in Embodiment 1 and Embodiments 3 to 5, wherein computer executable instructions are further executed to perform the following operations: classifying objects through a machine learning algorithm, and selecting tasks to be performed based on the classification and tool selection.

[0012] In a seventh embodiment, the technology described herein relates to the robotic system described in Embodiment 1 and Embodiments 3 to 6, further comprising a cantilever assembly comprising a cantilever tip supporting at least one manipulator, wherein computer executable instructions are further executed to perform the following operations: detecting a cantilever tip position, generating an inertial coordinate system comprising cantilever tip coordinates, determining position coordinates indicating an operating position for at least one manipulator, and autonomously moving the cantilever tip to position at least one manipulator at the operating position.

[0013] In an eighth embodiment, the technology described herein is directed to the robotic system described in the first embodiment and the third to seventh embodiments, wherein the computer executable instructions are further executed to receive the operation position as a waypoint uploaded by a user.

[0014] In a ninth embodiment, the technology described herein relates to the robotic system described in embodiments one and three to eight, wherein computer executable instructions are further executed to: receive a cantilever tip position via a position sensor, move at least one manipulator to a waypoint based at least in part on the position from the position sensor, and move at least one manipulator to an operating position using a camera.

[0015] In a tenth embodiment, the technology described herein is directed to the robotic system described in Embodiments 1 and 3 to 9, wherein the robotic unit is electrically insulated from a base of the aerial device.

[0016] In an eleventh embodiment, the technology described herein relates to the method described in the second embodiment, wherein the task is manipulating one of a tie wire, a bolt, a nut, a capacitor, or a transformer.

[0017] In a twelfth embodiment, the technology described herein relates to the method described in the second and eleventh embodiments, further comprising generating an inertial coordinate system, wherein the state of at least one manipulator includes at least a position in the inertial coordinate system.

[0018] In a thirteenth embodiment, the technology described herein is directed to the methods described in embodiments two and eleven to twelve, further comprising applying inverse kinematics to move the robotic unit to autonomously perform a task.

[0019] In a fourteenth embodiment, the technology described herein relates to the method described in Embodiment Two and Embodiments Eleven to Thirteen, further comprising receiving a waypoint indicating coordinates in an inertial coordinate system, and further comprising controlling an actuator to move the robot unit to the waypoint to perform a task.

[0020] In a fifteenth embodiment, the technology described herein relates to the methods described in embodiment two and embodiments eleven to fourteen, wherein the camera is a first camera and is connected to a robotic unit, and further includes acquiring an image from a second camera connected to an end of a pole away from the robotic unit, wherein the image is outside the field of view of the first camera.

[0021] In a sixteenth embodiment, the technology described herein relates to the methods described in the second embodiment and the eleventh to fifteenth embodiments, further comprising generating a three-dimensional representation of an object, and autonomously performing a task.

[0022] These and other aspects, features and advantages thereof may be further understood with reference to the following description, drawings and appended claims.This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, in which:

[0024] Figure 1 depicts aerial devices associated with some embodiments;

[0025] Figure 2 depicts an exemplary system architecture of a robotic cell including a control system and manual controls associated with some embodiments;

[0026] Figure 3 depicts an exemplary six-degree-of-freedom robotic unit associated with some embodiments;

[0027] Figure 4 Depicting an exemplary teleoperation system associated with some embodiments;

[0028] Figure 5 depicts an exemplary flow chart for driving cantilever operations related to some embodiments;

[0029] Figure 6 An embodiment of a robotic unit performing a task is depicted;

[0030] Figure 7 An embodiment of an aerial device for moving a boom assembly between locations is depicted;

[0031] Figure 8 Describing embodiments of object detection and alignment;

[0032] Fig. 9A and Fig. 9B depicting some embodiments of a grappling hook;

[0033] Fig.10depicts a clamp that may be used in some embodiments; and

[0034] Fig.11 depicts the process of performing tasks autonomously or semi-autonomously;

[0035] Fig.12 A hardware platform for some embodiments is depicted.

[0036] The accompanying drawings illustrate example embodiments of the presently disclosed subject matter. The claims are not limited to the example embodiments depicted in the accompanying drawings. The aspects and features depicted in the accompanying drawings are not necessarily to scale. Reference numerals used repeatedly in the specification and drawings represent the same or similar aspects or features. DETAILED DESCRIPTION

[0037] The following detailed description refers to the accompanying drawings, which illustrate example embodiments of the presently disclosed subject matter. The present disclosure, including the example embodiments shown in the accompanying drawings, describes the features, aspects and advantages of the disclosed subject matter by way of explanation and not limitation. Various modifications, combinations and variations may be made to the example embodiments or aspects or features thereof without departing from the scope of the presently disclosed subject matter. Therefore, the present disclosure covers such modifications, combinations and variations. The present disclosure provides sufficient details to enable those skilled in the art to practice the claimed subject matter. The present disclosure should therefore not be considered to have a limiting meaning. The scope of the present invention is defined solely by the appended claims and the full scope of equivalents granted by such claims.

[0038] In this specification, reference to "one embodiment," "an embodiment," or "an embodiment" means that the referenced feature or features are included in at least one embodiment of the presently disclosed subject matter. Individual references to "one embodiment," "an embodiment," or "an embodiment" in this specification do not necessarily refer to the same embodiment and are not mutually exclusive unless so stated and / or unless it would be readily apparent to one skilled in the art from the specification. For example, features, structures, actions, etc. described in one embodiment may also be included in other embodiments, but are not required to be included. Therefore, the scope of the presently disclosed subject matter covers various combinations and / or integrations of the example embodiments in this specification.

[0039] The terms "a", "an" and "the" do not indicate a quantitative limitation, but rather indicate the presence of at least one of the referenced item. The terms "first", "second", "third", etc., may be used interchangeably to distinguish one item from another, and are not intended to indicate the position or importance of the respective items. Range limitations in this specification and claims include all endpoints, and all such endpoints may be independently combined to provide another range limitation.

[0040] The term "coupled" when used herein with respect to at least two objects refers to direct or indirect mechanical or physical contact between two objects, wherein the two objects are linked, connected, fastened or joined to each other, including by an interference fit, one or more fastening elements or hardware, by welding, etc. The term "coupled" includes objects that are removably coupled to each other.

[0041] The terms "integral" or "integrally formed," when used herein with reference to at least an element, refer to two elements that define a portion, part, or fragment of one and the same object and / or two objects, respectively, being permanently coupled to one another so that the objects cannot be separated from one another without destructive means. The term "integral" includes a portion, part, or component of an element that cannot be separated from the entire element without employing destructive means. For example, the term "integral" may refer to objects that are welded together, objects that are formed or cast as a single unit, and portions, parts, or components of a single integral object.

[0042] In general, the present disclosure provides an aerial device, including a robotic unit located on a boom tip of the aerial device, for autonomously or semi-autonomously performing tasks. The robotic unit may include a plurality of manipulators and various sensors for detecting the state of the robotic unit and the operating environment of the robotic unit and the boom assembly of the aerial device. One or more processors may execute computer executable instructions to perform tasks that may be initiated and controlled by an operator or autonomously. The operation of the robot may be based on the detection of the state of the robot and the operating environment, the task to be performed, and the tool to perform the task.

[0043] Figure 1An aerial device 100 associated with some embodiments is depicted. As shown, the aerial device 100 can be connected to a multi-purpose vehicle 102. In some embodiments, as shown, the aerial device 100 includes a boom assembly 104 and a turntable 106 that can be set on the multi-purpose vehicle 102. As shown, the boom assembly 104 may include a lower boom portion 108 connected to the turntable 106 and an upper boom portion 110 pivotally connected to the end of the lower boom portion 108. In some embodiments, either or both of the lower boom portion 108 and the upper boom portion 110 may include a telescopic portion for telescopically extending and retracting the length of the boom assembly 104. Further, in some embodiments, a multi-functional platform connected to the distal end of the upper boom portion 110 may be included. Alternatively or additionally, in some embodiments, a remote component system 300 may be set at the distal end of the upper boom portion 110. In some embodiments, and as described in more detail below, remote component system 300 may include a robotic unit suitable for performing telecommunication repairs, power line repairs, general maintenance work, or other actions that may be performed by a robot. For example, the robotic unit may include one or more multi-function tools for performing actions such as sawing, cutting, screwing, wiring, or other actions associated with repair work. In some embodiments, boom assembly 104 is used to position remote component system 300 at a remote location (e.g., adjacent to an energized power line).

[0044] In some embodiments, the aerial device 100 can be used to perform work on or near high-voltage power lines. Therefore, the aerial device 100 can operate near high-voltage power cables. In some embodiments, the remote component system 300 and the boom assembly 104 include insulating materials for electrically insulating the aerial device 100. In addition, any electrical components arranged in the multi-purpose platform and on the boom assembly 104 can be independent and separated from the electrical components of the multi-purpose vehicle 102. In this way, a dielectric gap is generated between the remote component system 300 and the multi-purpose vehicle 102. In some embodiments, the multi-purpose vehicle 102 can be generally referred to as a base, and can be a vehicle, a crane, a platform, a truck bed, a mechanical tree pruning equipment, a hydraulic lift or any other base capable of supporting the boom assembly 104 and the remote component system 300, as will be described in further detail below.

[0045] In some embodiments, the boom assembly includes one or more cylinders for controlling the movement of the boom assembly 104, such as a lower boom cylinder 114 disposed between the turntable 106 and the lower boom portion 108 and an upper boom cylinder 116 disposed between the lower boom portion 108 and the upper boom portion 110, as shown. In some embodiments, the hydraulic system of the boom assembly 104 can be used to hydraulically actuate the cylinders 114 and 116. However, it is conceivable that other suitable actuation technologies can be used to actuate the embodiments of the cylinders 114 and 116 (e.g., electric actuation, pneumatic actuation, and magnetic actuation). Further, in some embodiments, a combination of different actuation technologies can be used. It is envisioned that the boom assembly 104 includes an embodiment of one or more rotary actuators. For example, in some embodiments, the boom assembly 104 includes a slewing drive for controlling the rotation of the corresponding joints of the boom assembly 104.

[0046] In some embodiments, the lower boom cylinder 114 can control the rotation angle of the lower boom portion 108 relative to the turntable 106. Similarly, the upper boom cylinder 116 can control the rotation angle of the upper boom portion 110 relative to the lower boom portion 108. Additionally, in some embodiments, a pivotable connection 118 can be included between the distal end of the upper boom portion 110 and the remote assembly system 300 for controlling the angle of the remote assembly system 300. In some such embodiments, the pivotable connection 118 can be configured to automatically maintain an upright orientation of the remote assembly system 300. For example, the pivotable connection 118 can include one or more gyroscopes and / or be connected to a control system for maintaining an upright orientation of the remote assembly system 300, such that the remote assembly system 300 remains in an upright position regardless of the orientation of the rest of the boom assembly 104. Additionally or alternatively, embodiments are contemplated in which the orientation of the remote assembly system 300 can be manually controlled by an operator using one or more input devices.

[0047] Figure 2An exemplary block diagram 200 related to an embodiment of the present disclosure is depicted. In some embodiments, the remote component system 300 includes various components, sub-components, parts or components for collecting sensory information and / or for performing actions such as repair work in a telecommunications environment. The remote component system 300 may include various circuits, parts or other components for collecting sensory information, including video, three-dimensional depth information, audio and other sensory data. Further, the remote component system 300 may include a manually controlled or autonomous robotic unit, which may be positioned at the end of the boom assembly 104 for interacting with the work site to perform one or more tasks. For example, as described above, in many real-life scenarios, the tasks to be performed may not be discovered until arriving at the work site, and therefore, the robotic unit may include various tools, features or functions to respond to various different tasks. Additionally, as described in more detail below, the remote robotic assembly may further include one or more parts, components or features for providing sensory information to the operator, providing the operator with additional information about the work site to improve efficiency, efficacy and / or safety of the remote component system 300 and the operator.

[0048] As shown in block diagram 200, remote assembly 202 includes at least a remote acquisition device 210, a computer 260, and a control system 280. In some embodiments, and as described in more detail herein, remote acquisition device 210 may be a device configured and adapted to acquire sensory information and may be located on a robotic unit for acquiring sensory information that may be used by computer 260 to, among other purposes, present the information to an operator via a control system. Figure 2 Exemplary sensors, cameras, and other devices that can collect sensory information via the remote collection device 210 are depicted. As described in more detail below, the remote collection device 210 can be mounted or positioned on a selectively movable mount or portion of a robotic unit. For example, the robotic unit can be a robotic unit positioned at the end of a boom assembly for aerial applications. However, the remote collection device 210 can also be used with a robotic unit that is not mounted on a boom assembly, for example, it can be used with a robotic unit for ground applications, or connected to a robotic arm or an aerial drone. Therefore, via the robotic unit, the remote collection device 210 can collect sensory information.

[0049] Through selective input, including manually input instructions and / or automated instructions, the remote acquisition device 210 can capture video, still images, three-dimensional depth information, audio, conductivity, voltage, and other information that can be collected by sensors or recording devices. For example, the remote acquisition device 210 may include a camera 212 for capturing video or still images (collectively referred to as "video"). In some embodiments, the camera 212 can be at least one camera or multiple cameras. The camera 212 can be a camera located on the remote acquisition device 210, which is used to capture video within a selected field of view. The resolution of the video captured by the camera 212 may vary, but in some embodiments, the camera 212 can be a camera configured to capture at least 720p resolution, but can be captured at higher resolutions, including but not limited to 1080p, 2K, 4K or 8K resolution. However, it should be understood that the camera 212 can be any currently known or yet to be discovered camera for capturing video. The video captured from the camera 212 can be stored locally in the local memory 214 of the remote acquisition device. The local memory 214 can be the following about Fig.12 Storing video at local storage 214 can help provide fail-safe or backup storage of captured video in the event of a transmission or upload failure. Further, storing video at local storage 214 can help in situations where the wireless connection is poor, or where the direct line is loose or interrupted and the captured video cannot be immediately transmitted. Alternatively or additionally, the video captured from the camera 212 can be transmitted to the computer 260 for processing, analysis, storage and / or for later transmission to the control system 280. In further embodiments, the video captured from the camera 212 can be transmitted directly to the control system 280 for processing.

[0050] In some embodiments, the remote acquisition device 210 may further include at least one 3D camera 216 or other device configured to collect 3D depth information. As described in more detail below, the 3D camera 216 can be used to collect 3D depth information within the field of view to create a point cloud, a 3-D model, or other digital representation of an object or area scanned or viewed by the 3D camera 216. The 3D camera 216 can operate in conjunction with the camera 212 or other components or parts of the remote component 202 and / or the remote acquisition device 210, or operate independently of them. As described in more detail below, in response to an instruction or input, the 3D camera 216 can begin to collect 3D depth information about an object or area within the field of view. Similar to the video collected by the camera 212, the 3D depth information collected by the 3D camera 216 can be stored locally at the memory 214. In some embodiments, the remote acquisition device 210 may include a separate memory 214 for the video collected by the camera 212 and a separate memory 214 for the 3D information collected by the 3D camera 216. As described in more detail below, the remote acquisition device 210 may include a microphone 218 and / or a sensor 220, which may be one or more or a plurality of sensors, for acquiring additional sensory information. Therefore, in some embodiments, a separate and different memory 214 may be used for each sensory acquisition device (i.e., camera 212, three-dimensional camera 216, microphone 218, and / or sensor 220). In further embodiments, the remote acquisition device 210 may include a single memory 214 for storing all acquired sensory information. As described above and in further embodiments, in addition to or instead of being stored locally at the memory 214, the three-dimensional information may be sent directly to the computer 260.

[0051] In addition to capturing video and / or three-dimensional information, it may also be advantageous for the remote capture device 210 to capture additional sensory information that can be presented to an operator or processed by the computer 260. For example, in some scenarios, it may be advantageous for the remote capture device 210 to capture audio via at least one microphone 218. Continuing with the above example, the remote component 202 for telecommunications maintenance can utilize audio information for diagnostic or safety purposes. For example, the audio information can capture sounds at a job site, and the audio information can be processed to determine whether the job site is safe. Therefore, in some embodiments, the remote capture device 210 may include at least one microphone 218 for capturing audio information. Similar to the video and three-dimensional information described above, the captured audio information can be stored locally at the memory 214 and / or transmitted to the computer 260 and / or the control system 280.

[0052] Similar to the audio information, the remote acquisition device 210 may further include a sensor 220 for collecting additional sensory information, metrics, or data. For example, continuing with the above example, the remote acquisition device 210 can be used with a remote component 202 positioned at the end of the boom assembly 104 for telecommunications or power line work. In such a work application, the remote component 202 can work on or near a live power line or other conductive line that transmits power. Therefore, in some embodiments, the remote acquisition device 210 may include a sensor 220 configured as an electrical sensor for determining whether a cable or power line has current passing through it. However, it should be understood that the remote acquisition device 210 may include additional sensors configured and adapted to provide additional information to the remote acquisition device and / or the remote component 202. As a non-limiting example, the sensor may be a sensor 220, which may include any of the following sensors: a gyroscope, an accelerometer, a thermometer, a barometer, a light emitter, and other sensors that may be used in the intended application of the remote component 202.

[0053] In some embodiments, the remote component 202 may further include at least one digital hub 222. In some embodiments, the remote component 202 further includes at least one digital hub 222. The digital hub 222 may receive the collected sensor information from the remote collection device and convert the collected sensor information into a format suitable for transmission to the computer 260 and / or the control system 280. In some embodiments, the digital hub 222 is a USB hub (e.g., USB 3.0).

[0054] like Figure 2As further shown, the remote component 202 may further include a controller 224. In some embodiments, the controller 224 may be a processor or other circuit or computer hardware for receiving commands or instructions from the control system 280 and / or the computer 260 and for relaying or providing commands to the remote acquisition device 210 and / or the motion control 230. Therefore, in some embodiments, instructions or commands from the controller 224 may be sent to the remote acquisition device 210. For example, the instructions sent from the controller 224 to the remote acquisition device 210 may include instructions to start recording video via the camera 212. However, it should be understood that the instructions sent from the controller 224 may cause any component of the remote acquisition device 210 to start collecting sensory information, including but not limited to three-dimensional information, audio information, or other sensory information collected by the sensor 220 of the remote acquisition device 210. In addition, the controller 224 may be used to send instructions to cause the remote component 202, the remote acquisition device 210, and / or the motion control 230 to perform other actions corresponding to the instructions. For example, the instructions from the controller 224 may instruct the remote acquisition device 210 to store the collected sensory information on the memory 214. Additionally, instructions from controller 224 may be sent to motion control 230 to instruct remote component 202 to perform a movement. Further, controller 224 may communicate with transceiver 244 for communicating with computer 260 and / or control system 280 to send sensory information or other data or information to computer 260 and / or control system 280. Similarly, controller 224 may be further configured to receive instructions, commands, or other information from computer 260 and / or control system 280.

[0055] like Figure 2 As further described in the block diagram of and in some embodiments, the remote component 202 may further include motion controls 230. The motion controls 230 may be configured and adapted to control the movement of the remote component 202, including any multi-function arm or camera support as described in more detail below. In some embodiments, the remote component 202 may include a 6DOF robotic unit configured with a multi-function arm and / or camera support capable of moving in 6DOF. Therefore, the motion controls 230 may be configured to provide instructions or commands to the remote component 202 to move in 6DOF. In some embodiments, the motion controls may include an x-axis control 232, a y-axis control 234, a z-axis control 236, a pitch control 238, a yaw control 240, and / or a roll control 242 for moving the remote component 202 in 6DOF. However, it should be understood that the remote component 202 may include different designs and, in some embodiments, may move in less than 6DOF. Therefore, in further embodiments, the motion controls 230 may include controls configured and adapted to move the remote component 202 in an appropriate number of planes.

[0056] As described above, motion control 230 can communicate with controller 224. Instructions or commands from controller 224 can be sent to motion control 230. When an instruction is received, corresponding controls 232, 234, 236, 238, 240, and / or 242 can be instructed to cause movement of remote component 202 based on the received instruction. As described above, one or more arms or limbs of remote component 202 can be configured to move with 6DOF. Based on the instruction, the corresponding motion control 230 can cause the movement of remote component 202 to correspond to the instruction.

[0057] As described above, remote component 202 can be communicatively coupled to computer 260. In some embodiments, computer 260 can be directly coupled to remote component 202, such that computer 260 and remote component 202 are a combined system. For example, computer 260 can be directly mounted into a frame or body of remote component 202. Thus, remote component 202 and computer 260 can communicate directly via a cable or other direct method. In further embodiments, computer 260 can be located outside of remote component 202. When located outside, remote component 202 and computer 260 can still be communicatively coupled. For example, in some embodiments, remote component 202 and computer 260 can be coupled via a physical connection, such as an Ethernet cable or a USB cable. In further embodiments, remote component 202 and computer 260 can be coupled via a wireless connection, such as WIFI, Cellular Connection or Another Wireless Connection. In embodiments where the computer 260 and the remote component 202 are connected via a wireless connection, the transceiver 244 may communicate with another transceiver 250 that is coupled or otherwise in communication with the computer 260 .

[0058] In some embodiments, the computer 260 can receive and process sensory information collected by the remote collection device 210 of the remote component 202. Therefore, the computer 260 can include at least a processor 262 for executing commands, which can include instructions for processing, analyzing, or utilizing the collected sensory information. For example, as described in more detail below, the computer 260 can utilize the collected three-dimensional information to generate a point cloud, a three-dimensional model, or other digital representation of an object or area collected by the remote collection device 210.

[0059] In some embodiments, control system 280 may be an interface, device, or system that provides an interactive medium for a user to interact with computer 260 and / or remote component 202. For example, in some embodiments, control system 280 may include at least one processor 282, at least one controller 284, at least one display 288, at least one sensor 290, and at least one transceiver 292. As described in more detail below, some embodiments of the present teachings provide a method of controlling remote component 202 from a remote location. Continuing with the above example, telecommunication maintenance or power line maintenance is sometimes performed during or immediately after a severe weather storm. Such conditions may present hazards such as exposed live wires, strong winds, lightning, and other hazards that pose a risk to human workers. Therefore, it may be advantageous for the operator of remote component 202 to control remote component 202 from a safe location, such as in a work truck or building away from the work site. Therefore, control system 280 may include at least one controller 284 that provides an interactive device for a user to enter commands or instructions for controlling or manipulating remote component 202. Controller 284 may be any interface for entering commands or instructions that can be transmitted and processed by a computer or other hardware. Reference below Figure 4An exemplary embodiment of controller 284 is provided. However, it should be understood that the depicted embodiments are intended to be illustrative, not limiting. As a non-limiting example, controller 284 may include a handheld motion control controller. As described in more detail below, the motion control controller may facilitate an operator to perform a specific movement or action that may be captured and relayed to remote component 202 for execution. Through the use of the motion control controller, the operator may be provided with a sensory effect similar to being at the work site and performing the action himself. However, controller 284 is not limited to motion controls, but rather, controller 284 may be any interface for an operator to input instructions or commands for remote component 202. For example, in a further embodiment, controller 284 may be a handheld controller similar to a video game controller including a thumbstick, button, trigger, and / or other interface inputs. In a further embodiment, controller 284 may include a joystick and button design. In a further embodiment, controller 284 may be a mouse and keyboard. In a further embodiment, controller 284 may be configured as an interactive model of a glove or hand, allowing an operator to perform native hand operations that may be captured and transmitted to remote component 202. In further embodiments, the controller 284 may include a camera assembly or other motion acquisition assembly for collecting the movement of the operator. For example, in addition to or in place of the physical controller manipulated by the operator, the camera assembly can collect the movement of the operator. The collected movement can be transmitted to the computer 260 for conversion or mapping of the movement of the remote component 202. Alternatively or additionally, motion acquisition auxiliary tools such as motion acquisition points can also be used to collect the movement of the operator. In some embodiments, it is understood that the examples provided herein are intended to be illustrative, not restrictive, and the controller 284 can be any device or method for receiving instructions or inputs from the operator.

[0060] In some embodiments, the control system 280 may further include a power medium 286 for powering one or more parts or components of the control system, including, for example, the controller 284, the display 288, or at least one sensor 290, or any combination thereof. In some embodiments, a single power medium may power all parts or components of the control system 280. In further embodiments, various parts or components of the control system 280 may include separate and different power media 286. For example, a first power medium 286 may be used to power the controller 284, while a second power medium 286 may be used to power the display 288. The power medium 286 may be any conventionally known power source for providing power to electrical devices, including but not limited to an internal power source such as a battery, or an external battery source such as an electrical outlet.

[0061] like Figure 2 As further shown, the control system 280 may further include at least one display 288. In some embodiments, refer to the following Fig.12 , the display 288 can be a monitor, a touch screen, a television screen, or other display. In some embodiments, at least a portion of the sensory information collected from the remote collection device 210 can be displayed on the display 288 for the operator to view. For example, the collected video can be displayed on the display 288. When the remote component 202 is remotely operated, providing the sensory information on the display 288 can provide the operator with a more immersive feeling. With real-time video feedback, the operator can experience the work site as if the operator was there in person, even if the operator is in a safe location miles away. Additionally, providing the sensory information to the operator via the display 288 can help the operator input instructions or commands via the controller 284.

[0062] In some embodiments, the control system 280 may further include at least one sensor 290 that may provide additional sensory influences to the operator and / or collect additional input that may be used by the computer 260 to provide instructions to the remote component system 300. In some embodiments, one or more sensors may be combined with the controller 284 and / or one or more sensors may be combined with the display 288. For example, in some embodiments, the sensor 290 may be at least one speaker or sound-generating device to provide the operator with audio information collected from the remote collection device 210 or pre-recorded or pre-rendered audio. In further embodiments, the at least one sensor 290 may be one of an accelerometer, a gyroscope, a light sensor, or any other type of sensor 290 suitable for detecting the user's perspective or the movement, position, or angle of the operator's body.

[0063] In some embodiments, and as described in more detail below, an operator can use controller 284, display 288, and at least one sensor 290 to provide instructions to remote component 202, which can be analyzed and converted into instructions to cause remote component 202 to move or perform an action. As also described in more detail below, the operator can input instructions or commands through control system 280. In some embodiments, input can be input or collected through a combination of controller 284 and display 288. For example, display 288 can be connected to a head-mounted unit, as described in more detail below. The operator can move their head or torso, and sensor 290 collects the operator's movements and / or perspective. The collected movement data or perspective can be sent to computer 260 via transceiver 292, and computer 260 can obtain the collected movement data or perspective and convert it into instructions for causing remote component 202 to move and imitate or copy the operator's movements and match the operator's perspective.

[0064] Figure 3 is an exemplary embodiment of a remote component system 300. In some embodiments, the remote component system 300 may include various components, sub-assemblies, parts or components, including but not limited to a robotic unit 302 fixed to the end of the boom assembly 104. Further, the remote component system 300 may correspond to the above-mentioned Figure 2 The remote assembly 202 described herein may include any and all components or parts as described above. In some embodiments, the robotic unit 302 may be configured and adapted to receive instructions from a computer or an operator to perform corresponding movements or actions. In some embodiments, the robotic unit 302 may be a fully manually controlled robot, wherein the robotic unit 302 will not perform movements or actions without instructions provided by an operator. In further embodiments, the robotic unit 302 may be a fully automated robot, wherein the robotic unit 302 performs actions or movements based on pre-programmed instructions for automation. In further embodiments, the robotic unit 302 may be a robot configured to respond to both manually input instructions and automated programming. Therefore, the various movements or actions performed by the robotic unit 302 and described herein may be performed based on manually provided instructions and / or automated programming.

[0065] As mentioned above and Figure 3 As shown, in some embodiments, the remote component system 300 can be positioned at the distal end of the boom assembly 104. As shown, in some embodiments, the distal end of the boom assembly 104 can include a pivot joint 130, which includes a motor 132. In some embodiments, the pivot joint 130 can be used to change the angle or position of the remote component system 300. In further embodiments, the pivot joint 130 can be paired with a sensor such as a gyroscope to help maintain the horizontal position of the remote component system 300. Figure 3As further shown, the pivot joint 130 may further serve as a connection point between the remote component system 300 and the boom assembly 104. For example, the base 150 may be coupled to the pivot joint. The base 150 may be adapted and configured to receive and couple the remote component system 300. Thus, through such coupling, the remote component system 300 may be secured and coupled to the boom assembly 104. In some embodiments, the base 150 may include a generally planar design for receiving and securing one or more components, sub-assemblies, parts or components of the remote component system 300. Further, the size and shape of the base 150 may vary and may depend on the design of the remote component system 300. Further, in some embodiments, the base 150 may further include a motorized turntable 152. The motorized turntable 152 may be a powertrain for rotating the base 150. The rotation of the base 150 may be advantageous for positioning the remote component system 300 during use.

[0066] In some embodiments, the remote component system 300 may generally include a robotic unit 302. The robotic unit 302 may be a controllable robotic unit capable of performing a range of movements and actions, such as performing maintenance work in a telecommunications environment. In some embodiments, and as described in more detail below, the robotic unit 302 may be a 6DOF robotic component that is configured and adapted to mimic the movements of an operator using a VR controller. Specifically, with a 6-DOF configuration, the robotic unit 302 may substantially mimic the torso, neck, and arm movements of an operator. With such movement, the robotic unit 302 may perform a greater range of movement and / or provide an operator with a more immersive experience than existing systems.

[0067] In some embodiments, the robotic unit 302 may include a central hub 304. The central hub 304 may be a central housing or base that may house a processor, power supply, circuitry, wireless communication devices, and other electronic components for the robotic unit 302, including the components described above with reference to FIG. Figure 2 Additionally, the central hub 304 can serve as a coupling or connecting member to secure the robotic unit 302 to the base. Further, the central hub 304 can also serve as a receiving point for one or more parts or components of the robotic unit 302. For example, as described below, the robotic unit 302 can include at least one multi-function arm and at least one camera bracket. Thus, the central hub 304 can accommodate and couple with at least one multi-function arm and at least one camera arm.

[0068] To collect sensory information, including but not limited to video and three-dimensional depth information, the robotic unit 302 may include at least one camera support 310. The camera support 310 may be a 6DOF, selectively controllable robotic arm that may be coupled to the central hub 304. As described in more detail below, the robotic unit 302 may receive movement instructions or commands from the computer 260 to move or change the position of the camera support 310. For example, the camera support 310 may correspond to a head-mounted support or other acquisition device to acquire the operator's perspective. The instructions or commands may be relayed to the robotic unit 302 to move the camera support 310 in a corresponding manner to match the operator's perspective. To enhance the operator's experience, the camera support 310 may include a plurality of camera support segments 312 that may be separated by motorized pivotable joints 314. The number and size of the camera support segments and pivotable joints 314 may vary depending on the embodiment and application of the robotic unit. Typically, in response to an instruction or command, one or more pivotable joints 314 may be activated to rotate or move the camera support 310. In some embodiments, the pivotable joint 314 can be used to move the camera support 310 in the X-axis, Y-axis, and Z-axis, as well as to control the roll, pitch, and yaw of the camera support 310. Thus, by moving in 6DOF, the camera support 310 can mimic or replicate the operator's perspective. Figure 3 As further shown, the distal end of the camera support 310 may further include a sensor acquisition device. In some embodiments, the sensor acquisition device generally includes at least one camera, a three-dimensional camera and / or a sensor for acquiring sensor information.

[0069] As described above, the robotic unit 302 may be adapted to perform tasks or actions for repair work, maintenance work, or other similar situations. To perform these actions, the robotic unit 302 may include at least one multifunctional arm. Figure 3The illustrated embodiment shows an exemplary embodiment of a robotic unit 302 that includes two multifunctional arms 330a, 330b. As with the camera support 310 described above, each of the multifunctional arms 330a, 330b may include a plurality of multifunctional arm segments 332 that may be separated by motorized pivotable joints 334. The number and size of the multifunctional support segments 332 and the pivotable joints 334 may vary depending on the embodiment and application of the robotic unit. Generally speaking, in response to an instruction or command, one or more of the pivotable joints 334 may be activated to rotate or move the multifunctional arms 330a, 330b. In some embodiments, the pivotable joints 334 may be used to move the multifunctional arms 330a, 330b along the X-axis, Y-axis, and Z-axis and to control the roll, pitch, and yaw of the camera support 310. Thus, by moving in 6DOF, each multifunctional arm 330a, 330b may mimic or replicate the movement of an operator's arms and hands. In some embodiments, the distal end 336 of the multi-function arm 330a, 330b may include one or more tools, flanges, or other devices for performing actions such as repair work. In some embodiments, the distal end 336 may include an adapter or may be otherwise configured to receive a tool.

[0070] The remote component system 300 may further include a remote power source 350. In some embodiments, the remote power source 350 may be secured to the base. In further embodiments, the remote power source 350 may be located within the central hub 304. The remote power source 350 may be used to power the camera support 310, the multi-function arm 330a, the multi-function arm 330b, or any combination thereof. The remote power source 350 may be a generator, a battery, or any other known power source.

[0071] In further embodiments, in addition to the sensors 410 described below, the robot unit 302 may include one or more additional acquisition devices or sensors 360 for acquiring additional information that can be analyzed and / or presented to a user or operator. For example, in some embodiments, the robot unit 302 may include a thermometer or thermal sensor for acquiring thermal information. In some embodiments, the robot unit 302 may include an electrical sensor for acquiring electrical data. For example, the robot unit 302 may be used to work on power lines or in other scenarios involving live power lines or other live wires or circuits. Therefore, in order to avoid damaging the robot unit 302, the boom assembly 104, or the multi-purpose vehicle 102, at least one sensor 360 may be a sensor for detecting current. Additionally, the robot unit 302 may include at least one sensor 360 that is an accelerometer, a gyroscope, a light sensor, or other sensors for detecting the positioning of the camera bracket 310, the multi-function arm 330a, and / or the multi-function arm 330b. As described in more detail below, sensors for detecting the positioning of the robot unit 302 may help to use motion controls to replicate or imitate the movements of an operator.

[0072] In some embodiments, Figure 3 As shown, in addition to the robot unit 302, the boom assembly and remote assembly system 300 may further include at least one heavy multifunctional arm 390 or an additional robot assembly that can be operated alone or in combination with the robot unit 302. For example, in many robot applications, delicate balance is usually considered when designing the features and functions of the robot. Typically, robots adapted and configured for fine work and fine adjustments are usually unable to transport or carry heavy objects. On the contrary, robots adapted and configured for carrying or transporting heavy objects usually lack structural components for performing fine or fine-tuning actions. As a non-limiting example, in telecommunications maintenance, heavy parts may need to be lifted from the ground to a telecommunications pole. Lifting heavy parts may require a robot configured for transporting heavy objects. However, once in place, the part may require a robot configured for fine or complex operations to install the part in place. The embodiments of the present disclosure solve this dilemma by pairing a robot configured and suitable for fine-tuning and / or fine work with a robot configured and suitable for carrying or transporting heavy objects. For example, in some embodiments, the robotic unit 302 can be configured and adapted to perform movements or actions for complex, delicate or fine-tuned work, such as loosening wires, cutting wires, loosening screws and bolts. In some embodiments, the remote component system 300 can include a heavy-duty multi-function arm 390 for holding or transporting heavy loads that may be too heavy for the robotic unit 302 to safely hold and transport. Thus, through the combination of the robotic unit 302 and the heavy-duty multi-function arm 390, the remote component system 300 can perform both dexterous and load-bearing actions.

[0073] In general, the term "manipulator" as used in the embodiments of the present invention may include the heavy multi-function arm 390, the first robotic arm 424 and the second robotic arm 428, the boom assembly 104, or any other articulated arm that can be used to manipulate components of the aerial device 100 or objects used to perform work.

[0074] Figure 4 An exemplary teleoperation system 400 relating to some embodiments is depicted. In some such embodiments, an operator 402 may be equipped with at least one user device 404. In some embodiments, the user device 404 includes a computer or other computing device. The user device 404 may include or be associated with a head-mounted display such as a virtual reality headset. In some embodiments, the user device 404 includes or is connected to at least one display 406 and one or more speakers 408. For example, the display 406 may be disposed in front of the headset, and the speakers 408 may be positioned on either side of the headset so that stereo audio may be provided. Additionally, in some embodiments, the user device 404 may include a head-mounted sensor 410 disposed on or in the user device 404. For example, in some embodiments, at least one sensor 410 may include any combination of an accelerometer, a gyroscope, or an angular position sensor for measuring an angle or angular position change of the user device 404.

[0075] In some embodiments, the user device 404 may further include one or more controllers, such as a first position-sensitive controller 412 and a second position-sensitive controller 414, as shown. In some embodiments, each of the first position-sensitive controller 412 and the second position-sensitive controller 414 may be configured to be held in the hand of the operator 402 for reading the position of the operator 402's hand. In some such embodiments, the first position-sensitive controller 412 and the second position-sensitive controller 414 may incorporate one or more position-sensitive sensors, such as any of an accelerometer, a gyroscope, a potentiometer, a load cell, and other sensors suitable for measuring the position and angular position of the operator 402's hand. In some embodiments, the user device 404 includes any number of joysticks, such as any combination of single-axis or multi-axis joysticks. For example, one or more 3-axis joysticks or 6-axis joysticks may be used to control the movement of any of the robot assembly 420 or the boom assembly 104. In some embodiments, a 6-axis joystick may be used to control the movement of six degrees of freedom. For example, in some embodiments, the position-sensitive controllers 412 and 414 may include a 6-axis joystick or other multi-axis control device.

[0076] In some embodiments, the user device 404 is connected to or includes at least one processor 416 and at least one data storage 418. For example, in some embodiments, the processor 416 and the data storage 418 may be included internally or externally to the user device 404. Additionally, embodiments are contemplated in which the processor 416 and the data storage 418 may be included externally to another device connected to the user device 404. For example, an external computer may be connected to the user device 404 via a wired or wireless connection.

[0077] In some embodiments, the remote operation system 400 includes a robot assembly 420, as shown, which can be connected to the user device 404. In some embodiments, the robot assembly 420 includes a base 422, as shown. In some embodiments, the robot assembly 420 can be set at the distal end 122 to replace the multi-functional platform or as a supplement to the multi-functional platform. For example, in some embodiments, the base 422 of the robot assembly 420 can be pivotally fixed to the cantilever tip of the cantilever assembly 104. The robot assembly 420 can include one or more robot arms. For example, in some embodiments, the robot assembly 420 includes a first robot arm 424 and a second robot arm 428, the first robot arm 424 includes a first robot arm end 426, and the second robot arm 428 includes a second robot arm end 430, as shown. Embodiments are envisioned in which the robot arms 424 and 428 include any number of sensors (e.g., at least one sensor 290) for measuring the position and angle of one or more joints in each robot arm. Therefore, information related to the position and orientation of the robot arm in three-dimensional space can be monitored.

[0078] Additionally, in some embodiments, the robot assembly 420 includes a camera robot arm 432, which includes one or more cameras 434 and one or more microphones 436 disposed at the end of the camera robot arm 432. In some embodiments, two or more cameras may be included to provide stereo vision, which can improve visual depth. Similarly, two or more microphones 436 may be included to provide stereo audio. In some embodiments, the camera 434 and the microphone 436 may be configured to continuously collect video data and audio data, respectively. Further, in some embodiments, the camera robot arm 432 is configured to rotate and / or pivot around one or more joints to adjust the positioning and angle of the camera 434 and the microphone 436.

[0079] In some embodiments, the robot assembly 420 includes a depth sensor 438 disposed at, for example, the end of the camera robot arm 432, as shown. The depth sensor 438 can be configured to collect three-dimensional range information near the robot assembly 420. In some such embodiments, the depth sensor 438 is operable to collect a three-dimensional scan of a work area associated with the robot assembly 420. Additionally, in some embodiments, the depth sensor 438 collects information related to the positioning of the robot arm. Further, in some embodiments, for example, the depth sensor 438 can be used for object recognition to prevent unintentional collisions of the robot assembly 420 with foreign objects. In some embodiments, the depth sensor 438 can be a lidar sensor or other type of depth sensor. Further, embodiments are envisioned in which multiple depth sensors 438 can be included. For example, an array of lidar sensors can be disposed around the robot assembly 420 to collect a three-dimensional scan of the work area. Additionally, in some embodiments, both a forward lidar sensor and a rearward lidar sensor can be included to generate a three-dimensional scan of the area in front of and behind the robot assembly 420, respectively.

[0080] In some embodiments, the robot assembly 420 is connected to or includes a computing device 439, such as a computer or controller. In some embodiments, the computing device 439 may be the controller 224, the computer 260, or the control system 280 in the case of a local control system. As shown, the computing device 439 may include at least one processor 440 and at least one data storage 442. For example, in some embodiments, the computing device 439 may be included inside or outside the robot assembly 420. Additionally, embodiments are contemplated in which the computing device 439 may be included as an external computing device connected to the robot assembly 420. Such an external computing device may be disposed, for example, near the robot assembly 420 or elsewhere, such as near the user device 404. Thus, embodiments are contemplated in which signal processing occurs at the robot assembly 420, the user device 404, or any combination thereof.

[0081] In some embodiments, a two-way communication connection can be established between the user device 404 and the robot assembly 420. In some embodiments, the robot assembly 420 and the user device 404 can be communicatively connected via any of a non-wired connection such as a fiber optic cable or a wireless connection such as a wireless network connection or a Bluetooth connection. Embodiments are envisioned in which a wire connection such as a conventional cable can be used, however, it may be unsafe to use an electrical communication connection when the aerial device is electrically connected to an energized power line. Therefore, in some embodiments using electrical connection, an alternative communication connection is used. In some embodiments, multiple data signals can be transmitted from the user device 404 to the robot assembly 420, and vice versa. For example, in some embodiments, a controller signal 444 can be transmitted from the user device 404 to the robot assembly 420, the controller signal including information indicating the position and orientation of the first position-sensitive controller 412 and / or the second position-sensitive controller 414. Similarly, a sensor signal 446 including information indicating the position and orientation of at least one sensor 410 can be transmitted from the user device 404 to the robot assembly 420.

[0082] Additionally, in some embodiments, data signals may be transmitted from the robotic assembly 420 to the user device 404. For example, video data 448 including images captured by the camera 434 may be transmitted from the robotic assembly 420 to the user device 404. Similarly, audio data 450 including audio captured by the microphone 436 may be transmitted from the robotic assembly 420 to the user device 404. Further, in some embodiments, scan data 452 may be transmitted from the robotic assembly 420 to the user device 404, the scan data including information indicative of three-dimensional data and / or distance data captured by the depth sensor 438. For example, in some embodiments, computer vision techniques may be applied to the image data and / or the depth data to identify one or more objects within the working area of ​​the teleoperated device.

[0083] In some embodiments, feedback signal 454 may be transmitted from robotic assembly 420 to user device 404. In some such embodiments, feedback signal 454 may include tactile feedback or some other notification from robotic assembly 420. For example, in some embodiments, feedback signal 454 may include instructions for causing either first position sensitive controller 412 or second position sensitive controller 414 to vibrate or generate force feedback for operator 402 in response to a collision or potential collision condition. Further, embodiments are contemplated in which feedback signal 454 causes any of vibration or force feedback within first position sensitive controller 412 and / or second position sensitive controller 414, audible feedback within one or more speakers 408 (such as a ringing alarm), visual feedback within at least one display 406 (such as a flashing light), or any combination thereof. In some embodiments, feedback signal 454 may be transmitted based on one or more triggers. For example, a trigger may be generated when robotic assembly 420 approaches a collision or singularity. In some embodiments, any of the above data signals may be submitted in real time. For example, video data 448 and audio data 450 may be continuously provided from robotic assembly 420 such that image data captured by camera 434 is displayed on display 406 in real-time with minimal delay.

[0084] In some embodiments, one or more signals described herein may be transmitted across dielectric gap 456. In some embodiments, user device 404 may be disposed at a remote location having a different voltage potential than robot assembly 420. For example, robot assembly 420 may be electrically coupled to an energized power line for performing work on or near the energized power line. Thus, robot assembly 420 is maintained at a potential similar to that of the energized power line and is insulated from contact with ground potential. Thus, in some embodiments, conventional electrical communication techniques may be avoided to prevent electric shock. Thus, embodiments are contemplated in which optical cables or wireless connections are used to communicate with robot assembly 420 to prevent electrical signals at ground potential from reaching robot assembly 420.

[0085] In some embodiments, at least one drone 460 may be included. For example, a drone or other autonomous vehicle may be positioned around the working area of ​​the robot assembly 420. In some such embodiments, the drone 460 may include at least one drone sensor 462, such as a camera or a lidar sensor for collecting additional information about the working area. In some embodiments, the at least one drone sensor 462 includes any combination of one or more cameras and one or more lidar sensors. For example, in some embodiments, the drone 460 may hover around the back side of a utility pole to collect additional image data that would not otherwise be visible to the robot assembly. Additionally, in some embodiments, the drone 460 may include a lidar sensor for collecting three-dimensional data of the working area. In some embodiments, the drone 460 may communicate with either the robot assembly 420 or the user device 404 via wired or wireless communications. In some embodiments, the image data collected by the drone 460 may be transmitted to the user device 404.

[0086] In some embodiments, the robotic arms 424 and 428 can be configured to move based on the movement of the position-sensitive controllers 412 and 414. For example, in some embodiments, the end of each robotic arm is positioned relative to the position of the position-sensitive controller such that the first robotic arm end 426 is positioned based on the position of the first position-sensitive controller 412, and the second robotic arm end 430 is positioned based on the position of the second position-sensitive controller 414. Thus, if the operator 402 moves the first position-sensitive controller 412 upward, the first robotic arm 424 will automatically adjust so that the first robotic arm end 426 also moves upward. In some embodiments, the robotic arm can be adjusted by pivoting around one or more joints disposed within the robotic arm. Similarly, in some embodiments, the camera robotic arm 432 can move so that the camera 434 disposed at the end of the camera robotic arm 432 is positioned based on the position of the headset of the user device 404. For example, when the operator 402 moves and tilts his head, the camera robotic arm 432 will move and tilt so that the position of the camera 434 matches the position of the operator's eyes.

[0087] In some embodiments, the position of the robotic assembly 420 can be moved relative to the position of the operator 402. For example, in some embodiments, the operator 402 can temporarily freeze the motion of the robotic assembly 420 to reset the original position of the robotic assembly 420. Thus, the operator 402 can pause the robotic assembly 420 to prevent unintended motion and lock the robotic assembly 420 in place, move their arms to a more comfortable position, and then resume operation, such as by submitting a subsequent resume input, and continue to control the motion of the robotic assembly 420 from a more comfortable position. Here, the operator 402 can comfortably position their arms in a low position while the robotic arms 424 and 428 are raised upward, rather than having to maintain their arms in an uncomfortable raised position for a long time. Further, embodiments are contemplated in which the operator 402 controls the motion of the robotic assembly 420 from a seated position in a remote location, such as in an office chair, in a vehicle seat, or in another location remote from the robotic assembly 420. In some embodiments, requests to pause and resume motion of the robotic assembly may be received as operator input, for example, via the user device 404 , such as through one or more buttons on the controllers 412 and / or 414 .

[0088] Embodiments in which an operator actively selects inputs to pause and resume motion control are discussed above. However, it should be understood that further embodiments are envisioned in which inputs such as operator 402 pressing buttons on position-sensitive controls 412 and 414 can be used to initiate movement of robotic assembly 420. Thus, in some such embodiments, robotic assembly 420 will only move while the buttons are held on the respective controls. Thus, operator 402 can shift the position of the motion control by releasing the buttons and moving the controls to the desired position.

[0089] It should be understood that in some embodiments, a variety of processing options are contemplated. For example, in some embodiments, a first processing stage may occur at or on the robot assembly 420, such as by a processor 440, while a second processing stage may occur at the user device 404, such as by a processor 416. Here, various processing techniques may be applied to the collected signals. For example, the processor 440 of the robot assembly 420 may employ data filtering and smoothing algorithms to smooth the data signals transmitted to the user device 404. Further, in some embodiments, a portion of the data collected by the robot assembly 420 may be stored in a data storage 442. Additionally or alternatively, data processing and storage may occur at the user device 404. For example, the raw data received from the robot assembly 420 may be filtered and converted using the processor 416 of the user device 404.

[0090] In some embodiments, the robotic assembly 420 may be disposed on the boom assembly 104. For example, the robotic assembly 420 may be included at the distal end of the boom at the boom tip of the boom assembly 104. In some embodiments, the robotic assembly 420 may be included in place of or in addition to the multi-function platform. Additionally, embodiments are contemplated in which the robotic assembly 420 may be connected to other equipment, such as directly to the multi-purpose vehicle 102 or to another suitable device not expressly described herein. Further, in some embodiments, the robotic assembly 420 may be included as a stand-alone device.

[0091] Embodiments are contemplated in which at least a portion of the remote operation system 400 described above may be used to control Figure 1 The movement of the boom assembly 104 can be controlled by the operator 402. For example, the joints of the boom assembly 104 can be rotated and adjusted to match the specified position of the boom tip with the speed requested by the operator 402. For example, the operator 402 can use the first position-sensitive controller 412 to control the movement of the boom assembly 104 so that the position change of the first position-sensitive controller 412 is repeated by the boom tip. It should be understood that the position change may not be proportional, and a scaling factor can be used to convert the controller movement into the boom tip movement. For example, a movement of about 3 inches of the first position-sensitive controller 412 can be converted to a movement of about 18 inches of the boom tip with a scaling factor of 6.0. However, it should be understood that other scaling factors can be used, and in some embodiments, the operator can select and adjust the scaling factor during operation. Further, in some embodiments, the scaling factor can be automatically set based on the type of operation performed by the boom assembly 104 and / or the robot assembly 420.

[0092] Embodiments are contemplated in which both the robot assembly 420 and the boom assembly 104 are remotely controllable by one or more operators. Here, the robot assembly 420 and the boom assembly 104 may be controlled simultaneously using separate input devices or using separate portions of the same input device. Additionally, in some embodiments, an operator may be able to switch a mode of a single input device to selectively switch between control of the robot assembly 420 and the boom assembly 104. For example, an operator may select between a robot control mode, a boom control mode, or other suitable operating control mode. For example, in some embodiments, multiple robot assemblies may be included such that the operator 402 may switch between modes of controlling each respective robot assembly.

[0093] In some embodiments, the camera pole 458 includes a first pole portion 466 and a second pole portion 468 connected at an angle, and the camera pole camera 464 can be coupled to the robot unit 302 or disposed on the base 150 adjacent to the robot unit 302. The camera pole camera 464 can be a visible light camera, an infrared camera, a depth camera, a lidar camera, or any other type of camera. The camera pole camera 464 can acquire images or videos of the operating environment of the robot unit 302. The operating environment may include any area where the robot unit 302 or any other related manipulator can perform work. The pole 458 can be extended by a computing device 439 or an operator-controlled actuator to extend to the back side of any object in the operating environment. The pole 458 can be extended, and in some embodiments, the pole 458 can include joints and actuators to move around the work object to image all sides of the work object. This provides a complete 360-degree view of the operating environment and any components therein.

[0094] In some embodiments, images from camera 434, depth sensor 438, and camera pole camera 464 may be used to generate a representation of operating environment 600 ( Figure 6 ) of the three-dimensional operation model. The operation model may include an inertial coordinate system 702 ( Figure 7 ) in the object. This concept is used in the following and Figure 7 Discussed in more detail in .

[0095] In some embodiments, drone 460 may include a camera, which may be a visible light camera, an infrared camera, a depth camera, a lidar camera, or any other type of camera. Drone 460 may fly around the operating environment of remote component system 300 to image the operating environment. Drone 460 may be controlled by an operator on the ground or at a remote location. In some embodiments, drone 460 may be controlled by manual controls 412, 414, or any other controller. In some embodiments, drone 460 may be autonomous and may fly along a predefined flight path around the operating environment to collect images required to generate a three-dimensional operating model of the operating environment.

[0096] Figure 5An exemplary flow chart 500 for driving a manual boom operation relating to some embodiments is depicted. In some embodiments, at an operator input stage 502, one or more operator inputs are received for each of three Cartesian coordinate directions. For example, a speed input value may be received for each of the X direction, the Y direction, and the Z direction. Additionally, embodiments are envisioned in which speeds may be input and calculated for any number of directions. Further, in some embodiments, speeds for any combination of linear directions and rotational axes may be included. For example, in some embodiments, inputs for motion control systems having six degrees of freedom may be received. In some such embodiments, operator inputs may be requested by an operator via an input device such as a joystick or other controller. In some embodiments, a saturation reduction stage 504 may be included in which a percentage reduction is received to reduce the requested speed value if at least one of the requested values ​​is saturated, as will be described in further detail below. Thus, the speed associated with the motion path may be monitored, and if the speed exceeds a predetermined threshold, the speed may be reduced for each of the three Cartesian directions.

[0097] In some embodiments, a joint velocity calculation stage 506 may be included in which joint velocities of each joint of the cantilever assembly 104 are calculated based on the requested velocity values ​​and the Jacobian matrix of the cantilever assembly 104. For example, the joint velocities of each of the turntable 106, the lower cantilever cylinder 114, and the upper cantilever cylinder 116 may be calculated. In some embodiments, the joint velocity calculation stage 506 provides a motion path including a plurality of joint velocities for the cantilever assembly 104 to achieve a requested Cartesian velocity at a predetermined point on the cantilever assembly 104. For example, in some embodiments, a motion path may be generated to achieve a specific velocity vector or position vector of the cantilever tip.

[0098] Further, in some embodiments, a feedback sensor input stage 508 may be included for receiving one or more position inputs indicating the measured current position of the cantilever assembly 104. In some embodiments, the position input may be received from one or more sensors disposed within the joints of the cantilever assembly 104. For example, any one of a rotary encoder, a linear potentiometer, an angular potentiometer, a gyroscope, other position sensitive sensors (angular or linear), or a combination thereof may be incorporated into the joints of the cantilever assembly 104 including the turntable 106, the lower cantilever cylinder 114, and the upper cantilever cylinder 116. In some embodiments, one or more extension sensors may be included for measuring the extension of the cantilever cylinder. For example, in some embodiments, one or more string potentiometers may be included inside one or more cylinders of the cantilever assembly 104 for measuring the extension of the corresponding cylinder. In some embodiments, other types of extension measurement devices may be used, such as magnetostrictive sensors or Hall effect sensors. In some embodiments, magnetostrictive sensors may be used for high performance applications. Similarly, in some embodiments, one or more rotary encoders may be disposed on the cantilever assembly 104 for measuring the rotation angles of the various joints of the cantilever assembly 104.

[0099] In some embodiments, a cantilever geometry calculation stage 510 may be included for calculating the cantilever geometry and Jacobian matrix based on the position input received from the feedback sensor input stage 508 and the known size and geometry of the cantilever assembly 104. In some embodiments, a plurality of predetermined parameters may be received before operation, indicating the size of the cantilever portion. In some embodiments, the Jacobian matrix may be generated by taking partial derivatives of the position input. The Jacobian matrix may then be provided to the joint velocity calculation stage 506 to calculate the respective joint velocities. In some embodiments, an actuator velocity conversion stage 512 may be included for converting the joint velocities of the joint velocity calculation stage 506 to actuator velocities based on the geometry of the cantilever assembly 104 and the specific links associated with each joint. Therefore, in some embodiments, the cantilever geometry calculated at the cantilever geometry calculation stage 510 may be provided to the actuator velocity conversion stage 512. Additionally, embodiments are envisioned in which the matrix calculations may be performed in advance and the results including a plurality of reference values ​​may be stored in a lookup table or other storage structure. In some such embodiments, real-time parameters may be used to approximate between the reference values ​​in the lookup table. Therefore, the processing burden can be reduced during real-time operation, further reducing control delays.

[0100] In some embodiments, a saturation check stage 514 may be included to limit the X, Y, and Z direction speed input values ​​based on the physical flow limit of the boom assembly 104. For example, the boom assembly 104 may be at least partially hydraulically actuated so that each hydraulic actuator is associated with a physical flow limit. Further, the sum of all actuators may saturate the flow, exceeding the flow that the hydraulic pump of the boom assembly can produce. Therefore, the saturation check stage 514 can determine whether the currently requested speed has exceeded any actuator limit, and if at least one of the physical flow limits is exceeded, a percentage reduction can be applied at the saturation reduction stage 504. In some such embodiments, the percentage value of the reduction can be calculated so that the X, Y, and Z speed inputs are reduced by the same amount until the flow limit is within the limit of the actuator. Alternatively, in some embodiments, a predetermined reduction value can be used. For example, each speed can be reduced by 1%, 50%, or another suitable percentage. In both cases, the amount of reduction in the X, Y, and Z speeds is the same for each case to maintain path accuracy. Conversely, if the flow limit is not exceeded, then no percentage reduction is applied at desaturation stage 504, or a percentage reduction of 0% may be applied.

[0101] In some embodiments, a valve command controller stage 516 may be included. Here, a valve command is generated for each actuator speed of the actuator speed conversion stage 512. In some embodiments, one or more PID controllers for multiple hydraulic valves of the boom assembly 104 may be included. Therefore, the PID controller may be configured to receive a signal indicating the requested actuator speed and generate valve commands respectively. In some embodiments, an output valve command stage 518 may be included, wherein the valve command is transmitted to the hydraulic valve driver for adjusting the hydraulic valve to achieve the requested speed. In some embodiments, the hydraulic valve may be electrically actuated so that the valve command is received as an electrical signal and each valve is operated based on the corresponding electrical signal. In some embodiments, the hydraulic valve may be included in the boom assembly 104 corresponding to each joint of the boom assembly 104. In some embodiments, the valve may be arranged at each joint. Alternatively, in some embodiments, the valve may be included in a common position, such as on a hydraulic manifold arranged at the turntable 106. For example, the rotary hydraulic valve may be disposed in or adjacent to the swivel table 106, the lower boom hydraulic valve may be disposed in or adjacent to the lower boom cylinder 114, and the upper boom hydraulic valve may be disposed in or adjacent to the upper boom cylinder 116. In some embodiments, the motion path may be achieved by adjusting the extension of each hydraulic cylinder of the boom assembly 104.

[0102] In some embodiments, the extension of one or more telescopic parts of the boom assembly can be controlled in a manner similar to that described above. For example, the extension length of the telescopic part can be retracted or extended based on a movement request received from an operator. Typically, movements such as boom extension are controlled separately from other movements of the boom assembly. However, embodiments are envisioned in which the extension of the telescopic part can be adjusted in unison with other movements of the boom assembly. For example, an operator can request that the boom tip move upward, and the telescopic part can be extended upward while simultaneously adjusting the hydraulic cylinder in response to the requested movement. In some embodiments, a machine architecture is provided that includes four actuators, for example, a boom assembly 104 including a turntable 106, a lower boom cylinder 114, and an upper boom cylinder 116, and a telescopic part of the boom as a fourth actuator. In some embodiments, in order to consider the fourth actuator in a three-degree-of-freedom motion system, additional constraints can be added using any one of a maximized stability parameter, a static attitude parameter, or a maximized capacity parameter to determine the solution for the fourth actuator. In some embodiments, a similar approach can be applied to systems with any number of actuators, such as systems with seven actuators and six degrees of freedom.

[0103] Further, embodiments are contemplated in which additional parameters may be considered when calculating the motion path. For example, in some embodiments, the current payout length of the winch line disposed on the boom assembly 104 may be considered. Here, the payout length may be monitored so that the winch line does not extend beyond the maximum available length to avoid overwinding and potential damage. Overwinding may refer to a situation where a lower load comes into contact with a higher load resulting in a significant amount of force being applied to the boom assembly 104. Thus, embodiments are contemplated in which the motion path may be adjusted based on the payout length of the winch line to prevent an overwinding situation of the boom assembly 104.

[0104] In some embodiments, the above reference Figure 4 The described processing flow can be used to simplify the controls that an operator must provide for moving the boom. Here, rather than individually controlling each joint of the boom assembly 104, the operator can simply provide an indication of the speed at which the boom tip should move. Thus, a simpler and more intuitive means of collecting operator input is provided, such that the operator does not need to be able to provide complex individual boom movements, and these movements will be automatically calculated within the motion processing system. Additionally, it should be understood that in some embodiments, the above-described process for driving manual boom operation can also be applied to controlling the operation of the robotic assembly 420. For example, a similar processing flow can be performed to control the rotation of joints within the robotic arms 424 and 428.

[0105] It should be understood that there may be multiple different motion paths for a given target location. Therefore, in some embodiments, various optimizations may be applied to the generated motion path. Here, for example, various adjustments to the motion path may be implemented based on any one of obstacles, singularities, types of work operations, and other factors. In some embodiments, the motion path may be optimized to provide maximized structural strength for the boom assembly 104. Additionally, in some embodiments, the motion path may be optimized to save energy. For example, the shortest collective motion path from all joints to the destination location may be selected. Further, the motion path may be selected based on avoiding collisions and singularities of the boom assembly 104 or the robot assembly 420.

[0106] In some embodiments, a closed-form inverse kinematics function may be used to calculate the motion path. Here, closed-form inverse kinematics provides a significantly faster method to generate the motion path than open-form calculations, which typically rely on a guess and check approach to converge to a solution, which can consume a significant amount of time and produce significant input delays. Closed-form inverse kinematics may use predetermined mathematical parameters for the device that take into account the geometry and dimensions of the device, such as the length of one or more components of the robotic arm and portions of the boom assembly 104. Thus, the lengths of the components of the device may be automatically calculated and pre-computed, thereby simplifying real-time calculations performed during operation. In some embodiments, the predetermined parameters of the teleoperated device allow for the use of a closed-form technique that receives one or more inputs and generates one or more outputs without the need for guess and check or repeated iterations to provide a solution.

[0107] In some embodiments, a motion path may be determined for a suspended load, for example, disposed at the tip of the boom. In some such embodiments, the suspended load may be held on a multi-function platform at the tip of the boom, or at a location that replaces the multi-function platform. For example, in some embodiments, a utility pole or other object may be grasped and supported at the distal end of the boom. Thus, a control input requesting the execution of a motion of the suspended load may be received. Thus, an operator may intuitively control the positioning and orientation of the suspended load.

[0108] In some embodiments, various forms of load monitoring may be applied to monitor one or more loads of the boom assembly 104 or the robot assembly 420. Thus, in some embodiments, tactile feedback and / or other forms of feedback may be generated in response to the detected load. The feedback response may include any combination of tactile feedback such as vibration of a controller or other user device, visual feedback such as a flashing light, or auditory feedback such as a ringing alarm. In some embodiments, the intensity or frequency of the feedback may be determined in proportion to the load. For example, a less intense feedback response may be generated for a load of 200 pounds compared to a load of 500 pounds.

[0109] Further, in some embodiments, the intensity (or frequency) of the feedback response can be carefully selected based on a comparison of the measured load to a predetermined maximum load value, such that the intensity increases as the measured load approaches the maximum load value. In some embodiments, the intensity and / or frequency can vary according to an exponential function so that load changes at higher loads approaching the maximum load value are more obvious to the operator. In some embodiments, linear functions and other types of feedback functions are also contemplated. In some embodiments, the intensity and / or frequency of the feedback response can become saturated after exceeding a certain load value. For example, in some embodiments, when the maximum load value is exceeded, the feedback response can become saturated so that the feedback response is similar at the maximum load and above the maximum load.

[0110] In some embodiments, feedback responses can be transmitted so that the operator is notified without leaving the work area or losing focus. For example, any one of vibration, light or sound alarms can be activated in controllers 412 and 414, display 406 and speaker 408, respectively.

[0111] Additionally, embodiments are contemplated in which the speed and motion path of the boom cylinder can be automatically adjusted based on one or more measured loads. For example, if a heavy load of 400 pounds is suspended at the boom tip, the speed of the valve command can be automatically increased so that the boom assembly 104 moves at a similar speed as if no load was suspended at the boom tip. Alternatively, in some embodiments, the valve command may not be increased based on the measured load, so that the boom assembly moves slower while supporting a heavy load to ensure extra caution. Further, in some embodiments, a pressure-compensated hydraulic valve can be used so that the speed of movement is independent of the varying load amount. Still further, embodiments are contemplated in which the allowable acceleration of the boom assembly is limited based on the sensed load, thereby minimizing dynamic load conditions and shock loads on the structure of the boom assembly 104. For example, if a heavy load of 400 pounds is supported at the boom tip, the acceleration limit can be reduced based on the additional weight of 400 pounds to prevent dynamic load effects and vibrations associated with rapidly accelerating a large load.

[0112] Figure 6 An exemplary embodiment of a robotic unit 302 operating in an operating environment 600 is depicted. As described above, a camera system (e.g., camera 434) can capture images of the operating environment 600. In some embodiments, in a semi-autonomous mode, the image can be displayed to a user and the user can select an object and an action to be performed on the object. As shown, the tether 602 is being removed from the conductor 604. In this exemplary scenario, the removal of the tether 602 can be caused by stored computer executable instructions, and the user can simply select the action to be performed. The user can select and label the object. For example, the user can select the tether 602 and label the object as a "tether" by typing in a label or selecting from a predefined list of parts. When the user has defined the label and action, the robotic unit 302 can access the executable instructions associated with the action and perform the task. The robotic unit 302 can perform the task based on the above Figure 3 and Figure 5 In some embodiments, the task may be performed using a feedback controller that detects the position of a tool (e.g., a ring tool 606) through video imaging from a camera 434 and at least one sensor 290 (e.g., a rotary encoder) located on the robot to detect the state of the robot or further detect the operating environment 600.

[0113] In some embodiments, the tool position can be tracked relative to the inertial coordinate system 702. The tool position can be tracked by sensors (e.g., at least one sensor 290) in the tool (e.g., accelerometers, gyroscopes, GPS, etc.). Similarly, the multi-function arms 330a, 330b can be tracked using rotary encoders, etc. The sensor data can be combined to provide an accurate position of the tool in the inertial coordinate system 702. In addition, the environment model can be based on the inertial coordinate system 702. Therefore, the positions of all tools and objects in the operating environment 600 can be mapped to the inertial coordinate system 702, so that the operating model can include all necessary information for the robot unit 302 to perform the task. Once the image is taken and the operating model is created, the robot unit 302 can assume that no objects in the operating environment are moving. Therefore, the robot unit can manipulate the line 602 according to the automation instruction set.

[0114] In some embodiments, an operational model may be generated that includes coordinates assigned to objects detected in an image. For example, in some embodiments, a machine learning algorithm, such as a convolutional neural network (CNN) trained for object detection and classification, may be used to detect and classify objects in the operating environment 600. In addition, the algorithm for object detection may be trained on a relatively small set of objects. This may result in relatively fast object recognition and classification to produce an effective response from the robotic unit and complete the task in a short period of time.

[0115] In some embodiments, object recognition and operation models coupled to the inertial coordinate system 702 can be used to create a fully autonomous robotic unit 302. For example, as described above, the camera 434, the depth sensor 438, and the camera pole camera 464 can be used to acquire images and / or video of the operating environment 600. Object recognition and classification algorithms can be used to identify and classify objects (e.g., fasteners, crossbars, insulators, transformers, and conductors) in the operating environment 600. Object positions can be mapped to the inertial coordinate system 702 that generates the operation model. The operation model can then be used with the task to be performed by the robotic unit 302 to move to an initial (home) position and begin an automated task that can be controlled by a feedback loop including any sensor that provides feedback (i.e., sensor 290). In some embodiments, feedback can be provided by a camera system, a rotary encoder, an accelerometer, and any other sensor that can be used to cause awareness of the state of the operating environment 600 or the robotic unit 302 when performing a task.

[0116] In some embodiments, the computer executable instructions are based on the type of tool used and the position of the reference point associated with the tool. In addition, specific instructions can be provided based on the geometry of the tool. For example, wrenches or ring tools of various sizes can be used. Therefore, the instructions for using the tool may be slightly different to accommodate different tools. In addition, the reference point for marking the tool coordinates can be defined in different places. The reference point can be defined at the point where the tool is connected to the multifunctional arm 330a, 330b, or it can be defined at the point where the tool is aligned with the object. Therefore, the computer executable instructions can be slightly different based on the reference point position. In some embodiments, a simple calculation can transform the reference position into a predefined position so that there is no need to store various instructions.

[0117] In some embodiments, the robot unit 302 can perform various tasks in semi-autonomous and fully autonomous modes. For example, the robot unit 302 can store computer executable instructions for automatically removing the tie line 602. The robot unit 302 can store the positions of various objects in the operating model and start from the "starting" position relative to the tie line 602. The robot unit 302 can then continue to perform tasks based on the object position, feedback information from the sensor 290, and the stored computer executable instructions. In some embodiments, the stored computer executable instructions can be accessed based on the storage mode of the robot unit 302 that can be selected by the user. The robot unit 302 can be able to remove and add fasteners (i.e., bolts, screws, tie lines, etc.) autonomously and semi-autonomously. The robot unit 302 can also be able to autonomously and semi-autonomously add, move, repair and clean capacitors, conductors, transformers, etc. Any task that can be performed by a worker can be performed by the robot unit 302.

[0118] Figure 7 An exemplary autonomous or semi-autonomous operating mode of the aerial device 100 is depicted. As shown, the aerial device 100, in operation, extends the remote component system 300 in a straight line along the x-axis in an inertial coordinate system 700 including an inertial grid 706. A boom coordinate system 704 (e.g., a local coordinate system) can describe the elements and motion of the boom assembly 104. In some embodiments, the remote component 202 can know or store the inertial coordinate system 702, and in some embodiments, the remote component 202 can be unaware of the inertial coordinate system 702.

[0119] In some embodiments, inertial coordinate system 702 may be stored for operation by remote component 202. Inertial coordinate system 702 may be based on or associated with global coordinates, such as GPS coordinates, or based on or associated with local coordinates described by local objects or short-range communications, such as Bluetooth, WIFI, and / or radio frequency identification (RFID). In some embodiments, points (waypoints) in inertial coordinate system 702 may be provided.

[0120] Waypoint A can be the first location of the remote component system 300, and waypoint B can be the second location of the remote component system 300. In some embodiments, the user can enter waypoints as the starting point and the end point. For example, the user can enter the starting coordinates (50, 150) in a two-dimensional plane. These coordinates can also be three-dimensional and can have a time associated with the coordinates to generate a controlled time movement. The user can move the remote component system 300 to the vicinity of the actual location of waypoint A. The user can then change the mode of the aerial device 100 to an autonomous mode or a semi-autonomous mode, and the remote component 202 can issue instructions through stored instructions and move according to the instructions based on the state of the boom assembly 104 and the location of waypoint B. The remote component system 300 can move to waypoint B. In some embodiments, waypoint A can be the "original" waypoint, and the remote component system 300 will move to this waypoint if any failure or communication loss is detected.

[0121] In some embodiments, remote component system 300 may be commanded to move from waypoint A to waypoint B by a user or a timer. Figure 5 The motion of the remote component system 300 from waypoint A to waypoint B is described as being achieved by cantilever dynamics; however, the input may be the coordinates of waypoint B, the error between the robot state and the reference state, or any other control signal instead of user input. A feedback controller may be used to detect the inertial space coordinates and, based on the error between the actual position and the commanded position, adjust the position according to the reference state. Figure 5 As described above, inverse kinematics may be used to move remote component system 300 from waypoint A to waypoint B using a feedback reference signal based on the position of waypoint B and a position sensor of remote component system 300. The position sensor may be one or more sensors 220, which may be a short-range transceiver or a global transceiver, such as a GPS.

[0122] In some embodiments, there may be one, two, three, four or any number of path points and algorithms to control the motion of the manipulator and cantilever between nonlinear path points. The path points may be positioned in any arrangement so that any manipulator and cantilever assembly can be moved autonomously as needed. For example, path points C, D, E and F are arranged in an exemplary rectangular pattern. Any pattern may be defined by path points, including squares, triangles, circles, any regular or irregular shapes or any abstract shapes. Sharp turns may not be ideal because the immediate change of direction can cause unnecessary dynamics, resulting in stress and oscillation on the structure. Therefore, slow changes can be implemented. As shown, the transition between line segments CD and DE can be connected by splines. Splines can be defined based on the angle between the line segments, the manipulator used, and the rate at which the manipulator moves. For example, the manipulator can be a high dexterity manipulator, such as a robot arm 424, 428. Compared with a high-capacity manipulator (e.g., a heavy multifunctional arm 390) moving at the same rate, the manipulator can move between line segments along a relatively short spline. However, the heavy multi-function arm 390 may move more slowly, which may allow it to employ shorter splines than the robotic arms 424, 428. The splines may be controlled by computer executable instructions provided by the hardware components described herein.

[0123] In some embodiments, remote component system 300 can detect objects and calculate coordinates to generate waypoint B. Remote component 202 can use sensor 290 as described above to detect the operating environment. A three-dimensional representation can be generated within the operating environment. The location of waypoint B can be determined based on the operating environment 600 and the stored three-dimensional representation. When waypoint B is determined, remote component 202 can command remote component system 300 to move to waypoint B and command the movement of boom assembly 104 and any operation of robot unit 302, as described above. In addition, in some embodiments, the waypoint can be dynamically moved in near real time through operator input at manual controller 412, 414. Figure 8An embodiment of detecting an object and autonomously or semi-autonomously performing a cutting or moving operation is depicted. For example, any of the above-mentioned cameras 434, depth sensors 438, and camera pole cameras 464 can be used alone or in combination to obtain an image of an object, in this example, a tree trunk. Here, the depth sensor 438 can be used in conjunction with the camera 434 to determine the edge of the tree. The background 802 is shown in a darker contrast. The edge of the tree (shown as a black dot) can be determined by any edge detection, object recognition, or object detection algorithm. Points on each relative edge can be determined and a cross line 810 can be drawn between these points. A vertical center line passing through the cross line 810 can be determined. Finally, a horizontal center line can be connected between the two edges. The camera reticle 806 can be aligned with the center of the cross line 810. The intersection between the vertical center line and the horizontal center line can mark the center 812 to be aimed at. Because this is a tree and can be identified as a tree, or the user can enter that the object is a tree, it is known that it is generally round. Therefore, the horizontal center line can be used as an estimate of the diameter of the tree.

[0124] As shown in the final figure, it is assumed that the first object representation 814 is vertical, but the measured angles are not aligned. The detected horizontal centerline 820 and vertical centerline 818 are tilted relative to the first object representation 814. As shown in the image, roll, pitch and yaw can be determined to best align with the object. Therefore, the virtual representation can be a rotation representation 816 to align with the determined measurement line. In addition, the diameter as described above and the distance to the object can be measured. The distance to the object can be measured by a depth sensor 438, which can be a point cloud camera, a laser rangefinder, etc. As described below, a gripping tool, a cutting tool, etc. can be configured to align with an object to be clamped, cut, or any other object required to be aligned by a connection method. In some embodiments, the detection and alignment method can be performed to connect the tool to a fastener, a pole, a transformer, a groove, a notch, a ratchet, a socket, etc.

[0125] Fig. 9A and 9B An exemplary gripper 900 is depicted. Fig. 9A An exemplary tool for cutting that may be used in the above examples is depicted. The gripper 902 may include tines 904 and a cutting device 906 (e.g., a saw) that may be operated hydraulically, pneumatically, electrically, etc. In some embodiments, the gripper 902 may include at least one joint 908 that provides tumbling motion, or similarly, may include actuation about x, y, and z axes in the inertial coordinate system 702 to provide pitch, roll, and yaw. Thus, the gripper 902 may be operable to move the gripper 902 based on a reference. Figure 8The object detection described above is rotated to a position for optimal cutting. In addition, since the roll, pitch, yaw, diameter and distance are known, the gripper 902 can be oriented to the object and moved to the object, placing the object in the fork 904, close to the fork 904 to the measured diameter or before cutting until the desired pressure is reached. When the desired pressure is reached on the fork 904, the cutting device 906 can be operated to cut the object. The above-mentioned cutting tasks can be performed completely autonomously or semi-autonomously. In some embodiments, the user can command the gripper 902 to perform the task.

[0126] also, Fig. 9B An embodiment of a load guide 910 is depicted for grabbing a pole and guiding the pole into position to be inserted into the ground. In some embodiments, the load guide 910 may include a Mecanum wheel 912 that provides translation and simultaneous rotation of the pole. In some embodiments, the pole may be a utility pole and the load guide 910 may be positioned at the end of the boom assembly 104. Using the object detection ( Figure 8 ), the load guide 910 can be used in a fully autonomous or semi-autonomous mode. For example, the user can position the load guide 910 so that the pole is located above the pre-dug hole. The user can then begin to automatically install the pole into the ground, where the boom assembly 104 moves the pole down into the hole while the load guide 910 translates and rotates the pole. In some embodiments, the boom assembly 104 can remain stationary and the load guide 910 can translate and rotate the pole.

[0127] Fig.10 An exemplary clamp 1000 for a utility pole is depicted. In some embodiments, the clamp 1000 includes two or more clamp tines 1002 configured to clamp onto a utility pole. The clamp tines 1002 can be connected to a first rod 1004, which can be coupled to a second rod 1006, which can be coupled to an actuator 1008 for rotating a screw 1010. When the screw 1010 rotates, or when the screw 1010 is stationary and the threaded bushing 1012 rotates about the screw 1010, the distal end 1014 of the clamp 1000 moves away from the proximal end 1016 of the clamp 1000, causing the clamp tines 1002 to come together. Similarly, when the actuator 1008 is rotated in the opposite direction, the distal end 1014 of the clamp 1000 moves toward the proximal end 1016, thereby separating the joint between the first rod 1004 and the second rod 1006, thereby causing the clamp tines 1002 to move away from each other, thereby releasing the utility pole. The clamp 1000 may be actuated electrically, mechanically, hydraulically, or pneumatically.

[0128] The fixture 1000 can be configured to couple to and support any bracket, three-phase lift, manipulator arm, etc. The fixture 1000 can be self-locking so that the fixture 1000 can be moved into place and locked to the pole. The fixture 1000 can be powered by a local power source, such as a battery or motor positioned at the tip of the cantilever. In some embodiments, the control system can be set locally and communicate with the robot unit 302, the heavy multi-function arm 390, or any other control system via wireless, wired, optical fiber, short-range communication, or any other communication method. In some embodiments, the fixture 1000 can be included in the remote component system 300.

[0129] Reference now Fig.11 , describes a method of operating an aerial device 100 including a remote component system 300, wherein the remote component system includes a robotic unit 302. The method of operating the aerial device 100 or its components may include manual, autonomous, or semi-autonomous control methods. The method may include any one or more operations currently disclosed herein. The one or more operations may be performed at least in part by a computing device including at least one processor and at least one memory device, which may be one or more non-transitory computer-readable media. The method and / or one or more operations may be embodied by a computer-readable medium. The computer-readable medium may include computer-executable instructions that, when executed by at least one processor, cause the at least one processor to perform one or more operations. The method may include one or more operations.

[0130] At step 1102, remote component system 300 may acquire sensory information via at least one sensor 290, camera 434, depth sensor 438, camera pole camera 464, and any other sensor devices described herein. The sensory information may indicate a state of aerial device 100, a state of remote component system 300, a state of boom assembly 104, and / or any associated components. In addition, the sensory information may indicate operating environment 600 and objects and locations of objects therein. At least one sensor 290 may be any accelerometer, camera, magnetometer, Hall effect sensor, rotary encoder, gyroscope, attitude sensor, and the like.

[0131] At step 1104, a three-dimensional representation may be created as described in the above embodiments. The three-dimensional representation may be a representation of the operating environment 600. The three-dimensional representation may include an inertial coordinate system 702. In some embodiments,

[0132] At step 1106, the acquired images may be displayed to the operator of the aerial device 100, as described in the above embodiments. The images may be photographs or videos and may be displayed to the operator at any location, including on-site or remote locations. The operator may decide further steps based on the images, including acquiring more images or continuing further operations.

[0133] In step 1108, instructions for the robotic unit 302 and / or boom assembly 104 may be received. These instructions may come from an operator or may come from at least one processor executing computer executable instructions. The operator may provide complete controls or may simply provide input (e.g., tags, mode changes, mode starts, etc.). The operator may simply approve the suggested next action proposed by the remote component system 300. In some embodiments, in step 1110, commands may be generated by receiving a reference signal indicating a desired state and comparing the desired state to a three-dimensional representation and a current state. The reference signal is compared to the current state of the aerial device 100 and its components. The desired state may be generated from the computer executable instructions based on the operating mode, the tool in use, and any other sensory information described above.

[0134] At step 1112, after comparing the reference signal and the current state, a controller including any P, PI, PID, adaptive controller or any artificial intelligence algorithm may generate signals to control the various actuators and components of the aerial device 100 to complete the desired task. In some embodiments, the desired task may be any of actuating fasteners, tightening couplings, lifting and moving objects, and any other tasks that may be typically performed by a worker in an aerial platform or in a cutting device for tree felling, trimming or pruning, etc. In some embodiments, the operational task may be the movement of the boom assembly 104 to a specified location, the operation of an excavation crane, or the placement of a utility pole.

[0135] exist Fig.12, an exemplary hardware platform of a computer system 1200 of some embodiments of the present invention is depicted. Computer 1202 may be a desktop computer, a laptop computer, a server computer, a mobile device such as a smart phone or a tablet computer, or any other form of general or special computing device. For the purpose of illustration, computer 1202 depicts several components. In some embodiments, certain components may be arranged differently or not exist. Additional components may also exist. System bus 1204 is included in computer 1202, whereby other components of computer 1202 may communicate with each other. In some embodiments, there may be multiple buses or components that can communicate directly with each other. Central processing unit (CPU) 1206 is connected to system bus 1204. One or more random access memory (RAM) modules 1208 are also attached to system bus 1204. Graphics card 1210 is also attached to system bus 1204. In some embodiments, graphics card 1210 may not be a physically separate card, but may be integrated into motherboard or CPU 1206. In some embodiments, the graphics card 1210 has an independent graphics processing unit (GPU) 1212, which can be used for graphics processing or for general-purpose computing (GPGPU). GPU memory 1214 is also on the graphics card 1210. A display 1216 for user interaction is connected (directly or indirectly) to the graphics card 1210. In some embodiments, there is no display, while in other embodiments it is integrated into the computer 1202. Similarly, peripherals such as a keyboard 1218 and a mouse 1220 are connected to the system bus 1204. Like the display 1216, these peripherals can be integrated into the computer 1202 or not. A local storage device 1222 is also connected to the system bus 1204, which can be any form of computer-readable medium and can be installed internally in the computer 1202 or removably connected externally.

[0136] Computer-readable media include both volatile and non-volatile media, removable and non-removable media, and encompass database-readable media. For example, computer-readable media include (but are not limited to) RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile discs (DVD), holographic media or other optical disk storage, cassettes, magnetic tapes, magnetic disk storage and other magnetic storage devices. These technologies can store data temporarily or permanently. However, unless otherwise expressly stated, the term "computer-readable medium" should not be interpreted as including physical but transient forms of signal transmission, such as radio broadcasts, electrical signals through wires, or light pulses through fiber optic cables. Examples of stored information include computer-usable instructions, data structures, program modules, and other data representations.

[0137] Finally, a network interface card (NIC) 1224 is also connected to the system bus 1204 and allows the computer 1202 to communicate over a network such as a local network 1226. The NIC 1224 can be any form of network interface known in the art, such as Ethernet, ATM, fiber optic, Bluetooth, or Wi-Fi (i.e., the IEEE 802.11 family of standards). The NIC 1224 connects the computer 1202 to the local network 1226, which can also include one or more other computers such as computer 1228 and network storage such as data store 1230. In general, a data store such as data store 1230 can be any repository from which information can be stored and retrieved as needed. Examples of data stores include relational or object-oriented databases, spreadsheets, file systems, flat files, directory services (such as LDAP and Active Directory), or email storage systems. The data store can be accessed via a complex API (e.g., structured query language), a simple API that only provides read, write, and find operations, or any level of complexity in between. Some data stores may additionally provide management functions for the data sets stored therein, such as backup or version control. The data store may be local to a single computer, such as computer 1228, accessible on a local network, such as local network 1226, or remotely accessible via Internet 1232. Local network 1226, in turn, is connected to Internet 1232, which connects many networks, such as local network 1226, remote networks 1234, or directly connected computers, such as computer 1236. In some embodiments, computer 1202 itself may be directly connected to Internet 1232. It should be understood that in some embodiments, computer 1202 may be a reference Figure 2 Controller 224 is described and may perform similar operations.

[0138] The following U.S. patent applications were all filed on July 28, 2022, and are all incorporated herein by reference in their entirety as if set forth verbatim: U.S. application serial number 63 / 392,927, entitled “REDUCING LATENCY IN HEAD-MOUNTED DISPLAY FOR THE REMOTE OPERATION OF MACHINERY”; U.S. application serial number 17 / 875,674, entitled “MANUAL OPERATION OF A REMOTE ROBOT ASSEMBLY”; U.S. application serial number 17 / 875,743, entitled “COOPERATIVE HIGH-CAPACITY AND HIGH-DEXTERITY MANIPULATORS”; U.S. application serial number 17 / 875,796, entitled “ROTARY TOOL FOR REMOTE POWER LINE OPERATIONS”; U.S. application serial number 17 / 875,821, entitled “OPERATION AND INSULATION TECHNIQUES”; U.S. application serial number 17 / 875,893, entitled “COORDINATE MAPPING FOR MOTION CONTROL”; U.S. application serial number 17 / 875,943, entitled “WIRE TENSIONING SYSTEM”; U.S. application serial number 17 / 875,990, entitled “CROSS-ARM PHASE-LIFTER”; and U.S. application serial number 63 / 393,047, entitled “ELECTRICALLY INSULATING BLANKET WITH MEMORY SET”. The subject matter described in the aforementioned U.S. patent applications may be combined with the subject matter of the present disclosure. For example, one or more embodiments, features, structures, actions, etc. described in any one or more of the aforementioned U.S. patent applications may be combined with one or more embodiments, features, structures, actions, etc. described in the present disclosure.

[0139] Although the presently disclosed subject matter has been described with reference to the embodiments shown in the accompanying drawings, it should be noted that equivalents may be employed and substitutions may be made without departing from the scope of the present invention as set forth in the claims.

[0140] Therefore, the matters claimed as novel and expected to be protected by Letters Patent include the following.

Claims

1. A robotic system for performing automated tasks in an operating environment of an aerial device, the robotic system comprising: at least one processor; a robotic unit disposed at the tip of the boom, comprising at least one manipulator configured to perform a task in the operating environment; a tool configured to couple to the at least one manipulator and perform the task; a camera for acquiring images of the operating environment; a sensor associated with the robotic unit; as well as One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by the at least one processor, perform a method of completing the task, the method comprising: acquiring, by the camera, information indicating the location of an object in the operating environment; acquiring sensor data from the sensor, wherein the sensor data indicates a status of the at least one manipulator; classifying the objects by a machine learning algorithm; identifying the tool based on the classification and the task; autonomously connecting the tool to the at least one manipulator; automatically transforming a tool reference point based on the geometry of the tool; and The task is performed based on the position of the object and the sensor data.

2. The robotic system of claim 1, wherein the method further comprises receiving a first input from a user indicating the task to be performed and receiving a second input from the user indicating initiation of performance of the task.

3. The robotic system of claim 1, wherein the method further comprises: generating an inertial coordinate system including coordinates of the object; determining tool coordinates of the tool coupled to the at least one manipulator in the inertial coordinate system; as well as The task is performed autonomously based on the position of the object and the tool coordinates in the inertial coordinate system.

4. The robotic system of claim 3, wherein the method further comprises: The task to be performed is selected based on the classification and the tool.

5. The robot system according to claim 1, further comprising: a cantilever assembly comprising a cantilever tip supporting the at least one manipulator; Wherein the method further comprises: Detecting the cantilever tip position; generating an inertial coordinate system including the coordinates of the cantilever tip; determining position coordinates indicative of a position for operation of the at least one manipulator; and The cantilever tip is autonomously moved to position the at least one manipulator at the location for the operation.

6. The robotic system of claim 5, wherein the method further comprises receiving the location of the operation as a waypoint uploaded by a user.

7. The robotic system of claim 6, wherein the method further comprises: receiving the cantilever tip position via a position sensor; moving the at least one manipulator to the waypoint based at least in part on the position from the position sensor, and The at least one manipulator is moved to the position for the operation using the camera.

8. The robotic system of claim 1, wherein the robotic unit is electrically insulated from a base of the aerial device.

9. A method for performing an automated task by a robotic unit in an operating environment of an aerial device, the method comprising: acquiring, by a camera, object information indicating an object and position information indicating a position of the object in the operating environment; classifying the object based on the information indicative of the object; identification means based on said classification of said object; Autonomous access to said tools; automatically transforming a reference position for a cantilever tip coordinate system based on the geometry of the tool; acquiring sensor data via sensors associated with the robotic unit, wherein said sensor data is indicative of a status of said at least one manipulator of said robotic cell; as well as manipulating the object by means of the at least one manipulator to perform a task, Wherein the manipulation is based at least in part on the position of the object and the sensor data from the sensor.

10. The method of claim 9, wherein the task is manipulating one of a tie wire, a bolt, a nut, a capacitor, or a transformer.

11. The method according to claim 9, further comprising: Generate an inertial coordinate system, Wherein the state of the at least one manipulator comprises at least a state position in the inertial coordinate system.

12. The method of claim 11, further comprising applying inverse kinematics to move the robotic unit to autonomously perform the task.

13. The method according to claim 12, further comprising: receiving a waypoint indicating coordinates in the inertial coordinate system; as well as Further included controlling an actuator to move the robotic unit to the waypoint to perform the task.

14. The method according to claim 9, wherein the camera is a first camera and is coupled to the robotic unit; and further comprising acquiring an image from a second camera connected to an end of a pole remote from the robotic unit, The image is outside the field of view of the first camera.

15. The method according to claim 14, further comprising: generating a three-dimensional representation of the object; as well as Perform the described tasks autonomously.

16. A robotic system for performing automated tasks in an operating environment of an aerial device, the robotic system comprising: at least one processor; a robotic unit comprising at least one manipulator configured to perform a task in said operating environment; a tool configured to couple to the at least one manipulator and perform the task; a camera for acquiring images of the operating environment; a sensor associated with the robotic unit; as well as One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by the at least one processor, perform a method of completing the task, the method comprising: acquiring, by the camera, information indicating the location of an object in the operating environment; acquiring sensor data from the sensor, wherein the sensor data indicates a status of the at least one manipulator; classifying the objects by a machine learning algorithm; identifying the tool based on the classification and the task; autonomously connecting the tool to the at least one manipulator; automatically transforming a tool reference point based on the geometry of the tool; and manipulating the object by means of the at least one manipulator and the tool to accomplish the task, Wherein the manipulation is based at least in part on the position of the object and the sensor data from the sensor.

17. The robot system according to claim 16, wherein the camera is a first camera and detects visible light; and Further included is a second camera mounted to the pole for imaging a side of the object that is not in the field of view of the first camera.

18. The robotic system of claim 16, further comprising: a depth camera for detecting at least the depth of the object; as well as a display configured to display information from the depth camera and the camera to a user, The depth camera at least detects the depth of the object from the camera and displays at least the depth to the user.

19. The robotic system of claim 18, further comprising: a gripping device coupled to the at least one manipulator, Wherein the gripping device is automatically controlled to grip the object based on the depth and orientation of the object.

20. The robotic system of claim 19, further comprising a cutting device configured to cut the object.

Images