Systems and methods for software-defined drones

Abstract

A vehicle controller for operating an autonomous vehicle includes an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle and a flight controller configured to execute a high-speed control loop to periodically update commands to a plurality of control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle. The vehicle controller also includes a mission computer including a processor programmed to control the mission of the autonomous vehicle. The vehicle controller also includes a cellular data radio configured to wirelessly communicate over a cellular network. The vehicle controller also includes a housing that houses the IMU, the flight controller, the mission computer, and the cellular data radio. A method for operating an autonomous vehicle is also provided.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT International Patent Application claims the benefit of U.S. Provisional Patent Application No. 63 / 019,558, filed May 4, 2020, the contents of which are incorporated herein by reference in their entirety.

[0002] Field The present disclosure relates generally to unmanned vehicles, also known as drones, and systems for controlling and communicating with drones. More specifically, the present disclosure relates to unmanned aircraft systems (UAS). [Background technology]

[0003] background Unmanned vehicle systems, such as aerial drones, include a flight management unit (FMU) mounted on the vehicle to manage the vehicle's operation. An inertial measurement unit (IMU) can be coupled to the FMU to measure specific forces acting on the vehicle using a combination of accelerometers, gyroscopes, and / or magnetometers. The FMU can be specifically configured to execute high-speed control loops to control the vehicle using actuators such as servo motors and electric speed controllers (ESCs) coupled to electric motors based on signals from the IMU. Conventional FMUs are designed to execute high-speed control loops while minimizing power consumption and weight. Some FMUs may include interfaces for control inputs, such as radios for receiving remote control commands, and additional control outputs, such as payload control signals. The payload control signals may include, for example, position commands for servo motors configured to move a camera gimbal.

[0004] Conventional IMUs can be mounted to the vehicle using anti-vibration mounts to reduce interference from vibration noise and to reduce shock loads in the event of a rough landing or collision with the ground or other objects.

[0005] Some unmanned vehicles may include a mission computer that may include one or more specialized and / or general-purpose processors with relatively high performance for performing computationally intensive tasks such as machine vision. Summary of the Invention [Means for solving the problem]

[0006] overview One aspect of the disclosed embodiments includes a vehicle controller for operating an autonomous vehicle. The vehicle controller includes an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle and a flight controller configured to execute a high-speed control loop to periodically update commands to a plurality of control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle. The vehicle controller also includes a mission computer including a processor programmed to control a mission of the autonomous vehicle. The vehicle controller also includes a cellular data radio configured to wirelessly communicate over a cellular network. The vehicle controller also includes a housing that houses the IMU, the flight controller, the mission computer, and the cellular data radio.

[0007] One aspect of the disclosed embodiments includes a vehicle controller for operating an autonomous vehicle. The vehicle controller includes an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle and a flight controller configured to execute a high-speed control loop to periodically update commands to a plurality of control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle. The vehicle controller also includes a mission computer including a processor programmed to control a mission of the autonomous vehicle. The vehicle controller also includes a housing that houses the IMU, the flight controller, and the mission computer. The mission computer includes an application programming interface for providing vehicle status information and payload data to third-party applications.

[0008] One aspect of the disclosed embodiments includes a method for operating an autonomous vehicle, the method including: measuring specific forces acting on the autonomous vehicle with an inertial measurement unit (IMU); sending commands by a flight controller to each of a plurality of control actuators to control a position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle; instructing the flight controller by a mission computer to change the position or orientation of the autonomous vehicle; and communicating with a server by the mission computer using a cellular data radio. The IMU, flight controller, mission computer, and cellular data radio are all located within a housing onboard the autonomous vehicle.

[0009] These and other aspects of the present disclosure are set forth in the following detailed description of the embodiments, the appended claims, and the accompanying drawings.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS Further details, features and advantages of the design of the present invention can be gleaned from the following description of example embodiments with reference to the associated drawings. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an exploded view of a controller for an autonomous vehicle according to one aspect of the present disclosure. [Figure 2] FIG. 2 is a block diagram of an autonomous vehicle having the controller of FIG. 1 in accordance with the principles of the present disclosure. [Figure 3] FIG. 1 is a block diagram of a controller for an autonomous vehicle in accordance with the principles of the present disclosure. [Figure 4] FIG. 1 is a block diagram of a system for controlling an autonomous vehicle in accordance with the principles of the present disclosure. [Figure 5] FIG. 1 is a block diagram of software components in a system for controlling an autonomous vehicle in accordance with the principles of the present disclosure. [Figure 6] FIG. 1 is a flow diagram generally illustrating a method of operating an autonomous vehicle in accordance with the principles of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Detailed Description Systems and methods for software-defined drones are described herein. The system includes a vehicle controller 20 configured to operate onboard an unmanned vehicle. In some embodiments, the unmanned vehicle is an unmanned aerial vehicle, such as a multicopter, helicopter, fixed-wing aircraft, or vertical take-off and landing (VTOL) vehicle. However, the unmanned vehicle may be any type of air, land, and / or sea vehicle. The vehicle controller 20 combines an integrated flight controller, mission computer, and networking devices into a single package. The unmanned aerial vehicle may be a multicopter, such as a quadcopter or octocopter.

[0013] The following terms, unless otherwise defined in this disclosure, shall be understood to have the following definitions.

[0014] AMC (Auterion Mission Control) - A ground control software package that provides complete flight control and mission planning for drones.

[0015] · API (Application Programming Interface) - A computing interface to a software component or system that defines how other components or systems can use it.

[0016] · Autonomous Vehicles - Unmanned vehicles that are capable of operating autonomously using embedded systems in conjunction with on-board sensors and GPS.

[0017] · CAN (Controller Area Network) - A vehicle bus standard that allows microcontrollers and devices to communicate.

[0018] Cloud Suite - A distributed web-based hardware and software platform configured to store, process, and present data from one or more drones.

[0019] Drones - Unmanned vehicles that can operate autonomously using embedded systems in conjunction with on-board sensors and GPS.

[0020] · Company PX4 - a software package configured to run on the processor on board the drone to control the drone's operation and collect and process data.

[0021] ·ESC (Electronic Speed ​​Control) - An electronic circuit that controls and regulates the speed of an electric motor. · FMU (Flight Management Unit) - a controller configured to manage the operation of an aerial drone.

[0022] · FPV (First Person View) - The point of view of a real or virtual driver or pilot within a vehicle.

[0023] · GPS (Global Positioning System) - A ground-based and / or satellite-based radio positioning system, including but not limited to the United States Global Positioning System, the Russian Global Navigation Satellite System (GLONASS), the Chinese Beidou Navigation Satellite System, the European Union's Galileo positioning system, India's NavIC system, and Japan's Quasi-Zenith Satellite System (QZSS).

[0024] IMU (Inertial Measurement Unit) - An electronic device that uses a combination of accelerometers, gyroscopes, and sometimes magnetometers to measure and report specific forces, angular velocity, and sometimes body orientation.

[0025] LTE (Long Term Evolution) - Any mobile (i.e., cellular) wireless data communication standard for mobile devices, including but not limited to GSM / EDGE, 4G, and 5G wireless data communication standards.

[0026] MAVLink - an ultra-lightweight messaging protocol for communicating with drones and between on-board drone components.

[0027] MAVSDK - MAVLink Software Development Kit (SDK) library with APIs for C++, iOS, Python, and Android.

[0028] MIPI CSI (Camera Serial Interface) - A widely adopted, simple, high-speed protocol primarily intended for point-to-point image and video transmission between a camera and a host device.

[0029] · Open PPB (Pixhawk Payload Bus) - A hardware and software interconnection standard for connecting a controller with one or more payload devices, such as a camera and camera gimbal.

[0030] · PWM (Pulse Width Modulation) - An interface standard for controlling devices such as servos or ESCs (Electronic Speed ​​Control).

[0031] PX4 - An open source autopilot system for operating autonomous aircraft.

[0032] ROS2 (Robot Operating System version 2) - a set of software libraries and tools for building robotic applications.

[0033] RTC - Real Time Clock. SBUS - A serial communications protocol developed by Futaba to transmit multiple device control channels over a single interface.

[0034] · SDK (Software Development Kit) - A collection of software development tools in one package.

[0035] SPI (Serial Peripheral Interface) - A synchronous serial communications interface specification used primarily for short-distance communications in embedded systems.

[0036] · UART (Universal Asynchronous Receiver-Transmitter) - A computer hardware device for asynchronous serial communication with configurable data format and transmission speed.

[0037] · UI - User Interface. ·UX-User Experience.

[0038] ·UTM - Traffic management for unmanned aerial systems. Unmanned - configured to be controlled by a computer.

[0039] VIO (Visual-Inertial Odometry) - the process of estimating the state (attitude and velocity) of an agent (e.g., an aerial robot) using only the inputs of one or more cameras and one or more inertial measurement units (IMUs) attached to them.

[0040] FIG. 1 illustrates an exploded view of a vehicle controller 20 for an autonomous vehicle according to one embodiment of the present disclosure. The vehicle controller 20 includes housings 22, 24 having a cover 22 and a base plate 24 that fit together in a clamshell configuration. The housings 22, 24 have a generally rectangular shape and can be made of metal, plastic, or any other suitable material. For example, the housings 22, 24 may be made of aluminum or magnesium. The housings 22, 24 may be coated with paint or an anodized coating to reduce wear while dissipating heat. The remaining components of the vehicle controller 20 are held within an interior space between the cover 22 and the base plate 24 of the housings 22, 24. The cover 22 defines an integral heat sink 26 for removing heat from the internal components. The integral heat sink 26 may be formed as a plurality of planar fins extending parallel to one another and spaced apart at regular intervals. However, the integral heat sink 26 may take other forms, such as a pin, a post, one or more heat tubes, or a radiator. The vehicle controller 20 may include one or more fans for increasing airflow over the integrated heat sink 26. The base plate 24 includes a plurality of mounting tabs 28 for securing the vehicle controller 20 to the autonomous vehicle. For example, the mounting tabs 28 may be secured to one or more structural elements of the autonomous vehicle with fasteners such as screws or bolts. The base plate 24 defines a plurality of holes 30 for accessing electrical connectors and / or other features of the vehicle controller 20.

[0041] The vehicle controller 20 also includes a flight controller 32, also referred to as a flight management unit (FMU). The flight controller 32 may include one or more processors mounted on a printed circuit board (PCB). However, the flight controller 32 may take other forms, such as a single integrated circuit. The flight controller 32 may be configured to operate a flight control loop to periodically issue or modify commands to one or more vehicle control actuators for controlling the operation of the autonomous vehicle. The vehicle control actuators may include, for example, one or more servo motors and / or electronic speed controls (ESCs). In some embodiments, the flight control loop may operate at a rate of at least approximately 100 Hz. In some embodiments, the flight control loop may operate at a rate of at least approximately 1 kHz.

[0042] The vehicle controller 20 also includes an inertial measurement unit (IMU) 36 configured to measure and report specific forces acting on the vehicle controller 20, angular velocities, and / or the orientation of the vehicle controller 20 in three-dimensional space. The IMU 36 may include one or more accelerometers, gyroscopes, and / or magnetometers to determine the forces, angular velocities, and / or orientation of the vehicle controller 20. The IMU 36 may communicate directly with the flight controller 32. In some embodiments, the flight controller 32 may poll or read updated status information from the IMU 36 for each iteration of the flight control loop.

[0043] In some embodiments, the IMU 36 may include a heat spreader base of thermally conductive material configured to provide uniform heat distribution between components on the IMU 36, thus improving the accuracy of the IMU 36.

[0044] The IMU 36 is attached to the flight controller 32 by an IMU bracket 38. The IMU bracket 38 includes a plurality of dampers 40 made of a resilient material, such as rubber or foam, disposed between the IMU 36 and the flight controller 32. The dampers 40 can provide vibration isolation and / or damping, which can reduce interference from vibration noise, such as periodic vibrations from a motor or other source. The dampers 40 can also reduce shock loads transmitted to the IMU in the event of a hard landing and / or crash.

[0045] In some embodiments, the IMU bracket 38 can provide a rigid connection between the IMU 36 and the housings 22, 24. Such a rigid connection may not include any vibration isolation between the IMU 36 and the flight controller 32 or any other components of the vehicle controller 32.

[0046] The vehicle controller 20 also includes a mission computer 42. The mission computer 42 may include one or more processors mounted on a PCB. However, the mission computer 42 may take other forms, such as two or more components, or may be a single integrated circuit.

[0047] The vehicle controller 20 also includes a baseboard 52. The baseboard 52 may include a PCB having a plurality of external ports 54 for connecting to devices external to the housings 22, 24 via the holes 30. The baseboard 52 also includes a plurality of internal ports 60 for accepting electrical connections to devices within the housings 22, 24, such as the flight controller 32 and the mission computer 42. The baseboard 52 may provide power distribution and / or communication between the flight controller 32 and the mission computer 42 devices and / or from one or more of the internal devices and the external ports 54.

[0048] The vehicle controller 20 also includes an LTE module 62 for providing wireless data communications over a cellular network. The LTE module 62 includes a pair of antenna ports 64. Alternatively, or additionally, the LTE module 62 may include one or more onboard antennas. The LTE module 62 may include one or more components mounted on a PCB that may extend parallel to the baseboard 52, with the baseboard 52 being sandwiched between the LTE module 62 and the mission computer 42. The LTE module 62 may connect to the baseboard 52 via one or more of the internal ports 60 to receive power and / or provide communications between the LTE module 62 and the mission computer 42.

[0049] FIG. 2 shows a block diagram of an autonomous vehicle 10 including a vehicle controller 20 configured as a vehicle controller 20. FIG. 2 shows details of the vehicle controller 20, including several functional units of the flight controller 32 and the mission computer 42. The mission computer 42 includes a first processor 70 coupled to a first memory 72 that holds first instructions 74 for execution by the first processor 70. The first memory 72 also holds first data 75, such as parameters and / or settings for operating the mission computer 42. The first data 75 may include mission data, such as data from one or more cameras and / or other sensors. The first memory 72 may be integrated into a package with the first processor 70. Alternatively, or additionally, the first memory 72 may include one or more packages separate from but functionally in communication with the first processor 70. The first memory 72 may include flash storage, although other types of non-volatile memory may be used. The first memory 72 may also include one or more removable storage devices, such as an SD card. Alternatively or additionally, the first memory 72 may include one or more non-removable storage devices, such as an eMMC storage device and / or a solid-state drive (SSD). The first memory 72 may include a storage capacity of at least approximately 16 GB. However, the first memory 72 may have a larger storage capacity.

[0050] The first processor 70 may include a mission computer, which may include one or more specialized and / or general-purpose processors with relatively high performance for performing computationally intensive tasks, such as machine vision. The first processor 70 may include multiple processing cores. In some embodiments, the first processor may include four or more processing cores. Each processing core of the first processor 70 may operate at a clock speed of 1.8 GHz or greater. The first processor 70 may include 4 GB of temporary storage memory (RAM). However, the first processor 70 may include any amount of temporary storage memory.

[0051] The mission computer 42 also includes one or more wireless communication radios 76, 78, including a Wi-Fi radio 76 and a Bluetooth® radio 78. The Wi-Fi radio 76 may support one or more Wi-Fi standards, such as, for example, any of the Wireless LAN (WLAN) and / or mesh standards included in the Institute of Electrical and Electronics Engineers (IEEE) 802.11x standard. The wireless communication radios 76, 78 may provide short-range and / or medium-range wireless digital communications with the mission computer 42. The wireless communication radios 76, 78 may be located on the same package as the first processor 70. The wireless communication radios 76, 78 may support other wireless communication standards, such as Zigbee®, Near Field Communication (NFC), etc.

[0052] The mission computer 42 is in operative communication with the LTE module 62 for communicating with remote devices over a wireless data network. The mission computer 42 is also configured to receive data from a camera 80 having a field of view 81 via a direct camera interface 82. The camera 80 may include mono and / or stereoscopic vision. The camera 80 may be configured to capture images in the visible light spectrum and / or non-visible spectrums, such as infrared (IR) or ultraviolet (UV). The direct camera interface 82 may include a high-speed digital interface, such as a High-Definition Multimedia Interface (HDMI) interface or a Mobile Industrial Processor Interface (MIPI) camera interface. The direct camera interface 82 may be configured to transmit video, still image, and / or audio data from the camera 80 to the mission computer 42. The direct camera interface 82 may also transmit data, such as control signals, from the mission computer 42 to the camera 80.

[0053] The vehicle controller 20 also includes an Ethernet (ETH) switch 84 connected to each of the flight controller 32 and the mission computer 42 for transmitting digital data therebetween. Payload devices 86, such as a camera gimbal and / or one or more sensors, may also be connected to the Ethernet switch 84. Thus, the mission computer 42 may send and receive data and / or control signals to and from the payload devices 86.

[0054] The flight controller 32 includes a second processor 90 coupled to a second memory 92 that stores second instructions 94 for execution by the second processor 90. The second memory 92 also stores second data 95, such as parameters and / or settings for operating the flight controller 32. The second memory 92 may be integrated into a package with the second processor 90. Alternatively, or additionally, the second memory 92 may include one or more packages that are separate from, but in functional communication with, the second processor 90. The second memory 92 may include flash storage, although other types of non-volatile memory may be used. The second memory 92 may include a storage capacity of at least approximately 16 GB. However, the second memory 92 may have a larger storage capacity.

[0055] The second processor 90 may include one or more special-purpose and / or general-purpose processors configured for low-power operation. The second processor 90 may include one or more processing cores. For example, the second processor 90 may include a 32-bit ARM Cortex M4 core with a floating-point unit (FPU). The second processor 90 may include a second 32-bit core for redundancy. Each processing core of the second processor 90 may operate at a clock speed of 200 MHz to 480 MHz or more. The second processor 90 may include 1 MB to 2 MB of temporary storage memory (RAM). However, the second processor 90 may include any amount of temporary storage memory.

[0056] The flight controller 32 communicates with the IMU 36. For example, the IMU 36 may have a direct digital connection to the flight controller 32. The direct digital connection may include any serial or parallel data connection. The flight controller 32 also communicates with a Global Positioning System (GPS) receiver 88. The GPS receiver 88 may have a direct digital connection to the flight controller 32, such as a serial or parallel data connection. Alternatively or additionally, the flight controller 32 may receive position data from the GPS receiver 88 via a data network, such as an Ethernet switch 84 or a universal serial bus (USB) hub. The flight controller 32 also operatively communicates with one or more control actuators 96, 98. The control actuators 96, 98 may include a servo motor 96 coupled to a control surface 97 to control the position of the control surface. Additionally or alternatively, the control actuators 96, 98 may include an electric speed control (ESC) 98 electrically connected to an electric motor 99 and configured to control the speed of the connected electric motor 99. As an example, one servo motor 96 and one ESC 98 are shown in FIG. 2 . However, an autonomous vehicle may include several different servo motors 96 and / or ESCs 98, depending on the configuration of the autonomous vehicle. For example, an autonomous vehicle configured as a quadcopter with four lifting rotor blades may include four ESCs 98 and no servo motor 96. An autonomous vehicle configured as a fixed-wing aircraft may include one or more ESCs 98 for controlling electric motors 99 connected to the propellers and several servo motors 96 configured to control the movement of corresponding control surfaces 97, such as the rudder, elevator, and ailerons. An autonomous vehicle configured as a vertical take-off and landing (VTOL) aircraft may include a combination of an ESC 98 used to control electric motors 99 connected to the rotor blades and one or more servo motors 96 configured to move the corresponding control surfaces 97.

[0057] The flight controller 32 may execute a fast control loop to periodically update commands to the control actuators 96, 98 to control the position and / or attitude of the autonomous vehicle based on data regarding the current position, attitude, and / or rate of change of the position and / or attitude, which may be provided by the IMU 36 and / or GPS receiver 88. The flight controller 32 may control the position and / or attitude to direct the autonomous vehicle toward a position and / or path, which may be provided by the mission computer 42. In some embodiments, the mission computer 42 may function as an autopilot and instruct the flight controller 32 to move the autonomous vehicle between a series of waypoints. In some embodiments, the mission computer 42 may instruct the flight controller 32 to change course based on, for example, obstacle avoidance, and / or to land at a landing location, which the mission computer 42 may determine is a safe place for landing.

[0058] FIG. 3 illustrates a block diagram of a vehicle controller 20 for an autonomous vehicle according to one embodiment of the present disclosure. The vehicle controller 20 may be similar to or identical to the vehicle controller 20 illustrated in FIGS. 1-2. The vehicle controller 20 includes a flight controller 32, a mission computer 42, and an Ethernet switch 84 configured to provide communications between the flight controller 32 and the mission computer 42. The vehicle controller 20 includes a direct camera interface 82 in the form of an HDMI and / or MIPI port coupled to the mission computer 42 for connection to one or more cameras 80. A second controller. A USB-C port 110 is connected to the mission computer.

[0059] The power management unit 114 is configured to receive power from a first power source 116 and a second power source 118. The power sources 116, 118 may include one or more batteries. The power sources 116, 118 may be configured for redundant operation, allowing the vehicle controller 20 to operate in response to an interruption of power from one of the power sources 116, 118. Either or both of the power sources 116, 118 may be configured to receive power from another source, such as one or more solar cells, or from a generator or alternator coupled to the autonomous vehicle's internal combustion engine. The power management unit 114 may be configured to distribute power from the power sources 116, 118 to other components in the second controller, such as the flight controller 32, the mission computer, and the Ethernet switch 84. The power management unit 114 may be configured to keep the components of the second controller functioning at full capacity, even in the event of a loss or interruption of power from either of the power sources 116, 118. Alternatively or additionally, the power management unit 114 may be configured to cause components within the vehicle controller 20 to operate in a reduced power mode in the event of a loss or interruption of power from any of the power sources 116, 118. The decision whether to operate components in a reduced power mode may depend on one or more factors. Such factors may include, for example, the operating mode of the second controller (e.g., human control or fully autonomous), the amount of charge remaining in a battery associated with a functioning one of the power sources 116, 118, or a desire to conserve energy to provide the autonomous vehicle with sufficient range to reach a predetermined location, such as a known safe landing zone.

[0060] The vehicle controller 20 includes a direct serial data communication link 120 between the flight controller 32 and the mission computer 42. This direct serial data communication link 120 can provide redundancy in the event that the Ethernet connection becomes unavailable. Alternatively or additionally, the direct serial data communication link 120 may be used to communicate messages such as periodic updates of GPS coordinates from the flight controller to the mission computer, and / or flight control commands and / or waypoint coordinates from the mission computer 42 to the flight controller 32.

[0061] The vehicle controller 20 also includes a USB hub 126 connected to the mission computer 42 and configured to provide multiple USB interfaces for communication with the mission computer 42. The vehicle controller 20 includes multiple USB ports 128 connected to the USB hub 126. FIG. 3 shows two such USB ports 128 configured as USB 2.0 with 6-pin JST-GH connectors for medium-speed connection to external devices. It should be understood that the vehicle controller 20 can include any number of USB ports 128, which can have different form factors and can operate using any version of the USB protocol, including, for example, USB 2.0, USB 3.0, USB 3.1, etc.

[0062] The vehicle controller 20 also includes a payload bus connector 130. The payload bus connector 130 may be configured based on the Pixhawk Payload Bus (PPB) standard. The payload bus connector 130 may combine USB, Ethernet, and / or serial (UART) data communication interfaces with the flight controller 32 and / or the mission computer 42. The vehicle controller 20 also includes an Ethernet port 124 coupled to the Ethernet switch 84. The Ethernet port 124 may include an RJ-45 connector. However, the Ethernet port may include other types of connectors, such as a JST connector.

[0063] 4 shows a block diagram of a system 150 for controlling one or more autonomous vehicles 10 according to one embodiment of the present disclosure. System 150 includes one or more autonomous vehicles 10, each having one or more corresponding vehicle controllers 20. For simplicity, a single autonomous vehicle 10 with one vehicle controller 20 is shown in FIG. 4. System 150 also includes a server 170 and a ground station 180.

[0064] The LTE module 62 of the vehicle controller is configured to communicate data to and from a server 170 over a communications network 160 using one or more digital wireless data connections 162. The communications network 160 may include the Internet and / or an isolated network. The communications network 160 may include a commercial wireless (i.e., cellular) data network, such as an LTE network. The vehicle controller 20 may be configured to transmit data, such as position and / or payload data, to the server 170 over the communications network 160. The server 170 may be configured to transmit data, such as mission planning and / or firmware updates, to the vehicle controller 20 over the communications network 160.

[0065] Server 170 includes a third processor 172 coupled to a third memory 174 that holds third instructions 176 for execution by third processor 172. Third memory 174 also holds third data 178, such as historical data for the fleet of the plurality of autonomous vehicles 10. Third data 178 may be stored in a database in third memory 174. Third data 178 may be distributed among one or more physical computer devices that may be physically separate from one another. Server 170 may be configured to present the historical data via a web interface. Alternatively or additionally, server 170 may be configured to present the historical data via other interfaces, such as programs or applications on one or more computers or mobile devices (e.g., smartphones).

[0066] Ground station 180 includes a fourth processor 182 coupled to a fourth memory 184 that holds fourth instructions 186 for execution by fourth processor 182. Fourth memory 184 also holds fourth data 188. Ground station 180 may present a user interface 190 for monitoring and / or controlling the operation of autonomous vehicle 10. For example, mission computer 42 may be configured to set and / or modify waypoints within a mission to be followed by autonomous vehicle 10. Alternatively or additionally, ground station 180 may provide an interface for an operator to control and / or monitor the operation of one or more payload devices, such as sensors, aboard the autonomous vehicle.

[0067] 4 , the autonomous vehicle 10 may include a flight control radio 194 that communicates with the flight controller 32 and / or the mission computer 42. The flight control radio 194 may provide two-way communication between the ground station 180 and the vehicle controller 20. The flight control radio 194 may receive commands from the ground station 180. Additionally or alternatively, the flight control radio 194 may transmit status data, such as telemetry data and / or payload data, from the vehicle controller 20 to the ground station 180. Alternatively or additionally, the ground station 180 may communicate with the mission computer 42 via the Wi-Fi radio 76 and / or the Bluetooth radio 78.

[0068] FIG. 5 shows a block diagram of software components within a system 200 for controlling an autonomous vehicle 10.

[0069] Server 170 is configured to execute a cloud suite of software 210. Cloud suite 210 includes a back-end processor 212. Back-end processor 212 may provide low-level communication with one or more of vehicle controllers 20. Back-end processor 212 may provide an interface to a database or other data storage system for storing and / or retrieving data regarding the operation of one or more autonomous vehicles. Cloud suite 210 also includes an OEM dashboard 214, an operator suite 216, and a third-party cloud API 218.

[0070] The OEM dashboard 214 may provide data and controls for visualizing and summarizing fleet data and / or data regarding planned and / or completed missions for one or more autonomous vehicles 10. The OEM dashboard 214 may also provide the ability to monitor fleet performance and / or pilot compliance and efficiency. The OEM dashboard 214 may also enable remote management of automatic software updates for updating software in other processors, such as the flight controller 32, the mission computer 42, and / or the payload controller. The operator suite 216 may enable remote control of the autonomous vehicle 10. For example, a pilot may modify the mission settings of the autonomous vehicle 10 and / or use the operator suite 216 to provide manual control for flying the autonomous vehicle 10. The operator suite 216 may provide controls for the pilot to control the autonomous vehicle 10. The operator suite 216 may also provide data regarding the operation of the autonomous vehicle to the pilot. Such data may include, for example, one or more live video feeds, telemetry data, position and heading, etc.

[0071] The third-party cloud API 218 provides a standardized interface for programs or other controllers to interface with the cloud suite 210. For example, the third-party cloud API 218 may enable a client application to obtain specific reporting data. Additionally or alternatively, the third-party cloud API 218 may enable an interface for monitoring and / or controlling the vehicle controller 20 and / or one or more payload devices onboard the autonomous vehicle 10.

[0072] As also shown in FIG. 5 , the vehicle controller 20 may execute an onboard software suite 230. The onboard software suite 230 includes flight control software 232, an operating system (OS) distribution 234, an autonomy stack 236, a MAVSDK application 238, and a third-party airframe configuration 240. The flight control software 232 may execute on the flight controller 32 and may control the operation of the control actuators 96, 98 to operate the autonomous vehicle 10. The OS distribution 234 may execute on the mission computer 42 and may be configured to control updates and version management of software applications executing on the mission computer 42, the flight controller 32, and / or any other onboard controllers, such as within one or more payload controllers. The autonomy stack 236 may operate on the mission computer 42 and may enable autonomous operation of the autonomous vehicle 10. In the most basic sense, the autonomy stack 236 may function as an autopilot, enabling the autonomous vehicle 10 to navigate between predetermined waypoints. Additionally or alternatively, the autonomous stack 236 may provide machine vision that may be used for obstacle avoidance or safe landing zone detection.

[0073] The MAVSDK application 238 may provide messaging between devices and / or processors onboard the autonomous vehicle and / or between the vehicle controller 20 and the server 170 and / or ground station 180. The MAVSDK application 238 may implement the MAVLink protocol. Additionally or alternatively, other messaging protocols may be used. The messaging protocol used may depend on the communication interface (e.g., serial, Ethernet, or USB) used to convey the messages.

[0074] Third-party airframe configuration 240 may include settings and / or applications tailored to a particular airframe configuration. For example, third-party airframe configuration 240 for an autonomous vehicle 10 configured as a quadcopter may differ from third-party airframe configuration 240 for an autonomous vehicle 10 configured as a fixed-wing aircraft.

[0075] As also shown in FIG. 5 , the ground station 180 may execute a ground station software suite 250. The ground station software suite 250 includes an Auterion Mission Control (AMC) 252, a management layer 254, and a user interface generator 256. The AMC 252 may provide low-level operations such as establishing and maintaining communications with the vehicle controller 20. The management layer 254 may translate and / or store data between the AMC 252 and the user interface generator 256. The user interface generator 256 may provide displays and / or controls for use by a pilot interfacing with the user interface 190 of the ground station 180. For example, the user interface generator 256 may provide a graphical user interface for applications running on the user interface 190. The graphical user interface may include several different displays and controls for adjusting various parameters related to the operation of the autonomous vehicle 10. The graphical user interface may include, for example, a waypoint control interface for viewing and modifying waypoints followed by the autonomous vehicle. The pilot can click on a waypoint control interface on the user interface 190 and enter coordinates of the modified waypoint. The management layer 254 can receive the modified waypoint coordinates, verify that they are valid, and pass the modified waypoint coordinates to the AMC 252. The AMC 252 can send the modified waypoint coordinates to the vehicle controller 20 and confirm that the modified waypoint coordinates were received.

[0076] 6 is a flow diagram generally illustrating a method 300 of operating an autonomous vehicle 10 in accordance with the principles of the present disclosure. At 302, the method 300 measures certain forces acting on the autonomous vehicle 10 via the inertial measurement unit (IMU) 36. For example, the IMU 36 may include one or more sensors, such as an accelerometer and / or a gyroscope, to determine acceleration in each of three orthogonal dimensions and / or rotation about each of three orthogonal axes.

[0077] At 304, method 300 includes, by flight controller 32, sending commands to each of a plurality of control actuators 93, 98 to control the position or attitude of autonomous vehicle 10 based on data from IMU 36 regarding specific forces acting on autonomous vehicle 10. Flight controller 32 may repeatedly perform step 304 using a high-speed flight control loop. In some embodiments, the flight control loop may operate at a rate of at least about 100 Hz. In some embodiments, the flight control loop may operate at a rate of at least about 1 kHz. In some embodiments, the flight control loop may operate periodically based on a clock signal.

[0078] At 306, method 300 includes, by the mission computer 42, commanding the flight controller 32 to change the position or orientation of the autonomous vehicle 10. For example, the mission computer 42 may command the flight controller 32 with a specific command, such as turn right, to cause the autonomous vehicle 10 to change heading or direction. Step 306 may include, for example, the mission computer 42 determining a command position based on one or more different criteria. In some embodiments, for example, the mission computer 42 may act as an autopilot and instruct the flight controller 32 to move to the next waypoint after the autonomous vehicle 10 passes a given position.

[0079] At 308, method 300 includes having mission computer 42 communicate with server 170 using a cellular data radio. For example, mission computer 42 can communicate with server 170 wirelessly over communication network 160. The cellular data radio can include, for example, an LTE module 62. Mission computer 42 can communicate with server 170 while autonomous vehicle 10 is moving. For example, mission computer 42 can communicate with server 170 using LTE module 62 while autonomous vehicle 10 is executing a mission. Alternatively or additionally, mission computer 42 can communicate with server 170 while autonomous vehicle 10 is stationary. For example, mission computer 42 can communicate with server 170 using LTE module 62 while autonomous vehicle 10 is parked at a home base location.

[0080] In some embodiments, step 308 may include transmitting a system software package by server 170. For example, updated firmware for a particular type of hardware may be made available by server 170. Step 308 may also receive the system software package by mission computer 42. Step 308 may also install firmware updates on at least one of mission computer 42 or flight controller 32 based on the system software package. For example, mission computer 42 may extract one or more firmware updates from the system software package and cause the one or more firmware updates to be installed and executed on mission computer 42 and / or flight controller 32. Mission computer 42 may wait and install firmware updates only when autonomous vehicle 10 is in a predetermined safe state, such as landed and / or plugged into a battery charger.

[0081] The method 300 may further include the IMU 36, flight controller 32, mission computer 42, and LTE module 62 all located within a housing onboard the autonomous vehicle.

[0082] At 310, method 300 can include, by mission computer 42, monitoring video feeds from cameras. For example, mission computer 42 can read and process streams of video data from one or more cameras 80 located on the autonomous vehicle. One or more cameras 80 may include stereo cameras capable of detecting visible light and / or light outside the visible spectrum. Additionally or alternatively, mission computer 42 can receive and process video feeds from one or more cameras located outside of autonomous vehicle 10, such as cameras mounted on another vehicle and / or cameras at fixed locations.

[0083] At 312, method 300 may determine, by the mission computer 42, one of an obstacle in the path of the autonomous vehicle 10 or a safe landing zone for landing the autonomous vehicle 10 based on the video feed. For example, the mission computer 42 may perform one or more image recognition techniques to determine and classify the obstacle and determine the location and / or path of the obstacle to determine whether the autonomous vehicle is at risk of colliding with the obstacle. In another example, the mission computer 42 may perform one or more image recognition techniques to determine and classify an area of ​​ground near the autonomous vehicle 10 and determine that area of ​​ground is a safe landing zone where the autonomous vehicle 10 can safely land. Image recognition techniques may include, for example, pattern matching or machine learning (ML).

[0084] At 314, method 300, which included instructing flight controller 32 to change the position or orientation of autonomous vehicle 10 in step 306, may further include instructing flight controller 32 to change the position or orientation based on determining one of an obstacle or a safe landing zone. For example, mission computer 42 may instruct flight controller 32 to perform an evasive maneuver to avoid collision with an obstacle. In another example, mission computer 42 may instruct flight controller 32 to land autonomous vehicle 10 at a location determined to be a safe landing zone, provided other conditions for landing are met.

[0085] In some embodiments, a vehicle controller for operating an autonomous vehicle includes an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle and a flight controller configured to execute a high-speed control loop to periodically update commands to multiple control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle. The vehicle controller may also include a mission computer including a processor programmed to control the mission of the autonomous vehicle. For example, the processor may control the navigation of the autonomous vehicle between multiple waypoint locations using position data from one or more sources, such as a GPS receiver. The mission of the autonomous vehicle may also include controlling one or more payload devices. The vehicle controller may also include a cellular data radio configured to wirelessly communicate over a cellular network. The cellular data radio may include, for example, an LTE module 62. The vehicle controller may also include a housing that houses the IMU, flight controller, mission computer, and cellular data radio.

[0086] In some embodiments, the mission computer may be configured to monitor a video feed from a camera process, use the video feed to detect objects, and cause the autonomous vehicle to avoid collisions with the objects.

[0087] In some embodiments, the mission computer may be configured to monitor video feeds from the cameras and process the video feeds to detect safe landing zones for landing the autonomous vehicle.

[0088] In some embodiments, the mission computer may be configured to monitor the video feed from the camera and encode the video feed into a compressed format for transmission to a remote receiver.

[0089] In some embodiments, the vehicle controller may further include a Wi-Fi radio disposed within the housing. The Wi-Fi radio may be configured to communicate with a ground station for controlling the autonomous vehicle. For example, the Wi-Fi radio may receive commands, such as one or more attitude or positioning commands or waypoint positioning commands, from the ground station. In some embodiments, the Wi-Fi radio may transmit information to the ground station. For example, the Wi-Fi radio may transmit telemetry data and / or live video data from one or more cameras mounted on the autonomous vehicle.

[0090] In some embodiments, the vehicle controller may further include a Wi-Fi radio disposed within the housing and configured to communicate with a second autonomous vehicle. For example, the autonomous vehicle and the second autonomous vehicle may share data or commands using the Wi-Fi radio. In some embodiments, multiple autonomous vehicles, each having a Wi-Fi radio, may form a mesh network.

[0091] In some embodiments, the vehicle controller may further include an Ethernet switch disposed within the housing and connected to the mission computer.

[0092] In some embodiments, the vehicle controller may further include a wired data connection to a second vehicle controller, and the flight controller is configured to receive control commands from the second vehicle controller in response to determining that the mission computer is in a fault condition. For example, an autonomous vehicle may include two independent vehicle controllers configured for redundant operation. In some embodiments, the wired data connection to the second vehicle controller may include an Ethernet connection. However, other types of wired data connections may be used.

[0093] In some embodiments, the IMU may comprise a substrate having a heat spreader base of thermally conductive material configured to provide a temperature differential across the substrate within a predetermined range.

[0094] In some embodiments, the vehicle controller may further include a bracket that holds the IMU to at least one of the flight controller or the housing. In some embodiments, the bracket may be configured to isolate the IMU from heat generated by at least one of the flight controller or the mission computer. In some embodiments, the bracket may include a damper configured to isolate the IMU from vibrations. In some embodiments, the bracket may include a rigid mount configured to transfer vibrations between the housing and the IMU. For example, the bracket may not include an isolation damper to isolate the IMU from vibrations.

[0095] In some embodiments, the flight controller may be configured to sample and filter data from the IMU at a rate substantially faster than the fast control loop, for example, the sampling and filtering of data from the IMU may be performed 4 to 100 times faster than the fast control loop.

[0096] In some embodiments, a vehicle controller for operating an autonomous vehicle includes an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle and a flight controller configured to execute a high-speed control loop to periodically update commands to a plurality of control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle. In some embodiments, the vehicle controller also includes a mission computer including a processor programmed to control the mission of the autonomous vehicle, and a housing that houses the IMU, flight controller, and mission computer. In some embodiments, the mission computer includes an application programming interface for providing vehicle status information and payload data to third-party applications. For example, the vehicle status information may include position and / or telemetry data. The payload data may include sensor data, such as images or video, from one or more payload sensors.

[0097] In some embodiments, the application programming interface may be further configured to receive commands from a third-party application. For example, the commands may include one or more attitude or positioning commands, or one or more commands for a destination position of the autonomous vehicle.

[0098] In some embodiments, a method of operating an autonomous vehicle is provided. The method includes measuring specific forces acting on the autonomous vehicle with an inertial measurement unit (IMU), sending commands by a flight controller to each of a plurality of control actuators to control a position or attitude of the autonomous vehicle based on data from the IMU regarding the specific forces acting on the autonomous vehicle, instructing the flight controller by a mission computer to change the position or orientation of the autonomous vehicle, and communicating with a server by the mission computer using a cellular data radio. In some embodiments, the IMU, flight controller, mission computer, and cellular data radio are all located within a housing onboard the autonomous vehicle.

[0099] In some embodiments, communicating with the server further includes transmitting, by the server, a system software package, receiving, by the mission computer, the system software package, and installing a firmware update on at least one of the mission computer or the flight controller based on the system software package. For example, the mission computer can be programmed to coordinate receiving the system software package, extracting the firmware update from the system software package, and / or installing the firmware update on the mission computer and / or the flight controller.

[0100] In some embodiments, the method of operating an autonomous vehicle further includes monitoring, by a mission computer, a video feed from the camera, and determining, by the mission computer based on the video feed, one of an obstacle in a path of the autonomous vehicle or a safe landing zone for landing the autonomous vehicle. In some embodiments, commanding the flight controller to change a position or orientation of the autonomous vehicle includes commanding the flight controller to change the position or orientation based on the determining one of the obstacle or the safe landing zone.

[0101] The computer executable code may be written using a structured programming language such as C, an object-oriented programming language such as C++, or any other high-level or low-level programming language (including assembly language, hardware description languages, and database programming languages ​​and techniques) that can be stored, compiled, or interpreted to be executed by one of the above devices, as well as heterogeneous combinations of processor processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.

[0102] Thus, in one aspect, each of the methods described above and combinations thereof may be implemented in computer-executable code that performs the steps when executed on one or more computing devices. In another aspect, the method may be implemented in a system that performs its steps, may be distributed in some manner among devices, or all of the functionality may be integrated into a dedicated stand-alone device or other hardware. In another aspect, the means for performing the steps associated with the processes described above may include any of the hardware and / or software described above. All such permutations and combinations are intended to fall within the scope of the present disclosure.

[0103] The foregoing description is not intended to be exhaustive or to limit the present disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, are interchangeable where applicable, and can be used in selected embodiments even if not specifically shown or described. It may also be modified in many ways. Such variations should not be considered a departure from the present disclosure, and all such modifications are intended to be included within the scope of the present disclosure.

Claims

1. 1. A vehicle controller for operating an autonomous vehicle, comprising: an inertial measurement unit (IMU) configured to measure specific forces acting on the autonomous vehicle; a flight controller configured to execute a fast control loop to periodically update commands to a plurality of control actuators for controlling the position or attitude of the autonomous vehicle based on data from the IMU regarding the particular forces acting on the autonomous vehicle; and a mission computer including a processor programmed to control a mission of the autonomous vehicle; a cellular data radio configured to communicate wirelessly over a cellular network; a housing containing the IMU, the flight controller, the mission computer, and the cellular data radio; the vehicle controller further comprises a radio disposed within the housing and configured to communicate with a second autonomous vehicle; the housing is independent of the structure of the autonomous vehicle and configured to be removably attached to the structure of the autonomous vehicle along with the IMU, the flight controller, the mission computer, and a cellular data radio; The vehicle controller also a plurality of mounting tabs, each of the plurality of mounting tabs extending from the housing and receiving a fastener to secure the vehicle controller to a structure of the autonomous vehicle; or a plurality of electrical connectors, each configured to be selectively and removably attached to an external device outside the housing; a vehicle controller including at least one of:

2. The mission computer monitors a video feed from a camera process and uses the video feed to: Detecting an object and causing the autonomous vehicle to avoid a collision with the object; Detecting a safe landing zone for landing the autonomous vehicle; or The vehicle controller of claim 1 , configured to encode the video feed into a compressed format for transmission to a remote receiver.

3. A vehicle controller as described in claim 1, wherein the radio is a Wi-Fi radio and is configured to communicate with a ground station for controlling the autonomous vehicle.

4. The vehicle controller of claim 1 , further comprising an Ethernet switch disposed within the housing and connected to the mission computer.

5. 10. The vehicle controller of claim 1, further comprising a wired data connection to a second vehicle controller, the flight controller configured to receive control commands from the second vehicle controller in response to determining that the mission computer is in a fault condition.

6. The vehicle controller of claim 5 , wherein the wired data connection to the second vehicle controller comprises an Ethernet connection.

7. 10. The vehicle controller of claim 1, wherein the IMU comprises a substrate having a heat spreader base of a thermally conductive material, the substrate configured to provide a temperature differential across the substrate within a predetermined range.

8. The vehicle controller of claim 1 , further comprising a bracket that holds the IMU to at least one of the flight controller or the housing.

9. The vehicle controller of claim 8 , wherein the bracket is configured to isolate the IMU from heat generated by at least one of the flight controller or the mission computer.

10. The vehicle controller of claim 1 , wherein the flight controller is configured to sample and filter the data from the IMU at a rate substantially faster than the fast control loop.

11. The vehicle controller of claim 1 , wherein the mission computer includes an application programming interface for providing vehicle status information and payload data to third party applications.

12. The vehicle controller of claim 11 , wherein the application programming interface is further configured to receive commands from the third-party application.

13. 1. A method of operating an autonomous vehicle, comprising: measuring specific forces acting on the autonomous vehicle with an inertial measurement unit (IMU); sending commands by a flight controller to each of a plurality of control actuators to control a position or attitude of the autonomous vehicle based on data from the IMU regarding the particular forces acting on the autonomous vehicle; commanding, by a mission computer, the flight controller to change the position or orientation of the autonomous vehicle; communicating with a server by said mission computer using a cellular data radio; the IMU, the flight controller, the mission computer, and the cellular data radio are all located within a housing onboard the autonomous vehicle; The method further includes communicating with a second autonomous vehicle via a radio disposed within the housing; the housing is independent of the structure of the autonomous vehicle and configured to be removably attached to the structure of the autonomous vehicle along with the IMU, the flight controller, the mission computer, and a cellular data radio; The autonomous vehicle includes: a plurality of mounting tabs, each of the plurality of mounting tabs extending from the housing and receiving a fastener to secure a controller of the autonomous vehicle to a structure of the autonomous vehicle; or a plurality of electrical connectors, each configured to be selectively and removably attached to an external device outside the housing; The method includes at least one of the following:

14. The step of communicating with the server comprises: transmitting, by the server, a system software package; receiving the system software package by the mission computer; and installing a firmware update to at least one of the mission computer or the flight controller based on the system software package. The method of claim 13.

15. monitoring, by the mission computer, a video feed from a camera; determining, by the mission computer, based on the video feed, one of an obstacle in a path of the autonomous vehicle or a safe landing zone for landing the autonomous vehicle; commanding the flight controller to change the position or orientation of the autonomous vehicle includes commanding the flight controller to change the position or orientation based on determining the one of the obstacle or the safe landing zone. The method of claim 13.

Images