Autonomous parcel delivery system

Abstract

The invention discloses an autonomous parcel delivery system. The present disclosure relates to systems and methods for enabling unmanned and optionally manned delivery of goods to ground persons. For example, an aircraft may be used to provide fast response cargo delivery to widely separated small units under harsh and unpredictable conditions that cause unacceptable risks to both ground rereplenishment personnel and crew personnel. Among ground vehicles, parcels from the aircraft may be deployed to ground personnel at interspersed combat locations without exposing the ground personnel to an open landing zone of the aircraft.

Description

[0001] This application is a divisional application of Chinese patent application 2018108943346, entitled “Autonomous Package Delivery System”, filed on August 8, 2018. Technical Field

[0002] The present disclosure relates to systems and methods for enabling unmanned and optionally manned cargo delivery. Background Art

[0003] Supply and casualty evacuation (CASEVAC) often results in casualties on the ground and in the air under hazardous field conditions. However, building infrastructure to facilitate CASEVAC is often impractical in this field. Therefore, a number of development programs have been pursued to facilitate package delivery under such austere conditions that are common to a variety of mission sets, such as humanitarian relief operations, noncombatant evacuations, routine cargo resupply, en route resupply, unconventional warfare, and ongoing conventional combat.

[0004] A package delivery method is precision airdrop, which involves dropping a package from an aircraft to the ground. For example, the Joint Precision Airdrop (JPAD) system tries its best to deliver the package to a specified waypoint. However, the JPAD system must be first programmed before it is loaded on the aircraft, and once it is deployed, it uses GPS data to form a glide path to the waypoint. The JPAD system can use airspeed data and self-adjust in flight and terrain data to avoid obstacles that would otherwise cause the drop to be damaged or miss its waypoint. Another method is to use an unmanned aerial vehicle (UAV) to deliver the package to the waypoint. However, each of these methods requires an unobstructed flight path to the waypoint, which generally requires the waypoint to be in an open area without obstacles (e.g., trees, mountains, etc.). However, such open areas expose ground personnel to unexpected attention and damage when collecting packages from aircraft. Therefore, delivery to a covered location can enhance the safety of ground personnel.

[0005] Instead of an aircraft, an unmanned ground vehicle (UGV) can be used to deliver packages to the destination. In practice, personnel can use remote control to navigate these UGVs from a covered location to a covered location. However, a limitation of UGVs is that a ground path must be available for the UGV to navigate to a waypoint. For example, obstacles (e.g., streams, canyons, buildings, etc.) may prevent the UGV from approaching a waypoint. Therefore, although UGVs are more flexible on the ground, UGVs are not suitable for long-distance travel to waypoints, which is a benefit for aircraft. In addition, UGVs have a limited field of view outside the surrounding environment they are close to, which introduces additional navigation challenges. Therefore, there is a need for systems and methods for achieving unmanned and optionally manned cargo delivery to ground personnel. Summary of the invention

[0006] The present disclosure provides systems and methods for enabling unmanned and optionally manned cargo delivery to ground personnel.

[0007] According to a first aspect, a package delivery system for use with an aircraft comprises: a land vehicle equipped with a first processor operably coupled to a first sensor package and a first communication transceiver, the first processor being configured to navigate the land vehicle along a navigation land route based at least in part on data received via the first communication transceiver; a perception system having a second sensor package operably coupled to a second processor and the second communication transceiver; and a human-system interface (HSI) device for communicating with the land vehicle and the perception system via one or more wireless links, wherein the HSI device, separate and distinct from each of the land vehicle and the perception system, comprises a third processor, a third communication transceiver, a user input device, and a display device, wherein the perception system is configured to transmit navigation commands to the land vehicle via the first communication transceiver based at least in part on data from the second sensor package, and wherein the HSI device is configured to transmit navigation commands to the land vehicle via the third communication transceiver and to receive a video feed captured by the first sensor package from the land vehicle.

[0008] In certain aspects, the second sensor package includes a first optical payload and a second optical payload, each of the first optical payload and the second optical payload being positioned at a different location on the aircraft.

[0009] In certain aspects, the first optical payload is configured to generate terrain data of the geographic area prior to landing at a landing zone within the geographic area, wherein the second processor is configured to determine a navigational land route from the landing zone to a field operator based at least in part on the terrain data.

[0010] In certain aspects, the second optical payload is configured to track the land vehicle from the landing zone as the land vehicle travels along the navigation land route.

[0011] In certain aspects, each of the first optical payload and the second optical payload comprises a LIDAR or stereo vision camera system.

[0012] In certain aspects, the second processor is configured to determine a plurality of navigational land routes from the landing zone to the field operator based at least in part on the terrain data.

[0013] In certain aspects, the HSI device is configured to present the plurality of navigational land routes to the field operator via the display device for selection by the field operator via the user input device.

[0014] In certain aspects, the perception system is configured to, in real time, (1) detect obstacles along the navigation land route, and (2) dynamically emit navigation commands to autonomously navigate the land vehicle as the land vehicle travels along the navigation land route.

[0015] In certain aspects, the HSI device is configured to display the video feed on the display device.

[0016] In certain aspects, the HSI device is configured to receive a second video feed captured by the second sensor package from the perception system, and the HSI device is configured to display the second video feed on the display device.

[0017] In certain aspects, navigation commands from the HSI device take precedence over navigation commands from the perception system.

[0018] In certain aspects, the user input device is a touch screen and the HSI device is configured to display a remote controller screen via the display device.

[0019] In certain aspects, the HSI device is configured to display a land vehicle route screen via the display device.

[0020] In certain aspects, the user input device is a touch screen, and the HSI device is configured to simultaneously display the video feed and one or more navigation controller icons via the display device.

[0021] In certain aspects, the aircraft is a vertical take-off and landing aircraft.

[0022] In certain aspects, the aircraft is configured to transport the land vehicle from a remote location to the landing zone.

[0023] In certain aspects, the aircraft is an unmanned aerial vehicle.

[0024] In certain aspects, the land vehicle is an unmanned ground vehicle.

[0025] According to a second aspect, an aircraft for transporting a land vehicle in a package delivery system comprises: a communications transceiver; a cargo hold for accommodating the land vehicle; a sensor package having a first optical payload and a second optical payload, wherein the first optical payload is configured to generate terrain data of a geographic area prior to landing at a landing zone within the geographic area, and the second optical payload is configured to track the land vehicle from the landing zone; and a processor operably coupled to the sensor package and the communications transceiver, wherein the processor is configured to (1) identify a landing zone within the geographic area, (2) autonomously navigate the aircraft to the landing zone, (3) determine a navigation land route for the land vehicle from the landing zone to a field operator based at least in part on the terrain data, and (4) transmit navigation commands to the land vehicle via the communications transceiver based at least in part on data from the second optical payload.

[0026] In certain aspects, the second optical payload is configured to observe the land vehicle from the landing zone as the land vehicle travels along the navigation land route.

[0027] In certain aspects, each of the first optical payload and the second optical payload comprises a LIDAR or stereo vision camera system.

[0028] In certain aspects, the processor is configured to determine a plurality of navigational land routes from the landing zone to the field operator based at least in part on the terrain data.

[0029] In certain aspects, the second optical payload is configured to, in real time, (1) detect obstacles along the navigation land route, and (2) dynamically transmit navigation commands to autonomously navigate the land vehicle as the land vehicle travels along the navigation land route.

[0030] In certain aspects, the aircraft is an unmanned aerial vehicle.

[0031] In certain aspects, the land vehicle is an unmanned ground vehicle.

[0032] In certain aspects, the aircraft is a vertical take-off and landing aircraft.

[0033] According to a third aspect, a human-system interface (HSI) device is used to facilitate two-way communication with a land vehicle and a perception system mounted to an aircraft at a landing zone, the perception system being configured to track the land vehicle as it travels along a navigational land route between the landing zone and a field operator, the human-system interface (HSI) device comprising: a processor; a display device; a user input device operably coupled to the processor; and a communication transceiver operably coupled to the processor to communicate with the land vehicle and the perception system via one or more wireless links, wherein the HSI device, which is separate and distinct from each of the land vehicle and the aircraft, is configured to: (1) transmit navigation commands to the land vehicle via the communication transceiver, (2) receive from the land vehicle via the communication transceiver a video feed captured by a first sensor package located on the land vehicle, and (3) receive from the perception system position data reflecting the position of the land vehicle along the navigational land route, wherein the position data is generated by a second sensor package of the perception system located on the aircraft.

[0034] In certain aspects, the display device is configured to display a plurality of navigational land routes for selection by the field operator via the user input device.

[0035] In certain aspects, the display device is configured to display the video feed.

[0036] In certain aspects, the HSI device is configured to receive a second video feed captured by the second sensor package from the perception system.

[0037] In certain aspects, the display device is configured to display the second video feed.

[0038] In certain aspects, the user input device is a touch screen and the display device is configured to display a remote controller screen.

[0039] In certain aspects, the display device is configured to display a land vehicle route screen.

[0040] In certain aspects, the user input device is a touch screen and the display device is configured to simultaneously display the video feed and one or more navigation controller icons.

[0041] According to a fourth aspect, an unmanned ground vehicle (UGV) for an aerial package delivery system comprises: a chassis for supporting a package; an optical payload for generating a video feed of a geographic area; a communication transceiver for facilitating two-way communication with a human-system interface (HSI) device and a perception system via one or more wireless links, wherein the perception system is coupled to an aircraft at a landing zone within the geographic area; and a processor operably coupled to the communication transceiver and each of the sensor packages, wherein the processor is configured to navigate the land vehicle along a navigational land route from the landing zone to a live operator based on navigation commands received from the HSI device or the aircraft via the communication transceiver, and transmit the video feed to the HSI device.

[0042] In certain aspects, the navigational land route is received from the perception system via the communication transceiver.

[0043] In certain aspects, the perception system is configured to track the land vehicle from the landing zone as the land vehicle travels along the navigation land route.

[0044] In certain aspects, the communication transceiver is configured to receive navigation commands from the aircraft and the HSI device in real-time to autonomously navigate the land vehicle along the navigation land route.

[0045] In certain aspects, the processor is configured to locate the UGV based on the position of the UGV relative to the aircraft.

[0046] Embodiments of the present invention relate to a package delivery system for use with an aircraft, the package delivery system comprising: a land vehicle equipped with a first processor operably coupled to a first sensor package and a first communication transceiver, the first processor being configured to navigate the land vehicle along a navigation land route based at least in part on data received via the first communication transceiver; a perception system having a second sensor package operably coupled to a second processor and the second communication transceiver; and a human-system interface (HSI) device for communicating with the land vehicle and the perception system via one or more wireless links, wherein the HSI device, separate and distinct from each of the land vehicle and the perception system, comprises a third processor, a third communication transceiver, a user input device, and a display device, wherein the perception system is configured to transmit navigation commands to the land vehicle via the first communication transceiver based at least in part on data from the second sensor package, and wherein the HSI device is configured to transmit navigation commands to the land vehicle via the third communication transceiver and receive a video feed captured by the first sensor package from the land vehicle. The second sensor package may include a first optical payload and a second optical payload, each of the first optical payload and the second optical payload being positioned at a different position of the aircraft. The first optical payload may be configured to generate terrain data of the geographic area before landing in a landing zone within the geographic area, wherein the second processor is configured to determine a navigation land route from the landing zone to a field operator based at least in part on the terrain data. The second optical payload may be configured to track the land vehicle from the landing zone as the land vehicle travels along the navigation land route. At least one of the first optical payload and the second optical payload may include a LIDAR or stereo vision camera system. The second processor may be configured to determine a plurality of navigation land routes from the landing zone to the field operator based at least in part on the terrain data. The HSI device may be configured to present the plurality of navigation land routes to the field operator via the display device for selection by the field operator through the user input device. The perception system may be configured to (1) detect obstacles along the navigation land route in real time, and (2) dynamically transmit navigation commands to autonomously navigate the land vehicle as the land vehicle travels along the navigation land route. The HSI device may be configured to display the video feed on the display device. The HSI device may be configured to receive a second video feed captured by the second sensor package from the perception system, and the HSI device is configured to display the second video feed on the display device.The navigation commands from the HSI device may take precedence over the navigation commands from the perception system. The HSI device may be configured to display a land vehicle route screen via the display device. The aircraft may be a vertical takeoff and landing aircraft. The aircraft may be configured to transport the land vehicle from a remote location to the landing zone.

[0047] Embodiments of the present invention relate to a human-system interface (HSI) device for facilitating two-way communication with a land vehicle and a perception system mounted to an aircraft at a landing zone, the perception system being configured to track the land vehicle as it travels along a navigational land route between the landing zone and a field operator, the HSI device may include: a processor; a display device; a user input device operably coupled to the processor; and a communication transceiver operably coupled to the processor to communicate with the land vehicle and the perception system via one or more wireless links, wherein the HSI device, separate and distinct from each of the land vehicle and the aircraft, is configured to: (1) transmit navigation commands to the land vehicle via the communication transceiver, (2) receive from the land vehicle via the communication transceiver a video feed captured by a first sensor package positioned on the land vehicle, and (3) receive from the perception system position data reflecting the position of the land vehicle along the navigational land route, wherein the position data is generated by a second sensor package of the perception system positioned on the aircraft. The display device may be configured to display a plurality of navigational land routes for selection by the field operator via the user input device. The display device may be configured to display the video feed. The device may be configured to receive a second video feed captured by the second sensor package from the perception system. The HSI device of claim 15, wherein the user input device is a touch screen and the display device is configured to simultaneously display the video feed and one or more navigation controller icons.

[0048] Another embodiment relates to an unmanned ground vehicle (UGV) for an aerial package delivery system, the UGV comprising: a chassis for supporting a package; an optical payload for generating a video feed of a geographic area; a communication transceiver for facilitating bidirectional communication with a human-system interface (HSI) device and a perception system via one or more wireless links, wherein the perception system is coupled to an aircraft at a landing zone within the geographic area; and a processor operably coupled to each of the communication transceiver and the sensor package, wherein the processor is configured to navigate the land vehicle along a navigation land route from the landing zone to a live operator based on navigation commands received from the HSI device or the aircraft via the communication transceiver, and transmit the video feed to the HSI device. The navigation land route may be received from the perception system via the communication transceiver. The perception system may be configured to track the land vehicle from the landing zone as the land vehicle travels along the navigation land route. The communication transceiver may be configured to receive navigation commands from the aircraft and the HSI device in real time to autonomously navigate the land vehicle along the navigation land route. The processor may be configured to position the UGV based on the position of the UGV relative to the aircraft. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] These and other advantages of the present disclosure will be readily understood with reference to the following description and accompanying drawings, in which:

[0050] Figure 1 An example aircraft equipped with an autonomous air cargo / utility (AACU) system is shown approaching a landing zone;

[0051] Figure 2 Shows Figure 1 A time-based illustration of various example mission phases for an aircraft;

[0052] Figure 3a to Figure 3c Shows Figure 1 Further detailed illustrations of example mission phases for an aircraft of;

[0053] Figure 4 Shows Figure 1 An example flow chart of an example mission for an aircraft;

[0054] Figure 5a An example package delivery system is shown in which an aircraft is approaching a live operator on the ground.

[0055] Figure 5b and Figure 5c An example package delivery system is shown with an aircraft on the ground.

[0056] Figure 6 An example data flow diagram for an example package delivery system is shown.

[0057] Figure 7a to Figure 7g An example HSI device with a single touch screen interface is shown.

[0058] Figure 8 A block diagram illustrating an example package delivery system architecture; and

[0059] Fig. 9 A high-level view of the proposed system architecture consisting of functional modules is shown. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present disclosure will be described below with reference to the accompanying drawings. The components in the accompanying drawings are not necessarily drawn to scale, but the focus is on clearly illustrating the principles of the present embodiment. For example, the size of the elements may be enlarged for clarity and convenience of description. In addition, wherever possible, the same reference numerals are used throughout the accompanying drawings to refer to the same or similar elements of the embodiments. In the following description, well-known functions or configurations are not described in detail because they may obscure the present disclosure with unnecessary details. The language in the specification should not be interpreted as indicating that any unclaimed element is essential to practicing the present disclosure.

[0061] Unless otherwise specified herein, the description of the range of values ​​herein is not intended to be limiting, but refers individually to any and all values ​​falling within the range, and each individual value within this range is incorporated into the specification as if it were individually quoted herein. When accompanied by numerical values, the words "about", "approximately", etc. should be interpreted as indicating deviations that are satisfactorily operated for the intended purpose as will be understood by those of ordinary skill in the art. The range of values ​​and / or numerical values ​​is provided herein only as an example, and does not constitute a limitation on the scope of the described embodiments. The use of any example or exemplary language ("for example", "such as", etc.) provided herein is only intended to better illustrate the embodiments, and is not intended to limit the scope of the embodiments. The language in the specification should not be interpreted as indicating that any unclaimed element is essential for practicing the embodiments. In the following description, it should be understood that terms such as "first", "second", "top", "bottom", "side", "front", "back" are words with convenience and should not be interpreted as restrictive terms. For the present disclosure, the following terms and definitions should apply.

[0062] The terms "aircraft" and "aircraft" refer to machines capable of flight, including but not limited to fixed-wing aircraft, rotorcraft, unmanned aerial vehicles, variable-wing aircraft, and vertical take-off and landing (VTOL) aircraft.

[0063] The term "and / or" refers to any one or more of the items in the list connected by "and / or". As an example, "x and / or y" means any element in the three-element set {(x), (y), (x, y)}. In other words, "x and / or y" means "one or both of x and y". As another example, "x, y and / or z" means any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y and / or z" means "one or more of x, y and z".

[0064] The terms "circuit" and "circuitry" refer to physical electronic components (e.g., hardware), as well as any software and / or firmware ("code") that may configure, be executed by, and / or otherwise be associated with the hardware. As used herein, for example, a particular processor and memory may comprise a first "circuit" when executing a first set of one or more lines of code, and may comprise a second "circuit" when executing a second set of one or more lines of code. As utilized herein, a circuitry is "operable" to perform a function whenever the circuitry includes the necessary hardware and code (if either is necessary) to perform the function, regardless of whether performance of the function is disabled or not enabled (e.g., by a user-configurable setting, factory trim, etc.).

[0065] As used herein, the terms "communication" and "communicating" include transmitting data from a source to a destination, and delivering data to a communication medium, system, channel, network, device, wire, cable, fiber, circuit, and / or link for transmission to a destination. As used herein, the term "communication" means data so transmitted or delivered. As used herein, the term "communication" includes one or more of the following: a communication medium, system, channel, network, device, wire, cable, fiber, circuit, and / or link.

[0066] The terms "exemplary" and "example" mean "serving as an example, instance, or illustration." The embodiments described herein are not limiting but merely exemplary. It should be understood that the described embodiments are not necessarily to be construed as preferred or advantageous over other embodiments. Furthermore, the terms "embodiments of the invention," "embodiments," or "invention" do not require that all embodiments of the invention include the discussed features, advantages, or modes of operation.

[0067] As used herein, the terms "couple," "coupled to," and "coupled with" each mean a relationship between or among two or more devices, equipment, files, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, and / or tools that constitutes any one or more of the following: (i) a connection directly or through one or more other devices, equipment, files, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or tools; (ii) a communication relationship directly or through one or more other devices, equipment, files, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or means; and / or (iii) a functional relationship in which the operation of any one or more of the devices, equipment, files, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or tools is dependent, in whole or in part, on the operation of any one or more of the other items.

[0068] As used herein, the term "data" means any mark, signal, sign, symbol, field, symbol set, representation, and any other physical form or forms representing information, whether permanent or temporary, whether visible, audible, acoustic, electrical, magnetic, electromagnetic or otherwise manifested. The term "data" is used to represent predetermined information in one physical form, including any and all representations of corresponding information in one or more different physical forms.

[0069] As used herein, the term "database" means an organization of related data, regardless of the manner in which the data or its organization is represented. For example, the organization of related data may be in the form of one or more of a table, a chart, a grid, a data packet, a datagram, a frame, a file, an email, a message, a document, a report, a list, or data presented in any other form.

[0070] The term "exemplary" means "serving as an example, instance, or illustration." The embodiments described herein are not limiting but merely exemplary. It should be understood that the described embodiments are not necessarily to be construed as preferred or advantageous over other embodiments. Furthermore, the terms "embodiments of the invention," "embodiments," or "invention" do not require that all embodiments of the invention include the discussed features, advantages, or modes of operation.

[0071] The term "memory device" means computer hardware or circuitry that stores information for use by a processor. The memory device can be any suitable type of computer memory or any other type of electronic storage medium, such as, for example, read-only memory (ROM), random access memory (RAM), cache memory, compact disc read-only memory (CDROM), electro-optical memory, magneto-optical memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), computer-readable media, etc.

[0072] As used herein, the term "network" includes both networks and internetworks of all kinds (including the Internet) and is not limited to any particular network or internetwork.

[0073] The term "processor" means a processing device, apparatus, program, circuit, component, system and subsystem, whether it is implemented in hardware, tangibly embodied software, or both, and whether it is programmable or not. The term "processor" includes, but is not limited to: one or more computing devices, hard-wired circuits, signal modification devices and systems, devices and machines for controlling systems, central processing units, programmable devices and systems, field programmable gate arrays, application specific integrated circuits, systems on chips, systems including discrete components and / or circuits, state machines, virtual machines, data processors, processing facilities, and any combination of the foregoing. For example, a processor may be any type of general purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an application specific integrated circuit (ASIC). The processor may be connected to or integrated with a memory device.

[0074] As disclosed herein, an aircraft (e.g., unmanned, optionally manned, or manned) may be employed in conjunction with a package delivery system to provide rapid response cargo delivery to widely separated small units under harsh and unpredictable conditions that pose unacceptable risk to both ground resupply personnel and aircrew. With the use of a UGV, packages from an aircraft may be deployed to ground personnel at dispersed combat locations. For example, an aircraft may be flown to a location of ground personnel (e.g., either piloted with minimal human assistance and / or autonomously), landing in a severe, potentially contested landing zone, where the UGV facilitates the final ground transfer step of delivering the package to a field operator without exposing the ground personnel to an open landing zone (LZ) of the aircraft. The package delivery system is broadly applicable to autonomous cargo transportation capabilities across different vehicle platforms, thereby facilitating sufficient reliability to be entrusted not only with precision cargo delivery, but also with evacuating casualties from remote sites. As will be discussed, the package delivery system may be implemented via a human-system interface (HSI) device with one or more computing resources and sensor suites mounted on a vehicle platform (e.g., an aircraft and / or UGV), and software that facilitates planning, sensing, and avoidance functions.

[0075] The package delivery system may be provided via modular, open architecture software and sensor suites, including supervisory control interfaces that may be quickly and cost-effectively integrated and physically installed on many different aircraft platforms. Since proprietary systems may hinder or prevent cost-effective upgrades and modifications, an open architecture such as a Global Open Architecture Layer (GOAL) may be employed to allow portability (e.g., the software will be open source to allow new or legacy platforms to be able to "plug in" the software and sensor suite). Thus, the software, sensor packages (e.g., sensor payloads / sensor suites that may also include processors, and / or other supporting hardware), and supervisory control interfaces of the package delivery system may be moved across different vehicles, so that both legacy and new platforms may exhibit the benefits of the package delivery system. Additionally, the concept of modularity may also be applied to components within a system or vehicle to enhance compatibility across systems and simultaneously simplify field-level maintenance and repair.

[0076] Another goal of the package delivery system is threat and / or obstacle detection and avoidance, autonomous landing site selection, and descent to landing capabilities, which combines autonomous mission planning technology in an open architecture framework that seamlessly interfaces with the aircraft, unmanned aerial system (UAS) network, control infrastructure, UGV and other systems.

[0077] A package delivery system used as a form of supervisory control system for aircraft and UGVs may generally include, among other things: (1) a sensor package for accomplishing obstacle detection and sensing of physical features and their characteristics, including sensing of physical features of the UGV; (2) a mission management layer that enables optional human supervisory control through advanced autonomy; (3) a route and trajectory planner for generating aircraft and UGV navigation commands; and (4) a mission-centric global open architecture layer. Employing an open architecture and / or GOAL further allows the capabilities of the package delivery system to be ported to other platforms. Alternatively, portions of the package delivery system (e.g., its sensor package and supervisory control system) may be integrated with an aircraft platform during aircraft manufacturing. The package delivery system may also include one or more human-system interface (HSI) devices to transmit data / commands between the package delivery system and a combat outpost (COP), a remote main operating base (MOB), and / or a forward operating base (FOB). During the approach phase, the package delivery system can sense the physical features of the ground (e.g., landing zone) to identify the intended landing zone (or possible alternative landing zones) and land the aircraft at the intended landing zone, in which case terrain data is generated. The generated terrain data can be used to generate a navigation land route between the landing zone and the ground personnel for the UGV.

[0078] The sensor package and supervisory control system of the package delivery system can be integrated and / or physically installed on multiple different vehicle platforms and aircraft platforms (including VTOL platforms). Although the aircraft depicted in the accompanying drawings is a VTOL aircraft, it should be understood that the autonomous vehicle described herein may include any vehicle, device, component, element, etc. that can be usefully navigated using the principles of the system disclosed herein, including but not limited to any unmanned vehicle, manned vehicle, aircraft, ground vehicle, water vehicle, spacecraft, remote control vehicle, large vehicle, small vehicle, etc., unless otherwise clearly stated or clearly stated from the text. For example, the autonomous vehicle described herein may include a helicopter or other vehicle (e.g., a multi-rotor aircraft) using a horizontal propeller for lift. The autonomous vehicle described herein may also include or alternatively include a vehicle with forward flight capability such as a fixed-wing aircraft.

[0079] Although many aircraft platforms are possible, a suitable aircraft may be configured to operate at low density, high altitude (density altitude greater than 12,000 feet) to deliver multiple cargo drops normally within a round trip distance of, for example, between 150 and 365 nautical miles, thereby reducing the number of ground transportation delivery items. The aircraft may also be configured to carry a payload of 1,600 to 5,000 pounds internally (e.g., with some internal capacity allocated for CASEVAC). For example, the aircraft may travel at speeds between 110 and 250 knots. Within a 5 nautical mile terminal area, the aircraft may be configured to descend and land within a two to four minute window / time frame and perform an autonomous landing as close to the desired location as possible (e.g., with a goal of less than 1 meter from the center point of the computer-specified landing site) without flying over the landing area (e.g., the vehicle performs a direct approach without first passing through). In addition, the aircraft is capable of operating at night (thereby facilitating 24 / 7 operation) in a possible satellite-denied configuration (e.g., blocking communications and / or satellite-based geolocation (such as that provided by the Global Positioning System, denoted "GPS Denied"), and in all types of environments, including steep and rugged terrain, instrument meteorological conditions (IMC) and non-icing conditions, high and hot environments, and dust and sand conditions with minimal visibility. The aircraft may be configured for operation in weather conditions that exceed those in which manned flight capabilities are concerned. While the foregoing describes the performance characteristics of an example aircraft, those skilled in the art will recognize that the sensor package and supervisory control system of the package delivery system may be integrated and physically mounted on a number of different aircraft platforms as desired for particular needs.

[0080] like Figure 1As shown, the aircraft 102 can be configured to autonomously detect an unprepared landing zone (LZ) 104 and perform an aircraft landing thereto, while negotiating and navigating threats and obstacles (e.g., vegetation, terrain, buildings, etc.), which may require evasive maneuvers. For example, the landing zone (LZ) 104 can have a radius of 50 to 150 meters. There are one or more landing zones within the landing zone (LZ) 104, such as a combat outpost (COP) landing zone (TZ) 106 designated by a ground person in use (e.g., via a COP interface at COP 110), and one or more alternative landing zones (ATZ) 108. For example, the radius of each landing zone can be 1 to 50 meters, more preferably 5 to 25 meters and most preferably about 10 to 20 meters. In operation, the package delivery system evaluates the area around the COP designated landing zone (TZ) 106 designated by the operator at COP 110 (e.g., via an HSI device). If the COP designated landing zone (TZ) 106 is deemed unsuitable for landing, an alternative landing zone (ATZ) 108 may be identified. One or more alternative landing zones (ATZ) 108 may be identified by the package delivery system or operators, including field operators 116 or other ground personnel working at a combat outpost (COP) 110, a remote main operating base (MOB) 112, and / or a forward operating base (FOB) 114.

[0081] The package delivery system can be configured to detect and avoid obstacles (both static and dynamic) in a potential satellite denial environment during flight and in the descent to landing phase. In addition, the consideration of the package delivery system is the approach, descent and landing phase, so the intention to avoid obstacles is to consider navigation in the low-altitude flight envelope. Such obstacles can be static (e.g., towers, trees, buildings, etc.) or dynamic (e.g., no-fly zones, other vehicles, etc. caused by enemy activities). The package delivery system can also be configured to sense the physical features of the ground to negotiate any conditions (e.g., unsafe / unstable ground composition, swampy / muddy ground, vegetation and / or water) that may prevent the safe approach and / or landing of the aircraft 102 and can negotiate the tilted landing site. The package delivery system can also generate a complete flight path from takeoff to landing (or landing), which can be dynamically (e.g., in real time or near real time) modified by a human operator (e.g., ground personnel) in a supervisory control role in some aspects, and generate and execute a new path according to a mission planning data contingency.

[0082] The aircraft 102 is able to have GOAL-based supervisory control that may be beyond visual line of sight (BLOS) from the takeoff position through unobtrusive devices from various operators (e.g., field personnel, medical personnel, supply personnel, command center personnel, etc.) without special training and from various locations. Therefore, an advantage of the package delivery system is that it allows operators without special skills to supervise and request services from the package delivery system. The package delivery system can be configured to operate in environments where there is currently a significant risk to manned aircraft (e.g., weather, threats, terrain, etc.), and ideally, in environments where manned aircraft cannot operate safely (e.g., high winds, steep terrain, low visibility, etc.). The package delivery system can be monitored and supervised by a ground control station using mission planning capabilities from a remote operations center. Therefore, as discussed below, an intuitive HSI device can also be used.

[0083] The package delivery system can be configured to operate in meteorological or operating conditions that may limit traditional manned cargo delivery, particularly in austere terrain with low visibility due to dust, precipitation, and fog. For example, three interfaces for interacting with the package delivery system include: (1) an operations center ground control station; (2) ground personnel (e.g., field operators); and / or (3) a vehicle-mounted system for ground communications.

[0084] Figure 2 A time-based illustration of various mission phases 200 is shown. Specifically, a flight mission generally includes five phases; including initialization and configuration 202, takeoff 204, en route 206, approach 208, and landing 214. Each of these phases will now be described in more detail.

[0085] Initialization and configuration 202. After power-up, the package delivery system's software initializes the hardware, runs built-in tests, and reports results. Once initialized, the processor may wait for a mission (e.g., in the form of mission data) to be uploaded or otherwise received at the aircraft 102. For example, the main operating base 112 may transmit mission data to the aircraft 102 that may include a set of routes (e.g., in accordance with NATO Standardization Protocol 4586 (STANAG-4586)), such as: (1) takeoff route; (2) approach route; (3) flight route; (4) contingency A; and (5) contingency B. STANAG-4586 is the North Atlantic Treaty Organization (NATO) standard interface for unmanned control systems (UCS) UAV interoperability. The standard includes data links, command and control, and human / computer interfaces. After the mission data has been uploaded to the mission manager, the mission manager may send the mission data to the route and / or trajectory planner.

[0086] Takeoff 204. Aircraft 102 may be equipped with an autonomous normal flight mode, a piloted flight mode, and / or a supervised flight mode. During the autonomous normal flight mode, MOB 112 may command aircraft 102 to takeoff and follow a predefined takeoff route, for example, based on mission data previously received by aircraft 102. Thus, according to one aspect, the following sequence of events may occur to navigate aircraft 102 to assembly waypoint 218: (1) the aircraft's switch may be set to autonomous (whether a physical switch or a remote command); (2) MOB 112 sends a takeoff command to a vehicle management system (VMS); (3) VMS commands a flight control system (e.g., a system having a flight controller operably coupled to one or more flight components and one or more sensors) to execute the takeoff and forwards the command to a mission manager; (4) the mission manager transitions from the ground to takeoff mode and waits for a takeoff complete message from the flight control system; (5) the trajectory planner sends a trajectory command to the flight control system to guide it to assembly waypoint 218; and (6) the aircraft arrives at assembly waypoint 218. Alternatively, during piloted flight mode, the aircraft may be piloted (e.g., remotely) directly to the assembly waypoint 218 or initial point 220, so that a takeoff route and / or sequence may not need to be executed. In certain aspects, the autonomous normal flight mode may be overridden by the pilot. For example, if a package delivery system appears to be malfunctioning or requires immediate action, the pilot may remotely and quickly regain control of the aircraft using mechanisms that are typically appropriate for alternatively piloting and testing the aircraft.

[0087] In order to maintain the same software for both autonomous mode scenarios and piloted mode scenarios, the following sequence of events may be employed to navigate the aircraft 102 to the initial point 220: (1) switches within the aircraft 102 may be set to manual; (2) the pilot completes takeoff; (3) the mission sequencer detects that the aircraft 102 has taken off without a takeoff command and automatically transitions to “FlyLaunchRoute” mode; and (4) the pilot flies the aircraft 102 to the assembly waypoint 218, at which point the aircraft 102 may resume autonomous normal flight mode.

[0088] If the aircraft 102 autonomously reaches the assembly waypoint 218, the aircraft 102 may autonomously transition to the flight path described in the mission data and begin autonomously executing the flight path. Thus, once in autonomous mode, the aircraft 102 may autonomously execute the flight path because, when the aircraft 102 has reached the assembly waypoint 218, it has determined that it has completed the takeoff path 204 phase. The aircraft 102 may autonomously execute the flight path, request landing confirmation at the notification waypoint 222, and, based on the information about the landing path, the aircraft 102 may autonomously execute the flight path. Figure 4 Describes the steps in the process execution.

[0089] If the aircraft 102 has been manually driven to the assembly waypoint 218, the following events may occur in order to switch to autonomous flight. The main operating base 112 can first send the STANAG-4586 aircraft instruction and status message #42, i.e., the vehicle operation mode command, to the VMS, thereby placing it in a standby mode, and the aircraft 102 can execute the standby mode until further commanded. For example, the main operating base 112 can send the STANAG vehicle mode #42, thereby commanding the aircraft to be in the waypoint mode, and the pilot does not need to interact with the package delivery system. Alternatively, the aircraft pilot can place the aircraft in the autonomous mode. In another alternative, the aircraft 102 can be configured to automatically enter the autonomous mode once it arrives at a predetermined location (e.g., the assembly waypoint 218). After the aircraft 102 has landed during the landing 214, the pilot can then take off again and fly back to the initial point 220 and repeat the process.

[0090] Approach 208 and land 214. Once the package delivery system receives landing confirmation at notification waypoint 222, the aircraft 102 may execute descent procedure 210, beginning at top of descent (TOD) point 224 via ground vehicle point 212. During descent procedure 210, the aircraft 102 relies on the package delivery system to identify and land at the intended landing zone (or an alternate landing zone), while determining whether the package delivery system has received a go-around command.

[0091] For example, now go to Figure 3a to Figure 3c When the aircraft 102 is a first predetermined distance from the landing zone (e.g., 1 km) and / or a first predetermined time from the landing zone (e.g., 55 seconds), the aircraft 102 may use a sensor package to begin scanning the ground (e.g., landing zone (LZ) 104) to detect, for example, terrain features and large obstacles. When the aircraft 102 is a second predetermined distance from the landing zone (e.g., 400 m) and / or a second predetermined time from the landing zone (e.g., 30 seconds), the aircraft 102 determines whether the designated landing zone 304 is feasible. If the package delivery system determines that the designated landing zone 304 is not feasible, the package delivery system may identify one or more alternative landing zones 306, which may be transmitted to an operator (e.g., an HSI device) for consideration and / or approval. Throughout this landing process, the aircraft 102 may receive or otherwise detect a go-around command (e.g., from an HSI device).

[0092] Between the second predetermined distance / time and the third predetermined distance / time (e.g., 60 meters, 10 seconds) from landing 214, the package delivery system can identify an alternative landing zone 306 and notify the operator (or pilot) of the alternative landing zone 306 via the HSI device. In response, the operator can (1) approve the alternative landing zone 306, (2) designate a second alternative landing zone 308, (3) make a go-around of the aircraft 102, (4) abort the mission, or (5) do nothing. If the operator fails to act (e.g., provide instructions) within a predetermined time period (e.g., between 5 seconds and 60 seconds, more preferably between 10 seconds and 30 seconds) from landing 214, the mission can be automatically aborted. If the operator chooses to abort the mission (either explicitly or without taking action), the aircraft 102 may be directed to a hold-off point 228 or an egress point 310 which will ultimately direct the aircraft 102 to navigate to a predetermined point, such as the initial point 220 .

[0093] Figure 4 An example flow chart of various mission phases 400 from the mission until landing is shown. At step 402, the software initializes and once completed, will wait to upload or otherwise receive mission data. After the mission data has been transmitted to the mission manager, the mission manager can send the mission data to the route and / or trajectory planner. At step 404, the aircraft 102 can be piloted or ordered by the main operating base 112 to take off and follow the takeoff route. At step 406, the aircraft 102 is en route 206, and in the absence of further instructions, it will automatically execute the flight route, request landing confirmation at the notification waypoint 222, etc. If the landing is approved, the aircraft 102 proceeds to approach and descend at step 408, relying on the package delivery system to identify and land in the landing zone in the case where the aircraft 102 performs a descent, as discussed above with respect to approach 208.

[0094] If the package delivery system determines at 408 that the designated landing zone 304 is not feasible, the package delivery system will identify one or more alternative landing zones 306 at step 414 and await approval. At step 414, an alternative landing zone may be provided, thereby directing the aircraft 102 to land at the alternative landing zone at step 410. Alternatively, the aircraft 102 may make a go-around, causing the aircraft 102 to perform a go-around maneuver at 416. In yet another alternative, the mission may be aborted via a command from a flight controller or a timeout (step 420), causing the aircraft 102 to return to a predetermined location, such as a base. At any of steps 406, 408, and 410, the aircraft 102 may receive a go-around command, thereby directing the aircraft 102 to the hold-off point 228 (step 416). The aircraft 102 may then remain (step 418) until the mission may be aborted (step 420) or a retry may be attempted. If the aircraft 102 determines at 408 that the designated landing zone 304 is feasible, the aircraft 102 can proceed to approach at step 410, where the aircraft 102 can perform a landing process at 410. Once the aircraft 102 is on the ground, the aircraft 102 is considered to have landed (step 412), and the UGV can then be used to perform ground operations. Once the ground operations are completed, the aircraft 102 can then be initialized again (step 402) to perform a new mission or return to base. Therefore, the skilled person will understand that additional steps can be performed between steps 412 and 402.

[0095] Figure 5aThe aircraft 102 is shown approaching a live operator 116 on the ground. During the approach 208, the aircraft 102 may scan the terrain of the scan area 502 to generate terrain data, which identifies, among other things, landing zones, obstacles 506, terrain features, etc. For example, the terrain data may include a map with landmarks and terrain details. In addition to identifying the landing zone 504, the terrain data may also be used to define one or more navigation land routes 118 for the UGV between the selected landing zone 504 and the live operator 116. For example, the aircraft 102 may scan the area 502 via a sensing and perception payload 508 during the approach 208 phase. The sensing and perception payload 508 may be mounted to the aircraft 102 (e.g., toward the front of the fuselage) via a gimbal system to enable the sensing and perception payload 508 to be oriented toward the scan area 502. As shown, the scan area 502 may be a geographic area surrounding and including the landing zone 504 selected for landing. For example, the field operator 116 may be located near a landing zone (TZ) 106, an alternative landing zone (ATZ) 108, etc. designated by the COP. The scan area 502 may include the landing zone 504 or a portion thereof (i.e., overlap with it). It is worth noting that the scan area 502 may extend beyond the scope of the landing zone 504, because although certain geographic areas may not be suitable for the aircraft 102, the geographic areas may be suitable for UGV navigation and therefore may be imaged. In addition, depending on the field of view provided by the sensing and perception payload 508 (which may be affected by the approach angle and the ground terrain), the scan area 502 may or may not be large enough to include both the landing zone 504 and the field operator 116. However, for illustrative purposes, the field of view is shown as including the field operator 116.

[0096] The purpose of the package delivery system is to deliver the package to the field operator 116 without exposing the field operator 116 or other ground personnel to the substantially unobstructed area around the landing area 504. Thus, upon landing the aircraft 102 at the landing area 504, one or more unmanned ground vehicles (UGVs) 512 may exit the aircraft 102 and travel to the location of the field operator 116, deliver the package, and return to the aircraft 102. The UGVs 512 may exit and enter the aircraft 102 using one of a variety of methods. To this end, the aircraft 102 may include a powered ramp or other infrastructure that enables the UGVs 512 to re-enter the aircraft 102. For example, the aircraft 102 may include a ramp 520 to enable the UGVs 512 to travel in and out of the cargo hold of the UGVs 512. Alternatively, the aircraft 102 can be equipped with a lifting mechanism that can be mounted on the aircraft 102 (e.g., adjacent to a door leading to the cargo hold) and powered using one or more hydraulic or electric actuators to raise and lower the UGV 512 to enter and exit the cargo hold. For example, an example lifting mechanism includes a crane, a hoist, or a lifting door, which may include a platform for supporting a UGV 512 that can be raised or lowered. In another example, the UGV 512 (e.g., a legged robot) can be configured to jump out of or into the cargo hold, thereby avoiding the need for a ramp 520 or a lifting mechanism. In operation, the UGV 512 can be fixed in the aircraft 102 during flight, but is configured to leave and enter quickly. The UGV 512 can be fixed in the aircraft 102 using, for example, blocking and supporting, tying, lashing, or fasteners. For example, an electromagnet can be used to selectively fix the UGV 512 and release it from the aircraft 102.

[0097] As shown, UGV 512 can be configured to reliably load, carry and unload packages. For this reason, UGV 512 can include chassis 512a and container 512b, and the container 512b can be modularly coupled to chassis 512a to accommodate one or more packages. Alternatively, one or more packages can be directly fixed to chassis 512a. Chassis 512a generally includes a powertrain for driving UGV 512. For example, the powertrain can include an engine (or one or more motors), a battery, a transmission, a drive shaft, a differential, and a final drive mechanism (drive wheel, continuous track, legged robot, etc.). For example, UGV 512 can be a vehicle with wheels, continuous tracks (e.g., tank tracks) and / or legs (e.g., legged robot). Packages can be pre-loaded before aircraft 102 arrives at landing area 504, or packages can be loaded according to the request of field operator 116. In this case, UGV 512 can also be configured with a mechanical arm to manipulate / unload packages (or container 512b) from aircraft 102.

[0098] Via chassis 512a, UGV 512 should be configured to travel to the location of the field operator 116 while traversing a wide range of terrain types (e.g., grass, concrete, dirt, sand, gravel, etc.). UGV 512 can be graphically configured with a sensor payload (e.g., optical payload 512c) to evaluate situations that will result in a low probability of mission success (e.g., obstacles, rivers, ditches, etc.). If UGV 512 detects a low probability of success situation, UGV 512 can send an alert to aircraft 102 or HSI device 514. Once the alert is transmitted, UGV 512 can wait for further instructions and, in the absence of further instructions within a predetermined time period, return to aircraft 102 or a predetermined location. During operation, when UGV 512 is maneuvered along navigation land route 118 to the location of field operator 116, UGV 512 must operate at a sufficient speed to minimize mission duration and risk. Furthermore, the behavior of the UGV 512 should be predicted and controlled by the field operator 116 via the HSI device 514 .

[0099] The sensing and perception payload 508 can scan the terrain of the scanning area 502 with high precision (e.g., accuracy within 0.2m) to generate terrain data. It may be a challenge to rely on UGV 512 to quickly perform its positioning, drawing and path planning on new terrain. In order to solve this challenge, UGV 512 can communicate with aircraft 102 and field operator 116 (e.g., via HSI device 514), and any one of the aircraft 102 and field operator 116 can dynamically enhance the navigation of UGV512. In fact, once on the ground, the field of vision beyond its adjacent surrounding environment of small UGV 512 is limited. Therefore, in order to perform ground delivery to field operator 116, a processor (e.g., positioned on aircraft 102, HSI device 514 or UGV 512) can generate a terrain map according to the terrain data generated, and the terrain map will be used to plan one or more navigation land routes 118 between aircraft 102 and field operator 116. In certain aspects, the processor may generate a plurality of navigational land routes 118 to be presented to the on-site operator 116 for approval. For example, some of the navigational land routes 118 may be faster (shorter distance) but subject to a greater risk of obstacle collision. Therefore, alternative navigational land routes may be provided to the on-site operator 116 via the HSI device 514 that are slower (longer distance) but have a lower risk of obstacle collision.

[0100] refer to Figure 5b and Figure 5cOnce on the ground, the aircraft 102 can enhance the UGV 512's ability to locate itself on a map (e.g., its point along a given navigation land route 118) to facilitate real-time navigation. To this end, an optical payload 510 can be positioned on the aircraft 102 to observe the UGV 512 as it traverses the navigation land route 118 to the on-site operator 116. The optical payload 510 can also be used to warn the UGV 512 if a collision with an obstacle 506 is expected. The optical payload 510 can be positioned near the top of the aircraft 102 (e.g., above the cargo door, but below the rotor) to provide guidance within its aircraft field of view 516. For example, the optical payload 510 can include light detection and ranging (LIDAR), stereo vision, etc. Additionally, the UGV 512 can use its position relative to the aircraft 102 to locate itself on a map. In effect, the aircraft 102 , which may remain stationary, may provide a reference point for the moving UGV 512 during the positioning process.

[0101] The field operator 116 may also have an operator view 518 that may be used to assist the UGV 512 along the navigation land route 118 of the UGV 512. If desired, the field operator 116 may observe and modify the navigation land route 118 of the UGV 512 via the HSI device 514, for example to avoid obstacles 506 or to take a different navigation land route 118. For example, the field operator 116 may take direct remote control of the UGV 512 and view video dynamically captured by the UGV 512 or the aircraft 102. Thus, the UGV 512 may also include an optical payload 512c that may be controlled (e.g., adjusted, scaled, rotated, pivoted, panned, etc.) via the aircraft 102 or the HSI device 514. To this end, the optical payload 512c may be coupled to the UGV 712 via a gimbal system. Thus, the UGV 512 may be “dumb” in that it merely follows navigation commands from the aircraft 102 or HSI device 514 while being supplied with a data feed from its sensor payload (eg, optical payload 512 c ).

[0102] Depending on the distance between the landing zone 504 and the on-site operator 116, the aircraft field of view 516 and / or the operator field of view 518 may or may not include the entire navigation land route 118. In the event that one or both of the aircraft field of view 516 and the operator field of view 518 include the entire navigation land route 118, the UGV 512 can be monitored and controlled by a single party for the entire route. However, in the event that a single system or operator cannot observe the entire navigation land route 118, the aircraft 102 and the on-site operator 116 can coordinate to enhance control of the UGV 512. For example, as shown, the aircraft field of view 516 can cover approximately 70% of the navigation land route 118, while the operator field of view 518 can cover approximately 45% of the navigation land route 118, thereby defining an overlapping field of view 522 of 15% of the navigation land route 118. Thus, during overlapping field of view 522, the on-site operator 116 may use, for example, a video feed dynamically captured by the UGV 512 or the aircraft 102 (e.g., live video) to assume remote control of the UGV 512. In certain aspects, the on-site operator 116 may choose to use a video feed from the UGV 512 or the aircraft 102 (e.g., from the optical payload 510 or the optical payload 512c) that may be dynamically displayed on the HSI device 514 to navigate the UGV 512 even if the UGV 512 is outside the operator field of view 518. To resolve conflicting navigation commands from the aircraft 102 and the HSI device 514, the UGV 512 may defer to (i.e., prioritize) the navigation commands from the HSI device 514, thereby giving the on-site operator 116 absolute control.

[0103] Although the present disclosure generally describes an embodiment of using an aircraft 102 to deliver a UGV 512 to a location (e.g., a landing zone 504), it will be understood by those skilled in the art in view of the present disclosure that other devices (e.g., vehicles) can be used to deliver the UGV 512 to a specified location (e.g., a landing zone 504 or its equivalent). In some aspects, the UGV 512 can be delivered to a location via a land vehicle. For example, a package delivery system can be deployed in an urban area to facilitate commercial package delivery. In such an example, a larger land vehicle (e.g., a van / truck) that can be autonomous can cooperate with a small UGV (e.g., UGV 512) to deliver a package. In operation, a larger land vehicle can stop and park on the side of the road, whereby a small UGV leaves a larger land vehicle at a roadside location to transport the package from the larger land vehicle to the final destination (e.g., doorstep). As with the UAV example, a larger land vehicle can still play the role of guiding a smaller UGV on and / or around complex terrain via an optical payload 510. In certain aspects, larger land vehicles may also navigate an area (eg, a neighborhood) while generating a point cloud in real-time (or near real-time).

[0104] Figure 6An example data flow diagram of an example package delivery system 600 is shown, which has an aircraft 102, a UGV 512, and an HSI device 514 (e.g., a portable electronic device such as a smart phone, tablet computer, and laptop computer) or other controller (e.g., a base station). Each of the aircraft 102, the UGV 512, and the HSI device 514 may include a wireless transceiver 602 that is coupled with an antenna and a processor (or other circuit system) to transmit data between the various systems of the aircraft 102, the UGV 512, and the HSI device 514. The aircraft 102, the UGV 512, and the HSI device 514 can use radio frequency to transmit data (processed data, unprocessed data, etc.) to each other directly (e.g., point-to-point, point-to-multipoint, etc.) or indirectly (e.g., through the network 604, using the UGV 512 as a relay, or via a mesh network). In certain aspects, the wireless transceiver 602 may be configured to communicate using one or more wireless standards, such as Bluetooth (e.g., short wavelength, ultra-high frequency (UHF) radio waves in the 2.4 GHz to 2.485 GHz Industrial, Scientific, and Medical (ISM) band), Near Field Communication (NFC), Wi-Fi (e.g., Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard), Military Standard 188 (MIL-STD-188), Standard Interface for Multi-Platform Link Evaluation (SIMPLE), etc. The HSI device 514 may facilitate monitoring and / or control of the aircraft 102 and the UGV 512.

[0105] As shown, each of the aircraft 102 and the UGV 512 can be configured to transmit navigation commands to the UGV 512. The sensor data (e.g., image / video feed) collected by the UGV 512 can be transmitted to each of the aircraft 102 and the HSI device 514. The aircraft 102 and the HSI device 514 can be configured to exchange sensor data, whether the sensor data is collected by the aircraft 102 or the UGV 512. For example, the sensor data collected by the aircraft 102 can be transmitted to the HSI device 514, such as video feed, terrain information, details about the UGV 512 and / or its payload, etc. In some cases, the HSI device 514 can generate and output sensor data, such as positioning information (e.g., via a GPS transceiver). In addition, the aircraft 102 and the UGV 512 can be configured to exchange system commands, which can be used to adjust one or more settings of any device. For example, the field operator 116 can instruct the aircraft 102 to take or stop the control of the UGV 512.

[0106] Figure 7a to Figure 7gAn example HSI device 514 is shown having a single touch screen interface and in some aspects a voice recognition interface to enable voice communications (e.g., commanding the UGV 512 or communicating with another operator). The HSI device 514 is shown with a toolbar area 716a and a main display area 716b. The HSI device 514 serves as the primary communication channel between the field operator 116 and the UGV 512 to enable the field operator 116 to command and receive feedback or instructions from the UGV 512 and / or the aircraft 102. The HSI device 514 can display the current status (e.g., current settings) of the UGV 512 and / or the aircraft 102 via a display device (e.g., a liquid crystal display (LCD)). The GUI display of the HSI device 514 can also be compatible with night vision goggles so that it is visible regardless of the field operator's 116 glasses.

[0107] refer to Figure 7a , the main display area 716b is shown with a home screen 718. The home screen 718 may include icons for selection for controlling or monitoring one or more UGVs 512 (e.g., via UGV selection icons 714a, 714b, 714c, 714d) and / or aircraft 102 (e.g., via aircraft selection icons 716ba, 716bb, 716bc, 716bd). The toolbar area 716a may include a plurality of selectable icons; such as a remote controller icon 702, a navigation map icon 704, a UGV video feed icon 706, an aircraft video feed icon 708, and a settings icon 710. In order to prohibit unwanted access to the HSI device 514, the HSI device 514 may require user authentication using, for example, a login screen 712 for entering a username and / or password. In other aspects, biometrics may be used to authenticate the user, such as a pulse / heart rate monitor and / or a scanner for scanning a retina and / or fingerprint.

[0108] For example, selecting the aircraft selection icon 716a can enable the field operator 116 to view and control various aspects of the first aircraft 102 (i.e., Aircraft_1). Similarly, for example, selecting the UGV selection icon 714a can enable the field operator 116 to view and control various aspects of the first UGV 512 (i.e., UGV_1). To this end, the main display area 716b can be divided into quadrants, each quadrant enabling different functions or operations of the first UGV 512. For example, the main display area 716b can display a remote controller screen 720, a UGV route screen 722, a UGV camera screen 724, and an aircraft camera screen 726. In order to enlarge any one of the screens, the field operator 116 can select an associated icon from the toolbar area 716a, or select a desired quadrant (e.g., double-click) to enlarge the view. For example, in order to enlarge the remote controller screen 720, the field operator 116 can select the remote controller icon 702 or select the quadrant in which the remote controller screen 720 is displayed.

[0109] Figure 7c An enlarged view of the remote controller screen 720 is shown. As shown, the remote controller screen 720 may include navigation controller icons, such as a speed controller icon 730 and a direction controller icon 732. The speed controller icons 730 may include: a forward speed control icon 730a for driving the powertrain (e.g., wheels, including those driving a continuous track) in a first direction; a stop control icon 730 for stopping (e.g., braking) the powertrain; and a reverse speed control icon 730c for driving the powertrain in a second direction (opposite to the first direction). For example, to control the speed in the first direction or the second direction, the user may engage the speed controller icon 730 at a contact point (e.g., point A), where the speed is a function of the distance between the contact point and the stop control icon 730. For example, point A may indicate that the UGV 512 is traveling in a forward direction at an intermediate speed. The direction controller icon 732 may be used to turn the UGV 512, thereby adjusting its forward direction. For example, the user may engage the speed controller icon 730 at a contact point (e.g., point B), where the heading is a function of the angular position of the contact point on the direction controller icon 732 (e.g., between 0 and 360 degrees). For example, point B may indicate that the UGV 512 is moving at an off-center (in Figure 7c Point A and point B together will instruct the UGV 512 to drive its wheels in a first direction and turn right.

[0110] Although the speed controller icon 730 and the direction controller icon 732 are shown as separate elements, they can be combined into a single controller. For example, the direction controller icon 732 can be adapted to control both speed and direction using a single point of contact from the operator. For example, the distance between the single point of contact (e.g., point B) and the center of the direction controller icon 732 can be used to determine speed, with larger distances equating to higher speeds, and the angular position (e.g., between 0 and 360 degrees) can be used to determine the heading of the UGV 512.

[0111] Figure 7d An enlarged view of the UGV route screen 722 is shown. As shown, the UGV route screen 722 may include a map of the scan area 502 with a selected navigation land route 118 connecting an origin 736 (e.g., a selected landing zone 504) with a destination 740 (e.g., a field operator 116). It will be appreciated that the selected navigation land route 118 is designed to avoid various obstacles 506. The UGV icon 738 may be used to indicate the progress of the UGV 512 along the selected navigation land route 118. Selecting (e.g., tapping, clicking, etc.) the UGV icon 738 may cause a dialog window 742 to be displayed, which provides various operating conditions of the UGV 512 (e.g., speed, heading, coordinates, etc.). The UGV route screen 722 may be saved, exported, rotated, or translated using the control window 728. For example, a map of the area may be saved or exported as a static image or a data set (or database) representing the navigation land route 118 and the terrain.

[0112] Figure 7e An enlarged view of the UGV camera screen 724 is shown, and Figure 7f An enlarged view of the aircraft camera screen 726 is shown. As shown, the UGV camera screen 724 can include a live video feed from the UGV 512 (e.g., from the optical payload 512c), and / or to provide a second viewpoint, a live video feed from the aircraft 102 (e.g., from the optical payload 510). Regardless of the source, the live video feed can be overlaid with information and / or various controls, such as the control window 728, the speed controller icon 730, and / or the direction controller icon 732. Thus, as shown in FIG. Figure 7g As shown, for example, HSI device 514 may simultaneously display a zoomed-in live video feed and various controls that may be used by field operator 116 to navigate UGV 512 to avoid obstacle 506 .

[0113] As described above, the aircraft 102 may be configured with a package delivery system to map the terrain as it approaches the landing zone (LZ) 104, which may be used to both assist in selecting the selected landing zone 504 and in mapping the route of the UGV 512. Once on the ground, the aircraft 102 may provide additional mapping using its optical payload 510, which may be mounted as high as possible on the aircraft 102. For example, the cargo bay ceiling of a Bell UH-1 "Iroquois" helicopter is more than 80 inches off the ground, which would provide a higher viewpoint than the UGV 512. These sensors, along with any supervisory commands from ground personnel (field operators 116), may help guide the UGV 512.

[0114] Figure 8 The architecture 800 of a package delivery system is shown, which includes sensors on board a vehicle, a mission manager, communications, route and trajectory planning, and flight controls / sensors. More specifically, as shown, the architecture 800 of the package delivery system may include a flight controller 802, a motion planner 804, a dynamic route planner 806, and a mission manager 808. In use, data may be transmitted between two or more of the following: the flight controller 802, the motion planner 804, the dynamic route planner 806, and the mission manager 808, either directly or via a vehicle dynamics lookup module 810. For example, the mission manager 808 may: (1) transmit mission and constraint data to the dynamic route planner 806; and / or (2) transmit route and mission data to the motion planner 804. The dynamic route planner 806 may similarly provide route and mission data to the motion planner 804 based on data (e.g., constraints) received from the vehicle dynamics lookup module 810. The motion planner 804 may provide trajectory data to the flight controller based at least in part on data received from the sensing and perception system 814 and / or the vehicle dynamics lookup module 810. The vehicle dynamics lookup module 810 may also be configured to receive data from the mission manager 808 (e.g., mission urgency, fuel / cargo weight, etc.).

[0115] The flight controller 802 can provide feedback (e.g., vehicle state data, wind estimation data, etc.) to the vehicle dynamics lookup module 810 via the vehicle dynamics module 816, which can generate vehicle dynamics data from one or more sensors (e.g., 816b, 816c, 816d). Such vehicle dynamics data can also be used to control one or more flight controllers or flight controller systems (e.g., flight component 816a). Therefore, the flight controller coupled to the flight component 816a and one or more flight controllers / systems and sensors can be used as a flight control system for the aircraft.

[0116] The motion planner 804 may be configured to receive data (e.g., obstacle and status data) from a sensing and perception module 814, which may receive data measurements from various sensors located within one or both of the sensing and perception payload 508 and the optical payload 510. For example, the various sensors may include a light detection and ranging (LIDAR) 824, a radio detection and ranging (RADAR) 826, an electro-optical infrared (EO / IR) imager 828, a stereo vision camera system 838, a radio altimeter (RADALT) 816b, an air data sensor 816c, and / or a GPS / inertial navigation system (INS) 816d. That is, some of the various sensors may be located within the sensing and perception payload 508 to identify the selected landing zone 504, while some other sensors (or redundant sensors) may be provided in the optical payload 510 to monitor the UGV 512. For example, one or more of the sensing and perception payload 508 and the optical payload 510 may each be provided with a LIDAR 824 and a stereo vision camera system 838 .

[0117] The mission manager 808 may be communicatively coupled to one or more remotely located systems or operators via a communication system or module 812. For example, the mission manager 808 may wirelessly communicate with the COP 110, the MOB 112, the FOB 114, the HSI device 514, and / or another system 822 (such as a doctor, ground crew, other vehicles, etc.). The mission manager 808 may be configured to send data reflecting the mission payload 830, countermeasures 832, and / or other data to a ground crew 834 (e.g., an HSI device 514, a cargo system, an auxiliary system, etc.) or another system 836 (e.g., a medical system). Similarly, the mission manager 808 may be configured to receive data indicating a ground threat 820 or a casualty 818 from a sensor. Any processor and / or other hardware may be powered by a power source, which may be an alternating current or a direct current (e.g., conventional line current, battery power, solar power, wind power, etc.).

[0118] To facilitate the various functions of the package delivery system, the package delivery system may employ one or more processors that are operably coupled to: (1) a memory device, (2) one or more sensors, and / or (3) other systems disclosed herein or known in the art. For example, to process and manipulate data, the processor may be configured to run software that may be stored in ROM, RAM, or one or more other computer-readable storage media. Similarly, data collected or created by the package delivery system may be stored in RAM, ROM, or another suitable storage medium for long-term retention. The package delivery system may receive and transmit data related to its position, speed, UGV, etc. via the processor. The flight controller 802 may also include a user interface (e.g., via the HSI device 514) that allows an operator (e.g., a human or computer-implemented operator, either of which may be local or remote) to enter commands, control operations, and / or adjust settings of the package delivery system. The user interface may be remotely coupled and / or may include, for example, a computer, keyboard, mouse, touch screen, joystick, etc.

[0119] Task manager 808. The task manager may include: (1) a task sequence generator; (2) a STANAG to Robot Operating System (ROS) bridge; (3) a ROS to STANAG bridge; (4) a task planning service; (5) a route monitor; and (6) a route feeder.

[0120] (1) Mission sequence generator. The mission sequence generator manages the overall behavior of the package delivery system, including the aircraft 102 and, if necessary, the UGV 512. The mission sequence generator may employ multiple state machines, one state machine for managing the overall behavior of the aircraft, and other state machines for managing commands issued by the main operating base 112 and combat outposts 110, HSI devices 514, etc. The mission sequence generator may track waypoints located by the aircraft 102, initiate communications with the combat outpost 110 for landing negotiations, and, if necessary, instruct the route feeder via mission commands to send a new route to the trajectory planner. The mission sequence generator may also communicate to the route feeder which route or stage of a route the aircraft 102 is executing (e.g., initialization and configuration 202, takeoff 204, en route 206, approach 208, etc.).

[0121] (2) STANAG to ROS Bridge. The STANAG to ROS Bridge acts as a bridge between the STANAG 4586 protocol used by the main operating base 112 and the aircraft's vehicle management system, the dynamic route planner (506) (which may or may not communicate using the STANAG-4586 protocol), and the ROS protocol used by the mission manager, trajectory planner, perception system 814, and other components of the package delivery system. The STANAG to ROS Bridge allows integration with the commonly used STANAG 4586 messaging standard, thereby improving the portability of the package delivery system. The STANAG to ROS Bridge can perform the following services: (1) Aggregate a series of STANAG 4586 messages into a single mission planning ROS message and publish the message via the ROS protocol. The mission planning data may include a set of routes (nominally: takeoff, flight, approach, contingency A, and contingency B), each route may have its own set of waypoints. The mission plan data for each mission plan may have a source identifier that indicates whether the plan is from the main operating base 112 or the dynamic route planner; (2) convert other STANAG 4586 messages into, for example, equivalent ROS messages and publish these messages via the ROS protocol; and (3) subscribe to selected ROS messages and convert them back to STANAG 4586 messages for broadcast on the STANAG 4586 multicast network, thereby being listened to by the vehicle specific module (VSM) within the VMS, the main operating base 112, possibly the dynamic route planner, and possibly the combat outpost 110. Although the ROS protocol is disclosed and described herein, it is only one of many ways in which messages can be published within the package delivery system. Therefore, one of ordinary skill in the art will recognize that other protocols such as DDS are possible. Therefore, the package delivery system should not be limited to use with the ROS protocol.

[0122] (3) ROS to STANAG Bridge. The ROS to STANAG bridge essentially performs the reverse function of the STANAG to ROS bridge; it converts ROS data into STANAG messages and supports the output interface between the internal components of the package delivery system and the main combat position 112.

[0123] (4) Mission Planning Service. The Mission Planning Service subscribes to mission planning ROS messages published by STANAG to ROS nodes, and stores (in non-volatile memory) and organizes the received mission planning data for use by elements of the mission manager and possibly other ROS nodes. Whenever the Mission Planning Service receives and processes new mission planning data, it may issue a Configuration ROS message that specifies that the configuration is complete and indicates the key waypoint identifiers within the mission planning data, currently the Assembly, Notification, and Approach waypoint identifiers. The Mission Planning Service will ensure that the waypoint identifiers are maintained in ascending numerical order corresponding to the order of visits indicated by the route (e.g., if the route indicates that the vehicle should visit Rome, Paris, and London, then Rome will have a numerically lower waypoint than Paris, which in turn will have a numerically lower waypoint identifier than London, in this order). Whenever the Mission Planning Service receives a request for route data from another ROS node, it may use the previous route and waypoint identifiers to obtain the corresponding waypoint series, and send a route data reply with the requested number of waypoints back to the requester. One request may request the current waypoint the vehicle is heading towards and all subsequent waypoints on the route.

[0124] (5) Route Monitor. The route monitor subscribes to two sensor messages, one from the inertial state reported by the vehicle via STANAG 4586 (converted into ROS messages via the STANAG to ROS node), and a trajectory state message provided by the trajectory planner using data from the perception system. The route monitor will select or combine the information from these two sources and report the progress along the route as a single floating point value representing the decimal distance from the last waypoint passed (e.g., 2.3 represents approximately 30% between waypoint 2 and the next waypoint along the route). This can be in the same format as reported by the trajectory planner, although this information may not necessarily come only from the trajectory planner. This can provide the option of generating this signal internally from a mix of inputs. The route monitor also measures route deviation and sends a Boolean flag embedded in the route state along with the route progress to the task sequence generator.

[0125] (6) Route Feeder. Based on the current route provided by the task sequence generator, the current vehicle position provided by the route monitor, and the route waypoint set provided by the task planning service, the route feeder block can create a task command to the trajectory planner. Upon receiving the route selection from the task sequence generator and the vehicle position from the route monitor, the route feeder requests a set of waypoints from the task planning service (currently from the current vehicle position to the end of the route). The task planning service replies with the requested set of waypoints. The route feeder can then form and publish waypoints in the form of task commands. The trajectory planner can receive these waypoints and adjust its trajectory accordingly.

[0126] Dynamic route planner 806. Route planning can be facilitated via a software algorithm called "4D-D*", which uses data from maps and a combination of information incrementally discovered from onboard sensors to explicitly solve the problem of determining routes for multiple vehicles. The software is different in two specific ways. First, it explicitly considers a 4D space consisting of x, y, z and time, allowing for explicit consideration of moving obstacles, such as weather or other vehicles that must be avoided. Second, it uses concepts from a well-known path planning algorithm called Field-D* to optimally update the trajectory when new information becomes available. This has the following advantages: a completely new trajectory on a large map can be calculated in a few milliseconds by modifying the old trajectory when new information is received instead of having to completely recalculate it. Depending on the mission urgency or the priority given to saving fuel, different paths may be preferred. The algorithm will select a path that optimizes a large set of criteria, some of which may change during the mission. The algorithm complexity (computation time) can implement an "any time" version that will continuously improve the solution within a given time, but produce an answer (in some cases suboptimal) at any time when an answer can be requested.

[0127] Motion planner 804. Given an initial path from the mission planner, the motion planner calculates a trajectory based on multiple objectives, including proximity to obstacles, desired landing vector (based on wind direction), vehicle dynamics, and positioning accuracy. The trajectory generation scheme also continuously improves and optimizes the trajectory based on specified criteria. The trajectory generation scheme quickly generates and iterates new command paths that avoid any detected obstacles and the vehicle can follow the path. An advantage of this approach can be that the fidelity of the parallel planning algorithm is proportional to the available computing power. If resources are limited, only a small portion of the search space can be explored while still obtaining a "good enough" solution. The planning system is robust and adapts to the changing dynamics of the cargo aircraft 102, as they will change depending on weight and wind conditions. Commands can be verified based on propagating uncertainty and execution uncertainty of different objectives in a stochastic optimal control framework. This increases confidence in the execution of the plan and allows high-performance control of the vehicle. Special maneuvers related to takeoff from and landing on sloping terrain can also be employed. The algorithm can be configured to account for unexpected discontinuities in vehicle dynamics, such as contact with undetected ground features, by adding hidden states. Such mission-level and motion planning algorithms can be applied to a wide range of aircraft. They combine multiple objectives and constraints in real time, incorporate and adapt dynamics into motion planning with higher fidelity control, and propagate uncertainty for robust control during banked landings.

[0128] Flight controller 802. Flight controller 802 may be operably coupled to motion planner 804, vehicle dynamics lookup module 810, and vehicle dynamics module 816. In operation, flight controller 802 generates flight control signal data based, for example, at least in part, on data received from motion planner 804 and one or more sensors (e.g., RADALT 816b, air data sensor 816c, and / or GPS / INS device 816d). The flight control signal data generated by flight controller 802 may be transmitted to flight components 816a or used to control flight components 816a. For example, example flight components 816a include rotorcraft flight controls (e.g., common, cyclic pedals, throttles, auxiliary components, etc.) and fixed-wing aircraft controls (e.g., ailerons, rudders, trim tabs, elevators, throttles, etc.).

[0129] In one aspect, for example, the flight controller 802 may also employ a user interface, and a processor operably coupled to a memory / data storage device and one or more sensors. For example, in order to process and manipulate data, the processor may be equipped to run software that may be stored in ROM, RAM, or one or more other computer-readable storage media. Similarly, data collected or created by the flight controller 802 may be stored in RAM, ROM, or another suitable storage medium for long-term retention. The flight controller 802 may receive and transmit data related to position, speed, attitude, etc. via the processor. The flight controller 802 may also include a remotely located user interface that allows an operator (e.g., a human or computer-implemented operator, either of which may be local or remote) to input commands and / or control the operation of the flight controller 802. The remotely located user interface may be the same as a remotely located user interface for controlling a package delivery system. The user interface may be coupled to the flight controller 802 and may include, for example, a computer, keyboard, mouse, touch screen, joystick, etc. To monitor various flight conditions, flight controller 802 may also employ one or more sensors (eg, weather radar, engine sensors, vertical / directional gyroscopes, accelerometers, thermometers, altimeters, etc.).

[0130] Sensing and perception 814. The goal of the above is to enable the aircraft 102 to operate safely during takeoff, cruise, and descent by making the aircraft 102 aware of its environment. Thus, a goal of the package delivery system may be to ensure a safe final approach and a stable landing, as well as to be aware of the environment during takeoff and / or cruise. For example, AACU perception may: (1) support cruise speeds up to 250 knots; (2) allow a five-mile approach to a landing zone in 2 to 4 minutes; and (3) be configured to operate in a visually degraded and GPS-denied environment. To achieve these goals, the sensor package and perception algorithms are preferably tightly coupled. Notable design features of the sensor package may be: range (the greater the range of the sensor, the higher the approach speed that can be supported), range resolution (the finer the resolution, the easier it is to detect the roughness and slope of the terrain), and field of view (FOR) (a large FOR may be required to be able to image the landing zone at all times when approaching from a far side to a viewpoint directly above the landing site). Operation in a visually degraded environment will also require sensors that penetrate obstacles, and GPS-denied operation will require the use of available sensors to implement alternative navigation methods. Finally, for autonomous aircraft to be accepted by ground forces operating from forward bases, if the aircraft 102 is present at the landing site, the aircraft 102 itself can unambiguously recognize common human gestures used by landing support specialists (such as "go-around"). Analysis of various military aircraft missions indicates that aggressive low-altitude flight profiles (such as those typically performed in adverse conditions) may be the limiting case for determining the minimum sensor range for collision-free flight. This assumes that the final approach begins at 110 knots (the current example threshold) at a location 600 meters from the landing site. At this speed, assuming that the collision is avoided through a combination of steering and deceleration, it may be necessary to sense approximately 300 meters ahead. A target speed of 250 knots may be expected during cruise, and a forward look ahead of approximately 1 km may be required. The final parameter associated with determining the exact brake position at landing may be a 15 cm (6 ” ) objects, which means that the resolution and accuracy of the ranging must be better than 5 cm. The sensor suite can perform landing zone (LZ) validation by classifying the terrain around the designated landing zone based on shape (rough / smooth, flat / level) and semantics (muddy / dry, vegetated / barren). During cruise flight and during final approach, the sensor suite can be used to discover obstacles 506, such as buildings, trees, power lines, and towers that may not be known in advance. Third, the sensor suite can be used to estimate the vehicle position in the event of GPS denial or interruption that may occur due to interference or due to terrain obstruction. The starting point design includes redundant, complementary sensors.

[0131] Scanning LIDAR 824. LIDAR (also known as laser scanner) refers to technology that can measure distance by illuminating a target with a laser and analyzing the reflected light. Thus, starting at 400 meters, a dual-axis scanning LIDAR can be used to provide high-resolution imaging to enable a "straight-in landing" at speeds of, for example, up to 135 knots. For very large energy fields of view (180°), the scanning LIDAR can be used as the primary sensor for environmental perception, whereby the laser ranging is able to penetrate all but heavy obstructions. The same LIDAR can be used for gesture recognition by personnel, which indicates final approval for landing or go-around. Appropriately positioned small LIDARs can also be used to determine whether the aircraft 102 is sinking in vegetation or soft terrain, and to avoid tail strikes during takeoff involving rudder steering.

[0132] Radar 826. Radars such as forward looking radars can be used to provide low resolution imaging through weather conditions and severe blackout conditions during landing. Forward looking radars can be configured to measure distances to objects such as transmission towers, tree lines, and other vehicles at distances up to 2 km. In addition, this modality can be used to perform "cruise missile" style navigation in GPS denied environments.

[0133] EO / IR Imager 828. Passive electro-optical infrared (EO / IR) imagers may be commonly used for navigation in GPS-denied environments, for terrain analysis, and for detection of water and vegetation.

[0134] Stereoscopic vision 838. A stereoscopic vision camera system 838 may employ two cameras that are horizontally displaced from each other to obtain two different views of an area (e.g., the area within the aircraft field of view 516). By comparing the two images, relative depth information may be obtained in the form of a disparity map that encodes the differences in horizontal coordinates of corresponding image points. The values ​​in the disparity map are inversely proportional to the depth of field at the corresponding pixel location.

[0135] Communications 812. The package delivery system architecture 800 may be operably coupled to one or more communications transceivers that may be used to wirelessly transmit data signals between the aircraft 102 and a remote system (such as a COP 110, a MOB 112, a FOB 114, an HSI device 514, and / or another system 822). For example, the wireless communications device may be configured to communicate data (e.g., surveillance data, mission planning data, flight control signal data, UGV navigation commands, etc.) with one or more remote systems. To facilitate optional wireless communications, the aircraft 102 may also include an air communications link capable of transmitting ("TX") and receiving ("RX") data using one or more antennas (e.g., top and bottom). The antennas may be controlled via a processor operably coupled to an RF switch. Thus, data collected or created by the architecture 800 of the package delivery system may be communicated with a remote system and / or any other device capable of wired or wireless communications using either a wired communications link or a wireless communications link.

[0136] Loss of Communications. For example, the package delivery system may provide three or more communication links between, for example: (1) MOB 112 and aircraft 102, (2) COP 110 and aircraft 102, (3) FOB 114 and aircraft 102, (4) UGV 512 and aircraft 102, (5) HSI device 514 and aircraft 102, and (6) between one or more of MOB 112, COP 110, FOB 114, UGV 512, and / or HSI device 514. Communications may be maintained by establishing any two communication links. For example, if communications are lost with both MOB 112 and combat outpost 110, aircraft 102 equipped with the package delivery system may perform a loss of communications contingency. If communications exist with either, aircraft 102 equipped with the package delivery system may continue its mission, except in situations where specific communications are required (e.g., landing confirmation from combat outpost 110 is required). In such cases, the behavior of the aircraft equipped with the package delivery system may be explicitly specified for the situation. Loss of communication with the main operating base 112 may be defined in accordance with the STANAG standard. Loss of communication with the combat outpost 110 may be defined as the absence of a heartbeat message for a predetermined time (e.g., 10 seconds to 300 seconds, more preferably 20 seconds to 100 seconds, and most preferably about 30 seconds). If communication is reestablished after executing the loss of communication emergency, the operator may override the behavior of the aircraft equipped with the package delivery system by uploading and executing new mission planning data. This may include commanding the aircraft 102 to execute its currently loaded mission. The main operating base 112 has the ability to upload new mission planning data, while the combat outpost 110 does not. Therefore, the only function that the combat outpost 110 can perform is to command the aircraft equipped with the package delivery system 102 to reestablish its mission to land at the previously designated landing zone (or its alternative zone).

[0137] GPS / INS 816d. When available, the GPS / INS device 816d can be used to provide latitude and longitude information and altitude. The GPS / INS device 816d uses GPS satellite signals to correct or calibrate the solution from the INS. GPS provides an absolute drift-free position value, which can be used to reset the INS solution or can be mixed with the INS by using a mathematical algorithm such as a Kalman filter. The angular orientation of the unit can be inferred from a series of position updates from the GPS. The error change relative to the position of the GPS can be used to estimate the unknown angle error. The benefit of using GPS with INS is that the INS can be calibrated by the GPS signal, and the INS can provide position and angle updates at a faster rate than GPS. For highly dynamic vehicles (such as missiles and aircraft), INS fills the gaps between GPS positions. Additionally, GPS may lose its signal, and the INS can continue to calculate position and angle during the loss of the GPS signal.

[0138] When the GPS / INS device 816d is unavailable (e.g., GPS denied operation), which may be due to poor reception or failure, the architecture 800 of the package delivery system remains functional in the event of a GPS signal interruption; for example, the interruption is due to active interference or obstruction from terrain. In general, the package delivery system can use visual landmarks to navigate the aircraft 102 for navigation during descent. Specifically, satellite maps of the area together with digital terrain evaluation data (DTED) can be used to determine significant features offline. Therefore, one or more databases that can be stored in a memory / data storage device can be used to store information related to (but not limited to) DTED, buildings and structures, geographic maps, military maps, and / or any other information that can be used to assist in navigating the aircraft. During the mission, features from the onboard camera are compared to the map to generate a navigation solution. Typically, the accuracy can be 1% of the travel distance for such applications. In the event that significant features are continuously available during the mission, the drift of navigation can be significantly reduced. Flight tests have shown that the navigation error can be up to 10 meters (3σ) within one hour of flight over different terrains.

[0139] Obscurant Penetration and Obstacle Detection during Enroute Cruise. Visually degraded environments are common in combat operations and may complicate the process of autonomous low-altitude flight. When flying close to or within clouds, or during the final approach where the landing area may be obstructed by a blackout, ensuring collision-free flight may require special consideration. The solution to this may be twofold. First, LIDAR can use a method called full waveform analysis to measure distance through the light of moderate obstructions. Compared with most laser rangefinders that use the first return that exceeds a threshold to calculate the range, this method looks at all the returns from the laser "chirp" to determine the farthest opaque object. The intermediate return can be automatically marked in the sensor hardware. Since laser ranging cannot penetrate the thickest obstructions, and because high cruising speeds may require km-level ranging, radar (e.g., W band (94GHz)) may be included to operate aircraft 102 in a blackout environment. At cruising speed, the radar look ahead range for objects such as transmission towers and ridge lines may be increased (relevant in the event of navigation failure). During final approach, the radar may provide returns corresponding to ground objects such as vehicles, buildings, and power lines. Radar returns from an ALG W-band (94Ghz) radar mounted to the front of the aircraft 102 descending into the landing zone may be plotted on the pilot's display during approach to the landing zone.

[0140] Gesture recognition for communicating with landing support specialists. LIDAR used for landing zone verification can be used to detect hand gestures by personnel on the ground. LIDAR used to perform landing zone verification and produce range images can detect simple hand gestures. Such images are robust to ambient lighting or require personnel to signal the aircraft using any fixed infrastructure. Hand and / or body gestures already used by marine landing support specialists and military pathfinders can be used to provide final confirmation of a touchdown or landing go-around.

[0141] The perception system provides at least three advantages, including: (1) combining scanning LIDAR and radar provides long-range detection and high resolution in degraded visual environments ("DVE") for determining optimal landing zones; (2) position estimates based on EO / IR imager / LIDAR / radar as needed (only EO / IR imager can be used for stealth), which is used to provide robustness (e.g., to GPS outages) and high performance; and (3) terrain analysis based on both geometry (from LIDAR) and vegetation properties (EO / IR imager). The primary sensor (scanning LIDAR) is a derivative of a common device for aerial surveying and provides higher accuracy, larger adaptive field of view, and lower price than other sensors such as flash LIDAR. The radar sensor can be configured to penetrate weather and blackout conditions to provide assurance of collision-free flight.

[0142] Internal interface. The configuration sent from the mission planning service to the mission sequencer indicates whether the configuration is complete and contains a catalog of specific action waypoints: (1) assembly waypoint 218 - the last waypoint of the takeoff route - used to change from takeoff to the flight route; (2) notification waypoint 222 - the point at which the package delivery system requests permission to land from combat outpost 110; and (3) approach waypoints - the last waypoint of the flight route and the first waypoint of the approach route. The configuration also contains information about the status of the updated mission and route deviation thresholds that may require route replanning if exceeded.

[0143] Route Data Request. A route data request is one half of a synchronous ROS service call to the Mission Planning Service that requests a set of waypoints along a given path. The request contains the route identifier, the starting waypoint identifier to retrieve, and either the number of consecutive waypoints to retrieve, or a threshold to retrieve all remaining waypoints in the path. Upon receiving this request, the Mission Planning Service replies with an array of waypoints.

[0144] Waypoints. This ROS message is the response to a request for route data. It includes an array of waypoints starting with the waypoint identifier indicated in the request and containing the requested number of waypoints. If the request indicated all remaining waypoints, this response will contain all remaining waypoints along the route.

[0145] Route Status. Route status messages indicate: the vehicle's current position relative to the route's waypoints, and a Boolean flag indicating whether the vehicle has veered too far off the planned route. The vehicle's waypoint position may be expressed as a floating point waypoint identifier, where the integer portion indicates the most recently passed waypoint, and the decimal portion indicates the decimal distance between the passed waypoint and the next waypoint the vehicle is flying towards (this may be the same representation as trajectory status). A Boolean flag indicates whether the vehicle flight path has deviated from the planned path beyond a certain limit, and may trigger a replanning if so determined by the mission sequencer.

[0146] Human-System Interface (HSI). The goal is to make autonomous aircraft generally fit in with the cargo delivery and casualty evacuation processes as seamlessly as possible. The advantages of the package delivery system come not only from the technical achievements, but also from its process that is easily adopted by personnel who interact with the vehicle at various stages of the mission without special training. Since the resources of units operating in remote locations are severely limited, landing in unprepared locations has historically required special attention. The interaction with the vehicle must be able to be achieved with minimal equipment and infrastructure. Therefore, field personnel can request services from aircraft equipped with the package delivery system for cargo resupply and CASEVAC. The interface must be intuitive and oriented around the tasks completed during cargo and casualty evacuation using an intuitive interface. In some cases, it may be necessary that the vehicle does not depend on the operator to land, because it may be guided to a location without ground personnel or communications. Therefore, during the critical landing phase, the most useful role of the operator may be to enhance safety, thereby allowing the complete human-machine system to perform at a higher level of capability than the machine or human pilot itself. Human-system interaction also enables a level of redundancy in the event that an onboard system fails or its performance may degrade.

[0147] The interface technology can be customized according to its usefulness to the MOB 112, FOB 114 and COP 110 operators. For example, at the MOB 112, this may result in a networked PC with a smart board. At the slightly more severe FOB 114, a UGCS-400 interface with a touch screen for simple negotiations may provide a possible solution. At the very severe COP 110, size, weight and power are particularly constrained, and we will consider mobile phone applications and functions. More complex functions (such as multi-vehicle tasking at the FOB 114, or complex negotiations with both the FOB and the vehicle at the COP 110) are possible.

[0148] Modular and open system architecture. The package delivery system can be provided with modular platform-independent processors, sensor suites, and software that can be adapted to a variety of aircraft and missions, thereby reducing the total cost of ownership and the time required to integrate developed technologies into field systems. Current solutions for complex architectures rely on a series of point-to-point devices, each with a dedicated interface. In order to reduce the number of components in these systems, many functions are usually combined in one device using tightly coupled code developed for specific hardware, operating systems, applications, and target platforms. Traditional architectures require a lot of time and financial resources to integrate, certify, and upgrade, while limiting lifecycle maintenance to one master integrator. However, the package delivery system will benefit from an improved architecture that allows functional modules to interoperate with clearly defined interfaces. In addition, the architecture also needs to support health monitoring, diagnosis, and restart of failed computer processes during the mission, all without the involvement of personnel on the vehicle.

[0149] The package delivery system functional architecture can be integrated in a robust open architecture framework to provide platform-independent automation for rapid resupply and CASEVAC. In order to open the development ecosystem, an open architecture and / or a global open architecture layer (GOAL) can be adopted to utilize module partitioning, hardware and software abstraction, loose coupling of functional modules, and the concept of a central standardized data exchange layer. For example, GOAL can be implemented as a service-oriented architecture (SOA) that uses the data distribution service (DDS) international standard to exchange data through a shared quasi-data model. Easily replace platform / task-specific modules in such architectures. DDS can be an anonymous publish / subscribe middleware standard that provides low overhead and low latency solutions for real-time processing and embedded applications. DDS allows extensive control of service quality attributes that ensure reliability, bandwidth, delivery deadlines, resource constraints, priority, security, and information assurance. Combined with hardware and operating system abstractions, the architecture facilitates component reuse, service "plug and play", open competition for development and integration, platform-independent functionality, and interoperability between system components and other systems. The architecture also allows other advances such as system health monitoring and sense and avoidance to be easily integrated into aircraft equipped with an entire package delivery system. GOAL enables generic AACU services to control "plug and play" platforms, sensors and devices via government controlled interface standards.

[0150] Fig. 9A high-level view of the proposed architecture consisting of functional modules is shown, each of which implements a clearly defined interface. The top layer may include multiple general services, such as task interface 902, task management 904, route planning 906, sensor perception and fusion 908, motion planning 910, and control device interface 912. One or more of the general services may be connected to the DDS open standard data bus 914 in communication. One or more plug-and-play hardware component layers are operably connected to the DDS open standard data bus 914. For example, an operating system abstraction 916 may be provided and designed to abstract the operating environment into at least one target operating system. The operating system abstraction 916 may implement an operating system application programming interface (API) for a specific underlying operating system and hardware platform. Therefore, the abstraction layer provides a direct implementation of functions that may not be supported by the underlying operating system. For example, an example abstraction layer may include a hardware abstraction layer 918, a hardware and driver abstraction layer 920, which may operably connect hardware such as communication and equipment 922, sensors 924, and platforms 926.

[0151] System Architecture. The package delivery system can be composed of a set of top-level subsystems. These subsystems are: ground units (e.g., forward operating base 114, main operating base 112, and combat outpost 110) and airborne units (e.g., VMS / VSM, mission manager, trajectory planner, perception system, and flight control system). Main operating base 112 and combat outpost 110 are ground control units, with main operating base 112 being a fully mature ground control system and combat outpost 110 being a small handheld device for use in the field. The airborne units include most of the package delivery system, with the exception of the VMS and flight control system, which are part of the aircraft.

[0152] Mission Manager Architecture. The mission manager is generally responsible for coordinating the main autonomous operations of the vehicle, including: (a) sequencing the main autonomous operations of the vehicle (such as notification of command outposts, receiving and processing go-arounds and aborts, etc.); (b) monitoring expected position information (waypoints) and providing the expected position information to the trajectory planner; (c) requesting automatic recalculation of the flight path when significant deviations from the expected flight path occur, or when the operating area (safety windage) of the mission changes; (d) The portability and openness of the present disclosure can be ensured by: (i) open communication standards (e.g., User Datagram Protocol (UDP)); and (ii) separating logic from messaging formats to facilitate future migration to a global open architecture; (e) Ease of use can be ensured by: (i) supporting IP-based radio communications with aircraft equipped with package delivery systems; (ii) developing using human-machine interface (HMI) methods available in modern mobile device applications (such as Apple's iOS or Google's Android); and (iii) being controlled by mobile, packable devices that utilize modern communication technologies.

[0153] Although the disclosure has been described with respect to what is currently considered to be a preferred embodiment, it should be understood that the invention is not limited to the disclosed embodiments. On the contrary, the present invention is intended to encompass various modifications and equivalent arrangements included in the spirit and scope of the appended claims. The scope of the appended claims should be given the broadest interpretation to include all such modifications and equivalent structures and functions. All documents cited herein (including journal articles or abstracts, published or corresponding U.S. or foreign patent applications, authorized or foreign patents, or any other documents) are each incorporated herein by reference in their entirety, including all data, tables, drawings and texts proposed in the cited documents.

Claims

1. A package delivery system for use with an aircraft (102), the package delivery system comprising: a land vehicle equipped with a first processor operably coupled to a first sensor package and a first communication transceiver (602), the first processor being configured to navigate the land vehicle along a navigation land route (118) based at least in part on data received via the first communication transceiver (602); a sensing system (814) having a second sensor package operably coupled to a second processor and a second communication transceiver (602); and a human-system interface device (HSI device (514) for communicating with the land vehicle and the perception system (814) via one or more wireless links, wherein the HSI device (514) individually and separately from each of the land vehicle and the perception system (814) comprises a third processor, a third communication transceiver (602), a user input device and a display device, wherein the perception system (814) is configured to transmit navigation commands to the land vehicle via the first communication transceiver (602) based at least in part on data from the second sensor package, and wherein the HSI device (514) is configured to transmit navigation commands to the land vehicle via the third communication transceiver (602) and to receive a video feed captured by the first sensor package from the land vehicle.

2. The package delivery system of claim 1, wherein the second sensor package comprises a first optical payload (510) and a second optical payload (510), each of the first optical payload (510) and the second optical payload (510) being positioned at a different location on the aircraft (102).

3. The package delivery system of claim 2, wherein the first optical payload (510) is configured to generate terrain data of the geographic area prior to landing at a drop zone (304, 504) within the geographic area, wherein the second processor is configured to determine the navigational land route (118) from the drop zone (304, 504) to the field operator based at least in part on the terrain data.

4. The package delivery system of claim 3, wherein the second optical payload (510) is configured to track the land vehicle from the drop zone (304, 504) as the land vehicle travels along the navigation land route (118).

5. The package delivery system of claim 4, wherein each of the first optical payload (510) and the second optical payload (510) comprises a LIDAR (824) or a stereo vision camera system.

6. The package delivery system of claim 3, claim 4 or claim 5, wherein the second processor is configured to determine a plurality of navigational land routes (118) from the drop zone (304, 504) to the field operator based at least in part on the terrain data.

7. The package delivery system of claim 6, wherein the HSI device (514) is configured to present the plurality of navigational land routes (118) to the field operator via the display device for selection by the field operator through the user input device.

8. A package delivery system according to claim 1, claim 2, claim 3, claim 4, claim 5, claim 6 or claim 7, wherein the perception system (814) is configured to (1) detect obstacles along the navigation land route (118) in real time, and (2) dynamically transmit navigation commands to autonomously navigate the land vehicle as the land vehicle travels along the navigation land route (118).

9. The package delivery system of claim 1, claim 2, claim 3, claim 4, claim 5, claim 6, claim 7 or claim 8, wherein the HSI device (514) is configured to display the video feed on the display device.

10. The package delivery system of claim 1, claim 2, claim 3, claim 4, claim 5, claim 6, claim 7, claim 8 or claim 9, wherein the HSI device (514) is configured to receive a second video feed captured by the second sensor package from the perception system (814), and the HSI device (514) is configured to display the second video feed on the display device.

Images